Two Sparks, one model: shipping box to a 284B cluster in a day

By Daniel S. · August 11, 2026

The second DGX Spark arrived — an ASUS GX10 this time, joining the Acer GN100 from the first two posts. Same GB10 board underneath, different chassis. The goal was the only goal that justifies owning two of these: models too big for one box. As always on this blog, an AI agent did most of the hands-on work below, and its wrong turns are recorded as its own.

Here's the headline, in timestamps rather than vibes — every one read off the boxes:

Shipping box on the desk to a 284-billion-parameter model answering questions, inside a day.

Worth saying plainly: almost none of that time was the hard part. The install was fast. What consumed the hours were failures that each pretended to be a different failure — and that's what this post is actually about, because it's the part that transfers to your hardware.

In this kind of work the expensive failures are not the loud ones. They are the failures that render as a different failure — or as success. Nearly every hour lost in this session went to a green light that meant nothing, and nearly every one was caught by a control, never by re-reading our own notes.

1. "Is the interconnect done?" took four wrong answers

The first simple question — do both boxes look healthy, and is the interconnect complete? — produced four wrong answers before a right one.

The dark box was a red herring. The first Spark stopped answering entirely: ping dead, SSH timeout, serving endpoint gone. The discriminating detail was the ARP entry reading (incomplete) — nothing answered at layer 2, so the host was genuinely off the segment, not merely firewalled. (A box that drops ping still populates ARP. Not answering and not there look identical until you check the right layer.) The cause: someone was physically handling the machine. Never diagnose during somebody else's hands-on.

"Two cables" — wrong, and a human caught it. ethtool -m reported a QSFP module present with Link detected: yes on two interfaces, so the agent published "two DAC cables are seated." There is one. Three independent readings eventually proved it: the same transceiver serial on both ports, both ports flapping in the same second in the kernel log, and an identical phys_switch_id on both. On this hardware one QSFP cage is visible through two PCI domains, so it enumerates twice with different MACs. Netdev-count is not port-count. Read the serial before claiming a cable count.

The fabric was up and carrying nothing. Link ACTIVE, RoCE ACTIVE, 200000 Mb/s — and zero addressing, no config, no peer answering an all-nodes multicast. What made that reading meaningful was a control: the same probe on the LAN port found five neighbours instantly. Without it, "no answer" is indistinguishable from a broken probe.

Link detected: yes on a DAC means the far-end NIC is alive and negotiated — a cable into a dead box does not link at 200G. So not-configured ≠ not-connected, and that distinction was worth about two hours.

2. The 200 GbE link that can never do 200 Gb/s

Once both ends had point-to-point addresses, the fabric carried traffic immediately:

It doesn't scale with more queue pairs — so it's not a concurrency limit. Instead of hunting a bottleneck, compute the ceiling:

ConnectX-7:  LnkCap = Speed 32GT/s, Width x4    <- capability, not a degraded negotiation
PCIe Gen5 x4 -> ~126 Gb/s after encoding
measured 111.86 = 89% of ceiling

This "200 GbE" link cannot deliver 200 Gb/s on this hardware, and no configuration will change it. The port genuinely negotiates 200G; the host can't feed it faster than a Gen5 x4 slot allows. Budget ~112 Gb/s. And 89% of theoretical is excellent RDMA efficiency — the fabric is performing near its true limit, which is a design property, not a fault to chase.

Our own cable suspicion died here too, correctly. The DAC's EEPROM identifies as a 40G-era QSFP28 part while the port runs 200G. After a 40,000-iteration run: zero CRC errors, zero symbol errors, zero retransmits — and the link-flap counter matched the two kernel-log flaps exactly, a second instrument corroborating the first. The cable is clean. Retired by measurement, not by argument.

3. A gallery of failures that rendered as other failures

Five in one day, all the same shape.

The container that ran with no network — while docker ps said Up. After a reboot, vLLM logged "Application startup complete," the port bindings looked correct in the config — and the endpoint returned nothing, from its own localhost. The container had no IP at all. Its restart policy was on-failure, and vLLM wasn't failing — so nothing would ever retry it. It would have sat there, green, forever. docker restart didn't fix it; a full stop/start didn't; restarting the Docker daemon didn't. What found it: a fresh throwaway container from the same image got a network instantly — so the daemon was fine, and that container's stored state was the fault.

A stale lock file turned a service into a silent no-op — and systemd called it success. A timer was enabled, active, firing on schedule — and every run "Deactivated successfully" in the same second, against a launcher that should have run for 280 seconds. A lock file left by an unclean exit made the job exit 0 without doing anything. The box vanished from its message bus for hours while every instrument read green. Never exit 0 when you decline to do the job.

"3 incorrect password attempts" — with a perfect password. A script piped a password into ssh and fed the remote commands as a heredoc. The heredoc silently outranks the pipe as stdin, so the password was discarded and sudo read the script's first line as the password. It presented as a credential fault — and the natural fix, deleting and re-entering the stored secret, would have destroyed a good credential and left the bug in place. What prevented that: inspecting the secret's shape without revealing it (right length, clean ASCII) — password fine, transport wrong.

Smart quotes, baked into a credential. A pasted command turned straight quotes into curly ones, so a keychain entry's account name literally contained “…”. Storage and retrieval worked by one lookup path; the canonical lookup returned nothing — which reads exactly like "no credential exists," and would have been filed as a blocker.

One corrupt file broke a subsystem, silently, for ten months. netplan generate failed on the new box because a factory-image file dated ten months earlier contains control characters at byte 1. netplan parses every file in the directory before doing anything, so one bad file breaks all netplan operations — invisible until the first time anyone ran it.

4. Models that don't fit, and the arithmetic nobody does first

The plan was four frontier-class open models across the pair. Parameter counts are not download sizes, and download sizes are not memory footprints — checking the actual repos changed the plan:

At NVFP4 all four fit (129–184 GB) — and none fits on a single Spark. That's the honest version of "two Sparks is the sweet spot": it isn't a preference, it's the smallest configuration where these models exist at all. One provenance note worth saying out loud: three of the four have first-party quantised checkpoints; the fourth only has third-party conversions. That's a real difference in what you're trusting, and it shouldn't be buried in a --model flag.

5. The recipe that would have failed on copy-paste

Two published dual-Spark recipes exist. Both are good. Both hardcode the RoCE device names — and on our boxes the port they name is down with no cable in it; the live port is the other one. Pasted verbatim, NCCL targets a dead device, and the plausible worst case isn't a crash — it's a silent fallback to the 1 GbE LAN, which reads as "the cluster works, just slower than advertised." A 112 Gb/s fabric quietly replaced by 1 Gb/s is exactly the class of failure this whole post is about.

Same shape: the RoCE GID index has to be resolved dynamically. One recipe says 0, the other says "head 3, worker 6," ours was 3/3 — and it drifts across reboots.

Every recipe is written on somebody else's hardware. The parts that are true everywhere are the reasoning; the parts that are true nowhere else are the device names.

6. It serves — and the first "failure" was the rule we'd written that morning

The cluster came up: rank 0 on one Spark, rank 1 on the other, NCCL over the RoCE fabric, API live in about 6 minutes against a published ~11. KV-cache pool: 1,498,260 tokens — 1.43× the model's 1M context ceiling.

Then the first real request returned HTTP 200 with completely empty content.

finish_reason: length. The config ran the model's thinking mode at maximum, all 800 tokens of the budget went into private reasoning, and none reached the answer. At a 6,000-token budget it used 1,307 and finished stop — with 6,317 characters of reasoning behind a 284-character reply.

That is the thesis of this post firing on its own author, hours after writing it down: discriminate on finish_reason, never on HTTP status — truncation presents as a clean 200. The only reason it took thirty seconds instead of an afternoon is that the rule was already written.

And the right fix is not the obvious one. The instinct is to cut the reasoning budget. But these are local models on idle GPUs — tokens cost seconds, not money — and we've already recorded an occasion where rationing them quietly depressed the speed numbers being published. The fix was documenting a floor (max_tokens ≥ 2000), not shrinking the model's thinking.

One more second actor worth naming: docker stop on the old single-box server was undone by its own --restart on-failure policy. It came back holding 44 GB of GPU memory, and the cluster launch found 66 GiB free where it needed ~115. On unified memory this is nastier than it sounds — the GPU allocation doesn't show in RSS, so ps showed a 4 GB process while nvidia-smi showed 44 GB. docker update --restart=no before stop.

Every stop needs to answer "what would un-stop this?" — a restart policy is a second actor with an opinion about your intentions.

The reusable rules this day paid for

1. A green light is worth exactly its discriminating power. Ask: if I were wrong, would this output look any different? If no, it isn't evidence.
2. can't-run is not pass — and must never render as one.
3. Run a control in the same session. Every real error here was caught by a control or a measurement; none was caught by re-reading our own work.
4. Compute the ceiling before arguing about the bottleneck.
5. Never exit 0 when you decline to do the job.
6. Inspect a secret's shape, not its value, when a credential looks "wrong."
7. Port-count ≠ netdev-count. Read the transceiver serial.
8. One writer per stdin. A heredoc silently outranks a pipe.
9. Adapt device names from every recipe; inherit only the reasoning.

There's a part two. The cluster was serving, so we benchmarked it against the published reference numbers — and the best number we got was one we ended up taking back ourselves. That story is the next post.