The run that took my best number away

By Daniel S. · August 11, 2026

Part two of the two-Spark story (part one is the build). The cluster was serving a 284B model, published reference numbers exist for this exact setup, and the obvious next step was to compare. As usual, an AI agent ran the benchmarks; the retraction below is of its number, published on this blog's watch, so it's ours to take back.

1. First read: we're 5× slower. Also first read: wrong.

We ran the reference's own harness, on its own grid, against its published numbers. First read looked bad — prefill 4.8× slower, time-to-first-token 6× worse.

There was a culprit ready and waiting: the server was warning that a scheduling limit had been clamped and "may lead to suboptimal performance." Plausible, official-sounding, and attached to a real warning message.

That culprit was impossible. The failing case sends ~280 prompt tokens against a limit of 8,168. 280 is not close to 8,168. The knob we were about to turn could not have caused the symptom we were looking at — and the only reason we checked was a house rule about not tuning against numbers whose provenance you haven't read.

The real cause: we weren't running the same benchmark. The reference runs produce ~512 output tokens per request. Ours produced 2,437–5,680 — because our config ran the model's thinking mode at maximum, and the harness counts reasoning tokens as output. We were doing five times the work per request and calling the difference a hardware deficit.

2. Then the checkpoint bit back

The fix should have been one flag: lower the reasoning effort to match. The recipe's own menu offers low. Setting it made the server fail every single request — the value is valid for one checkpoint of this model and invalid for the one we were serving, and it doesn't fail at startup. It fails on each request afterwards, with the server otherwise healthy and the models endpoint returning 200.

high worked and got us to ~828 output tokens. Closer. Still not 512. So the first comparison table went out with the workload mismatch disclosed in its own caveats — hold that thought.

What the semi-matched run did establish: time-to-first-token tracks the published reference within noise at every prompt length from 8K up —

TTFT is the one metric that output-length differences can't distort, which is exactly why it's the load-bearing row. And the "6× worse" short-prompt number? Those cells ran 8.37 → 9.45 → 5.19 → 0.92 s across the run. The hardware reaches 0.92. The early cells were cold — a warmup artefact that was one careless read away from being published as a hardware deficit.

3. Two numbers we refused to publish

The aggregate column at 131K showed +4323% and +6050% in our favour.

Both compare against an upstream failure — every reference request in those cells is marked cancelled in their own data. It's a probe that didn't complete, documented as such. Quoting it would have been the most flattering number in the whole exercise and completely meaningless.

Our side had one too: a single request that generated 1,015,784 tokens — a runaway that ran to the context ceiling — dragging its row's mean four orders of magnitude up. Ours to disclose, not to bank.

The last discipline in benchmarking is refusing your own best number. Everything before that is measurement. That part is character, and it's the only reason anyone should trust the rest of the table.

4. The control run — and what it cost

The semi-matched table ended with a number worth being proud of: decode +8% to +101% over the published reference from 32K up. It also ended with an admission: different weights revision, different output lengths, so nothing in it was a clean hardware-vs-hardware claim. The only fix was to serve the same checkpoint the reference used and match the workload for real.

Two traps before a single token was measured:

• The tested revision had moved. The reference documents one commit; the repo's main now resolves to a different one. The obvious one-line download would have quietly reintroduced the exact confound the run existed to remove. Pin the revision — main moves. The tested commit and the default branch are different claims.
• pkill -f killed the shell running it. The kill pattern matched the SSH command line that contained the pattern. The only symptom was an empty output block — an exit code cannot show you this one.

Then the real question: what does "same workload" actually mean? Probing the reasoning levers directly, five requests each:

Two findings fell out of one small table. low isn't "barely thinking" — it's unstable, and anything averaged over it inherits the variance. And an unrecognised setting failed open: reasoning_effort: "off" returned 200 and produced more reasoning than the default. The setting that reads as "less" did "more." Verify a setting by measuring its effect, never by its acceptance.

With thinking actually off: 512, 512, 512, 512, 515 output tokens against the reference's 512–529. Parity, at the cost of a six-minute restart.

5. What matched weights and matched workload showed

Decode: 18 of 18 comparable cells within ±10% of the published reference. Median ratio 1.011.

Not a win. A reproduction — which is the thing we were actually trying to establish and had no standing to claim before. And then the part that has to be written down:

The +101% became −5%. It was never the hardware. It was five times the tokens per request, and per-request decode rate rewards long outputs. The confound was disclosed in the original caveats — and the caveat turned out to be the entire story.

What survived: one real edge at 131K single-stream (69.4 vs 65.2), and a genuine TTFT result at 32K — 16.73 s vs 22.96 s, 27% faster. Smaller claims. Defensible ones.

One more correction of our own explanation: part one blamed the short-prompt TTFT spikes on cold shapes loading from storage. The control run warmed everything first — and the spike moved to different concurrency levels instead of disappearing. A storage story can't produce that pattern; per-batch-size CUDA-graph capture can. It's recorded as suspected, unconfirmed, with the deciding experiment named — because the previous tidy explanation was tidy and wrong, and only the better warmup surfaced it.

The rules this run paid for

10. Pin the revision — main moves. The tested commit and the default branch are different claims, and only one is the thing you compared against.
11. A self-matching pkill -f kills the shell running it. Exit codes cannot show you this.
12. An unrecognised enum can fail open. A setting that returns 200 has been accepted, not obeyed — measure its effect.
13. The confound you disclose is still a confound. Caveats are not controls. The only thing that settles it is removing the variable and re-running — 30 minutes of compute, in this case, and it retracted our best result.

Rules 1–9 are in part one. The numbering continues because it's one list — the same day, the same discipline, and the same lesson wearing different clothes each time.