The candidate that failed eight gates

In the first week of August 2026 I put a 117-billion-parameter model, gpt-oss-120b, through the acceptance gates I had written for my two DGX Sparks. It failed eight of them. Its 95th-percentile latency on the verification suite was 18,893 ms against the incumbent's 4,259 ms, a 4.4x gap against a ceiling of 1.5x, and its median time to first token was 6,157 ms against 89 ms. Nearly 90 percent of its tool calls were malformed under the harness: 765 of 851. On the verification suite the incumbent already scored 45 of 45, it scored 40. The decision line at the top of the report read KEEP INCUMBENT (default), and nothing about the model made me want to argue with it.

The time-to-first-token gap, about 69x at the median, carries a method caveat I come back to below: it was measured over the mixed-size prompts of the evaluation matrix, not the fixed 8k-token probe the gate specified. The latency gate was measured as specified.

Where the candidate came from

gpt-oss-120b arrived as a recommendation from a benchmark-led research pass. Its published SWE-bench Verified score was 62.4, against 50.3 for the 30-billion-parameter Qwen3-Coder model already serving on the first Spark. A second candidate, Qwen3-Coder-Next in FP8, published 70.6. On a leaderboard, both beat the incumbent and the decision is made.

An independent review of that ranking found three problems before any weights downloaded. The 70.6 was measured on the BF16 checkpoint, not the FP8 artifact I would serve. The scaffolds behind the three numbers were unknown and almost certainly different from mine. None of them were produced on a GB10 with 273 GB/s of memory bandwidth in an NGC vLLM container. The numbers ranked published artifacts under unknown conditions. They did not rank what I would run.

So the rule became: published benchmarks may be cited as a prior for what to expect, never as gate evidence. Both models became candidates. Promotion would require passing locally measured thresholds that were written down first.

Keep the incumbent, by default

The thresholds document was written on 2026-08-04, before any candidate had been served and before the weights finished downloading. Every number in it was chosen without knowing a result. Its first rule: the default decision is to keep the incumbent. Promotion requires every hard gate to carry a pass verdict backed by local evidence. A gate with no evidence is not a pass. If the evaluation is abandoned partway, the incumbent stays.

That default is cheap to hold, because the incumbent is never touched during an evaluation. "Keep" is the null action. It also puts the burden of proof entirely on the challenger, which is where it belongs when the challenger is a leaderboard number and the incumbent is a service other things depend on.

The incumbent doesn't have to pass these gates; they set the bar for replacing it. A candidate has to match it on the verification suite, beat it by 10 points on the agentic suite (half the 20.3-point leaderboard gap), and reach 0.60 there outright, since going from 0.20 to 0.30 is not a usable coding model.

The second rule is change control. Changing any threshold after results exist requires recording the old value, the new value, the reason and the date, in the same file. I used it once, on the agentic-task gate, after the evidence showed that two of the four tasks were unsolvable by every Layer 1 arm, the incumbent included. The record is in the document. So is the fact that the fix was flagged as a measurement-design defect rather than a grading argument, and that the 0.60 floor itself did not move.

The third rule is that a waiver is not a pass. A hard gate can be waived only by me, only in writing, only before promotion, and it is reported as waived forever. Stability gates cannot be waived at all. I waived two gates for the Qwen candidate: the malformed-tool-call gate, because the harness counted every narration turn as malformed and measured the incumbent at 28.3 percent under identical bytes, so the bar could not discriminate; and the time-to-first-token gate, because 1.98x the incumbent at 8k tokens sat well inside the 3-second absolute backstop. Neither waiver applied to gpt-oss-120b. At 89.9 percent malformed, the tool-call gate was discriminating truthfully, and it stood as a fail.

What "no_evidence" buys you

Six of the gpt-oss-120b gates did not fail. They read no_evidence: decode throughput, the 24-hour soak, memory headroom, restart, post-restart reproduction, the Codex protocol probe. None had been run for that arm: the four stability gates are a pre-promotion step, and the decode and Codex probes were simply not run for it.

A leaderboard has no such cell. It has a number or a blank, and a blank can mean "not tested" or "not applicable". no_evidence is a verdict. It says: this gate exists, it matters, we did not measure it, and it counts against promotion exactly as a fail does. When the second candidate reached its soak, the same cell read "IN PROGRESS: 6.9h/24h, 43 samples, 0 bad so far" and still did not count as a pass. The gate report issued no verdict before the 24-hour mark.

The candidate I wanted

Qwen3-Coder-Next was the candidate I expected to promote. In the second round it passed the verification suite 45 of 45, passed the fabrication gate, ran at 48.8 tokens per second, and scored a utility ratio 11 times the incumbent's. It failed only the 128k-context gate, which the thresholds had already provided an escape hatch for: promote with a 64k cap, recorded as a scope limit. I wanted it.

Its agentic-suite result was marked provisional, because the subset it was scored on had 18 runs and the sample-size gate demands 30. The expansion ran 56 of 56 planned runs. The candidate scored 12 of 39 on the solvable subset, 0.308 against a 0.60 floor. Not provisional, not close. The round-1 rows that had read 0.667 came from a serving instance no later launch could reproduce. The reproducible serving is the system under test, because it is what a promoted lane would run. On 2026-08-07 the evaluation closed. Keep incumbent, both candidates.

The gate held against the result I wanted, using rules I had written before I wanted anything.

The same discipline, one repo over

I run a read-only code-quality scanner across my repos, driven by the same incumbent model. On 2026-08-09 its evaluator was accepting 6 percent of the findings it produced. The charter I wrote that day says the objective is useful novel findings per scan, not total findings, and every change should improve at least one of precision, calibration, measurement integrity or reproducibility. The benchmark, it says, is becoming the product as much as the scanner.

The first prompt revision I put through the gate drove findings to zero. Every secondary check passed: no marker noise, less than half the latency, no coverage regression. The gate still failed it. Useful novel findings over total findings was 0 over 0, the one genuine finding the old prompt had surfaced was gone with the noise, and the rule that zero findings never passes was made mechanical the same day. A gate that only measured noise would have promoted the silent prompt. Two revisions later, one passed on one repo, mechanically, with an accept rate of one in three on three judged findings, and the report says so in those words. Extending it to two more repos failed both.

What changed after

The gates became a pipeline. A watcher finds new models and serving images, a trial on the second Spark runs the fixed ladder against a same-window incumbent control, an auto-drafted findings document goes through an advisory second-model review, and two yes/no/defer asks stand between discovery and a promotion pull request. The default verdict is still keep_incumbent. The first thing the pipeline evaluated was the incumbent as its own candidate, to measure what a trial costs before any budget was registered.

The model on the first Spark today is the one that was there in July. The serving image under it has changed; the model has not. That refresh, on 2026-09-04, fired two of the standing re-evaluation triggers, a new vLLM image and new runtime flags, and the incumbent's gates were re-derived under change control the same day.