The candidate that failed eight gates
In the first week of August 2026 I put a 117-billion-parameter model, gpt-oss-120b, through the acceptance gates I had written for my two DGX Sparks. It failed eight of them. Its 95th-percentile latency on the verification suite was 18,893 ms against the incumbent's 4,259 ms, a 4.4x gap against a ceiling of 1.5x, and its median time to first token was 6,157 ms against 89 ms. Nearly 90 percent of its tool calls were malformed under the harness: 765 of 851. On the verification suite the incumbent already scored 45 of 45, it scored 40. The decision line at the top of the report read KEEP INCUMBENT (default), and nothing about the model made me want to argue with it.
The time-to-first-token gap, about 69x at the median, carries a method caveat I come back to below: it was measured over the mixed-size prompts of the evaluation matrix, not the fixed 8k-token probe the gate specified. The latency gate was measured as specified.
Where the candidate came from
gpt-oss-120b arrived as a recommendation from a benchmark-led research pass. Its published SWE-bench Verified score was 62.4, against 50.3 for the 30-billion-parameter Qwen3-Coder model already serving on the first Spark. A second candidate, Qwen3-Coder-Next in FP8, published 70.6. On a leaderboard, both beat the incumbent and the decision is made.
An independent review of that ranking found three problems before any weights downloaded. The 70.6 was measured on the BF16 checkpoint, not the FP8 artifact I would serve. The scaffolds behind the three numbers were unknown and almost certainly different from mine. None of them were produced on a GB10 with 273 GB/s of memory bandwidth in an NGC vLLM container. The numbers ranked published artifacts under unknown conditions. They did not rank what I would run.
So the rule became: published benchmarks may be cited as a prior for what to expect, never as gate evidence. Both models became candidates. Promotion would require passing locally measured thresholds that were written down first.
Keep the incumbent, by default
The thresholds document was written on 2026-08-04, before any candidate had been served and before the weights finished downloading. Every number in it was chosen without knowing a result. Its first rule: the default decision is to keep the incumbent. Promotion requires every hard gate to carry a pass verdict backed by local evidence. A gate with no evidence is not a pass. If the evaluation is abandoned partway, the incumbent stays.
That default is cheap to hold, because the incumbent is never touched during an evaluation. "Keep" is the null action. It also puts the burden of proof entirely on the challenger, which is where it belongs when the challenger is a leaderboard number and the incumbent is a service other things depend on.
The incumbent doesn't have to pass these gates; they set the bar for replacing it. A candidate has to match it on the verification suite, beat it by 10 points on the agentic suite (half the 20.3-point leaderboard gap), and reach 0.60 there outright, since going from 0.20 to 0.30 is not a usable coding model.
The second rule is change control. Changing any threshold after results exist requires recording the old value, the new value, the reason and the date, in the same file. I used it once, on the agentic-task gate, after the evidence showed that two of the four tasks were unsolvable by every Layer 1 arm, the incumbent included. The record is in the document. So is the fact that the fix was flagged as a measurement-design defect rather than a grading argument, and that the 0.60 floor itself did not move.
The third rule is that a waiver is not a pass. A hard gate can be waived only by me, only in writing, only before promotion, and it is reported as waived forever. Stability gates cannot be waived at all. I waived two gates for the Qwen candidate: the malformed-tool-call gate, because the harness counted every narration turn as malformed and measured the incumbent at 28.3 percent under identical bytes, so the bar could not discriminate; and the time-to-first-token gate, because 1.98x the incumbent at 8k tokens sat well inside the 3-second absolute backstop. Neither waiver applied to gpt-oss-120b. At 89.9 percent malformed, the tool-call gate was discriminating truthfully, and it stood as a fail.
What "no_evidence" buys you
Six of the gpt-oss-120b gates did not fail. They read no_evidence: decode throughput, the 24-hour soak, memory headroom, restart, post-restart reproduction, the Codex protocol probe. None had been run for that arm: the four stability gates are a pre-promotion step, and the decode and Codex probes were simply not run for it.
A leaderboard has no such cell. It has a number or a blank, and a blank can mean "not tested" or "not applicable". no_evidence is a verdict. It says: this gate exists, it matters, we did not measure it, and it counts against promotion exactly as a fail does. When the second candidate reached its soak, the same cell read "IN PROGRESS: 6.9h/24h, 43 samples, 0 bad so far" and still did not count as a pass. The gate report issued no verdict before the 24-hour mark.
The candidate I wanted
Qwen3-Coder-Next was the candidate I expected to promote. In the second round it passed the verification suite 45 of 45, passed the fabrication gate, ran at 48.8 tokens per second, and scored a utility ratio 11 times the incumbent's. It failed only the 128k-context gate, which the thresholds had already provided an escape hatch for: promote with a 64k cap, recorded as a scope limit. I wanted it.
Its agentic-suite result was marked provisional, because the subset it was scored on had 18 runs and the sample-size gate demands 30. The expansion ran 56 of 56 planned runs. The candidate scored 12 of 39 on the solvable subset, 0.308 against a 0.60 floor. Not provisional, not close. The round-1 rows that had read 0.667 came from a serving instance no later launch could reproduce. The reproducible serving is the system under test, because it is what a promoted lane would run. On 2026-08-07 the evaluation closed. Keep incumbent, both candidates.
The gate held against the result I wanted, using rules I had written before I wanted anything.
The same discipline, one repo over
I run a read-only code-quality scanner across my repos, driven by the same incumbent model. On 2026-08-09 its evaluator was accepting 6 percent of the findings it produced. The charter I wrote that day says the objective is useful novel findings per scan, not total findings, and every change should improve at least one of precision, calibration, measurement integrity or reproducibility. The benchmark, it says, is becoming the product as much as the scanner.
The first prompt revision I put through the gate drove findings to zero. Every secondary check passed: no marker noise, less than half the latency, no coverage regression. The gate still failed it. Useful novel findings over total findings was 0 over 0, the one genuine finding the old prompt had surfaced was gone with the noise, and the rule that zero findings never passes was made mechanical the same day. A gate that only measured noise would have promoted the silent prompt. Two revisions later, one passed on one repo, mechanically, with an accept rate of one in three on three judged findings, and the report says so in those words. Extending it to two more repos failed both.
What changed after
The gates became a pipeline. A watcher finds new models and serving images, a trial on the second Spark runs the fixed ladder against a same-window incumbent control, an auto-drafted findings document goes through an advisory second-model review, and two yes/no/defer asks stand between discovery and a promotion pull request. The default verdict is still keep_incumbent. The first thing the pipeline evaluated was the incumbent as its own candidate, to measure what a trial costs before any budget was registered.
The model on the first Spark today is the one that was there in July. The serving image under it has changed; the model has not. That refresh, on 2026-09-04, fired two of the standing re-evaluation triggers, a new vLLM image and new runtime flags, and the incumbent's gates were re-derived under change control the same day.
The pre-registered design
Two documents, both dated 2026-08-04, before any candidate weights finished downloading. The evaluation spec says how to measure. The thresholds document says what counts as passing. Neither may be edited after results exist without recording the edit and the reason.
The spec separates two questions that model evaluations habitually conflate. Layer 1 holds everything except the model constant: byte-identical prompts hashed into the run record, one shared tool schema, the same cached retrieval chunks replayed to every arm, temperature 0, fixed seed, one raw chat-completions scaffold. It is deliberately unflattering to every model, because a model that only wins with a bespoke prompt has shown that someone tuned a prompt. Layer 2 lets complete products compete as shipped, Claude Code in the cloud and Codex CLI driving the local candidate. Layer 2 is a reference, never a promotion path, and the two layers are never reported as one number.
Three suites. Suite A is verification and judging probes: single shot, no tools, deterministic ground truth computed by plain code, frozen fixtures. It is the workload the incumbent already serves and the only suite where a regression is unambiguous. Suite B is bounded agentic tasks against a pinned copy of a private repo, each with a deterministic component such as a pytest exit code, one of them injecting a clobbering write and a false-green test run to see whether the model recovers. Suite C is retrieval-grounded question answering over my personal knowledge vault, with retrieval recall scored separately from answer correctness and a negative control the corpus cannot answer.
Gate 0 came before any candidate: re-measure the incumbent through the same harness on the full suite A set, five trials, at least 35 scored runs, capturing time to first token, total latency and decode throughput. Every relative gate is expressed against the incumbent, and without a same-harness incumbent measurement the comparison silently becomes candidate-versus-nothing. The planned matrix was 376 runs across five arms; round 2 closed at 460 of 460.
The gate table
Table 1 is the round-2 gate table for gpt-oss-120b, condensed from the report. Round 2 re-graded 19 suite A rows offline after an arm-neutral grader fix (the quoted-evidence check had whitelisted only the log block, so models quoting the spec text were misgraded as fabricators), which moved this candidate from 36 to 40 of 45. It still failed.
| Gate | Verdict | Observed | Threshold |
|---|---|---|---|
| Q1 suite A agreement | fail | 40/45 = 0.889, Wilson 0.765 to 0.952; incumbent 1.000 | >= incumbent and >= 0.90 and Wilson lower >= 0.80, n >= 35 |
| Q2 suite B success | fail | 1/27 = 0.037 full suite; 1/18 = 0.056 on the solvable subset (Wilson 0.010 to 0.258) | >= incumbent + 10 pp (full suite) and >= 0.60 (frozen subset) |
| Q3 fabrication | pass | 0 in 15 control runs | 0 hard fabrications, never worse than incumbent |
| Q4 trial variance | pass | 0 flaky units; rule later found vacuous as implemented, rewritten 2026-09-05 | no unit flips in more than 2 of 5 trials (suites A, C) or 1 of 3 (suite B) |
| S1 decode | no_evidence | probe not run for this arm | >= 25.0 tok/s, floor 20.0 |
| S2 time to first token | fail | p50 6,157 ms, p95 13,364 ms; incumbent 89 / 141 ms (69.18x / 94.78x); mixed-size matrix prompts, method caveat | p50 <= 1.25x, p95 <= 1.5x incumbent at an 8k prompt |
| S3 suite A p95 latency | fail | 18,893 ms vs 4,259 ms (4.44x) | <= 1.5x incumbent p95 |
| S4 agent-loop utility | pass | U = 0.061 vs 0.050 (1.23x) | >= 1.0x incumbent |
| R1 malformed tool calls | fail | 765/851 = 89.9%; incumbent 28.3% under identical protocol | <= 2% of tool calls |
| R3 structured output parse | pass | 0 failures in 45 | <= 1 failure in 35 |
| R4 retries per success | pass | 0.00 vs incumbent 11.00 | <= 1.5x incumbent |
| R5 human intervention | pass | 0 of 122 runs | <= 10% |
| X1 32k to 64k | fail | B-04 retention 1/3 to 0/3 (33 pp drop); 64k suite A sweep missing for this arm | <= 5 pp drop |
| X2 32k to 128k | fail | B-04 1/3 to 0/3; needle probe not run for this arm | <= 10 pp drop and needle >= 90% |
| ST1 to ST4 stability | no_evidence | soak not started for this arm | 24 h clean; restart within 10 min; <= 2% memory growth; post-restart within 10% |
| P1 responses API | no_evidence | probe not run for this arm | 20 consecutive clean Codex turns |
| $1 cost per success | fail | $0.01044 vs $0.00159 | <= incumbent, or suite B delta >= 15 pp |
| $2 operator interventions | pass | 0 recorded | <= 5 |
| E1 to E4 evidence quality | pass | 0 provenance gaps; two non-contestant judges; 0 contaminated canaries; suite A n = 45, B n = 27, C n = 40 | complete provenance; judge independence; no leakage; n >= 30 with Wilson intervals |
Eight fails, six no_evidence, the rest passing. The passes are worth a sentence, because a leaderboard would not have shown them either: zero fabrications on the negative controls, zero JSON parse failures in 45 structured-output runs, zero human interventions in 122 runs. The model had strengths. It was also slow, it could not hold the tool protocol under this harness, and it regressed on the suite the incumbent already aced. Any one of those is disqualifying for the lanes it would have served.
How the thresholds were set
Every relative threshold is anchored to the incumbent, and the document records why each number is what it is.
Q1 requires the candidate to match or beat the incumbent, score at least 0.90 absolute, and carry a Wilson 95 percent lower bound of at least 0.80, at n >= 35. The 0.90 allows three misses in 35 and still refuses a coin flip. The Wilson floor stops a lucky 32 of 35 from reading as proof.
Q2 requires the candidate to beat the incumbent by 10 percentage points on the full agentic suite and to reach 0.60 absolute. The published gap between the two Qwen models was 20.3 points, measured on BF16 with an unknown scaffold, so the gate asks for half of it to survive FP8 quantization, my scaffold and my tasks. The 0.60 floor exists because a jump from 0.20 to 0.30 is not a usable coding model regardless of the delta.
S1 asks for 25.0 tokens per second single stream, which is 79 percent of the incumbent's measured 31.6, with a hard floor of 20.0. The arithmetic behind it is a 12-turn agentic task at roughly 14,400 output tokens: 7.6 minutes at the incumbent's speed, 9.6 at 25 tok/s, 16 at 15 tok/s. The last of those changes what you are willing to attempt.
S2 asks for p50 within 1.25x and p95 within 1.5x of the incumbent at an 8k-token prompt, with backstop ceilings of 3.0 and 8.0 seconds in case the incumbent turned out to be unusually slow. 8k is the approximate size of the Codex CLI system prompt, so every turn pays that prefill. S3 asks for p95 end-to-end latency within 1.5x, percentile and never mean, because the tail is what makes an agent loop unusable. S4 states the quality-versus-speed exchange rate as a formula, success rate over median wall-clock minutes, so the trade is arguable rather than vibed.
R1's 2 percent comes from arithmetic too: at roughly 15 tool calls per agentic task, 2 percent per call is already a 26 percent chance of at least one malformed call per task. The cost gate uses cost per success on a calendar-amortization basis that includes hardware carry, energy and operator time, because cost per attempt flatters an unreliable model, and E4 demands n >= 30 on every rate gate because below that a percentage point tells you nothing.
Wilson intervals at small n
A point estimate of 0.889 on 45 runs sounds like a near miss against 0.90. The interval says the true rate is plausibly anywhere from 0.765 to 0.952, and the gate is on the lower bound, so the candidate would fail even if the 0.90 rule were dropped. The Qwen candidate's 45 of 45 in round 2 has a lower bound of 0.921, which clears 0.80 with room.
The interval also governs what the report is allowed to call provisional. The Qwen candidate's round-2 Q2 verdict was a pass at 12 of 18 on the solvable subset, 0.667 against the 0.60 floor, with an interval of 0.437 to 0.837. The subset had n = 18, below E4's 30, so the verdict was labelled provisional pending expansion. When the expansion landed, the rate was 12 of 39, 0.308. The interval had been honest about how little 18 runs knew.
The evidence-quality gates make this mechanical rather than a matter of taste. E4 requires n >= 30 for any rate gate and a Wilson interval beside every rate. E2 refuses any gate whose only non-deterministic judgment came from a contestant model. E3 refuses tasks with high leakage risk or a contaminated canary. E1 refuses any run without complete provenance, because a run that cannot be reproduced cannot support a decision.
Method caveats the report carries
The report flags its own weak cells.
The 69x TTFT figure compares p50 over the mixed-size prompts of the matrix run, not the 8k probe the gate specifies. Both arms were measured the same way, so the ratio is like for like, but it is not the gate as written. The method-correct 8k probe was run for the Qwen candidate and the incumbent in round 2 (952 ms incumbent p50, 1,885 ms candidate) and never for gpt-oss-120b.
X1 for gpt-oss-120b rests on the B-04 retention component only; the dedicated 64k suite A sweep ran for the Qwen candidate (45 of 45, zero drop) and not for this arm. X2's needle-retrieval half was not run for this arm either. For the Qwen candidate it scored 30 of 30 at roughly 116k tokens, which sharpened the finding: that candidate reads 128k fine and loses the tool protocol over accumulated agentic context.
Q2's solvable-subset definition is post hoc, recorded as such under change control on 2026-08-06, and the round-1 rows for the Qwen candidate's suite B were later annotated as measured on unreproducible serving. They stand in the journals unmodified under the immutability rule, and they are not gate evidence for any future decision. The provenance code had a defect found during the drift investigation: it recorded the image the arm's config claimed, not the image the container actually ran. Flagged for a fix; the WS5 function was never changed, and the continuous-eval pipeline replaced it with an attestation of the running container. Noted here because E1 was not as strong in round 1 as it read.
The one that got close
Table 2 is the Qwen3-Coder-Next-FP8 picture across the evaluation.
| Stage | Q1 suite A | Q2 suite B | Speed | Disposition |
|---|---|---|---|---|
| Round 1 | 41/45 = 0.911, fail (Wilson lower 0.793) | 12/27 = 0.444 full suite, fail against the 0.60 absolute floor | S1 48.8 tok/s pass; S2 fail on mixed prompts | keep incumbent |
| Round 2 | 45/45, pass after the arm-neutral regrade | 12/18 = 0.667 on the frozen subset, provisional pass at n < 30 | S2 1.98x at 8k, waived; S3 0.96x pass; S4 11.25x | keep incumbent; X2 fail, soak in progress |
| Expansion, 56/56 runs | unchanged | 12/39 = 0.308 on the same subset, hard fail at n >= 30; Q4, R4 and $1 flipped to fail with it | post-drift decode 30.0 tok/s, below the incumbent | not promoted, 2026-08-07 |
The expansion result was not noise. Transcripts of the same agentic task were byte-identical between a round-1 pass and an expansion fail through turn 5, then diverged at the assistant's sixth turn under temperature 0 and a fixed seed. Replaying the recorded turn-6 request against the live instance was deterministic and reproduced the expansion output. A reboot did not restore round-1 behavior; a fresh launch on the stock NGC image did not either; spot-hashed weights matched the source. The container that produced round 1 was gone. The decision of record treats the reproducible serving as the system under test and carries two unknowns: the drift is characterized, not explained, and the incumbent's own stability evidence is single-instance.
What re-opens the thread
The thresholds document says gates are bound to a specific artifact and serving stack, and they do not expire on a schedule. They expire when the thing they measured changes. The standing triggers are a new quantization artifact, a new NGC image or vLLM version, any runtime flag change, a context-length stage change, or a Codex CLI major version. The close-out added two, a serving stack that ships a GB10 fused-MoE kernel configuration, the leading mechanism candidate for the drift, and an output-equivalence launch gate that can pin an instance's behavior before evidence is collected against it, and restated the Codex-lane trigger: a toolchain shift that makes the responses-API gap load-bearing, since the incumbent's serving stack at close could not serve the current Codex CLI. The serving refresh of 2026-09-04 fired two of the standing triggers, a new vLLM image and new runtime flags, and the incumbent was re-baselined under change control that morning: suite A 45 of 45, tool probe 40 of 40, 29.9 tokens per second decode at a 4k fill.
"It has been a quarter" is no reason to re-run a 460-run matrix; a changed artifact is.
From a report to a pipeline
The evaluation rounds cost a bespoke multi-hour chain each, with a hand-written plan, and three things still depended on someone noticing: that a new model or serving image existed, that the live configuration still equalled what was accepted, and that the accepted configuration was recorded in one place. "Accepted" itself was spread across a runbook, the thresholds document and a running container.
The continuous-evaluation spec, approved 2026-09-04 after a Codex review whose 18 findings were all applied, turned the report into a state machine. One declarative manifest per box is the single source of truth for what was accepted; drift, discovery, trial and promotion are all diffs against it. The shape:
- A daily watcher reads model hubs, NGC vLLM tags, and grammar-library and driver releases, applies deterministic filters (fits 128 GB unified memory at the tier's context floor, license permits local personal use, a served quantization exists), and issues the first ask: run a trial on this candidate, yes, no or defer. No language model reads upstream text; release notes are data, never instruction.
- A nightly drift check diffs a read-only state document from each box against its manifest. Digest only, no remediation.
- A yes on the first ask leases the second Spark and runs a fixed ladder: smoke, a tool-reliability probe at n >= 40, suite A at 45 rows against a contemporaneous incumbent control in the same daytime window, and a single-stream speed ladder at two fill sizes. Findings are auto-drafted from structured evidence, passed through an advisory Codex review, and delivered with the second ask: start a side-by-side?
- A yes on the second ask runs up to seven nights of side-by-side on replay-safe corpora only, with a pre-registered stopping rule: two disagreement nights is a fail, seven calendar nights with at least five valid and at most one disagreement is a pass, anything else is inconclusive.
- A pass opens a promotion pull request that sets the manifest's status to proposed. Merging changes nothing live. Applying to the production Spark is manual, and the status becomes applied only when the drift check reads MATCH.
The gate contracts are now machine-readable JSON per role tier with a pre_registered flag; only a pre-registered contract can drive a trial, and the human-readable mirror was written on 2026-09-05 before any trial evidence existed. Every gate input must be admitted before it is scored: right envelope, right run group, image digest equal to the attested image, roster complete. Anything else is inconclusive, and so is running over budget. The default verdict is keep_incumbent, and no_evidence is never pass.
The first trial was the incumbent against itself, on 2026-09-06, to measure what a trial costs before registering a budget. Identity matched on image, model, served name, context length, vLLM version and normalized flags. Q1 was 45 of 45 in both arms, Wilson lower bound 0.9213. Decode was 25.3 against 25.6 tokens per second at a 16k fill and 29.3 against 29.6 at 4k; p95 latency 4,554 against 4,496 ms; total wall 1,228 seconds including a 540-second cold start; zero review findings. The registered wall budget for a trial is now 3,830 seconds, cleanup reserve included, derived from that measurement rather than guessed. The verdict has its own name, self_trial_complete, because keep_incumbent would have been true and uninformative.
The second instance: a scanner with a 6 percent accept rate
repo-gardener is a local, read-only code-quality scanner: deterministic repository analysis, local-model reasoning on the incumbent Qwen model, and an evaluator that accepts or rejects each finding. On 2026-08-09 its portfolio baseline was a 6 percent evaluator accept rate. The charter written that day sets the objective as useful novel findings per scan, not total findings; says every change should improve at least one of precision, calibration, measurement integrity or reproducibility; treats the benchmark as a scientific experiment with immutable prompt hashes and commit-pinned arms; and defers wall-clock optimization until precision improves.
The precision experiment reads as the same discipline with different nouns. Two arms scan the same pinned commit with the same model and budgets, differing only in the unit-analysis prompt version. The gate is pre-registered: below roughly 20 percent accept, still too noisy; 20 to 40, material; above 40, expand. The comparison tool refuses to produce a verdict when the arms differ on anything other than the prompt, exiting with COMPARABILITY: FAIL. Table 3 is the run sequence.
| Run | Arms | Findings | Accepted | Accept rate | Gate |
|---|---|---|---|---|---|
| 1 | v1 vs v3 | 16 vs 0 | 1 vs 0 | 6.2% vs undefined | not passed: v3 over-suppresses, and the one real finding is gone |
| 2 | v1 vs v4 | 16 vs 2 | 1 vs 0 | 6.2% vs 0% | not passed: recall recovered, the true finding rejected as miscalibrated at 0.95 confidence, n = 2 |
| 3 | v1 vs v5 | 17 vs 3 | 1 vs 1 | 5.9% vs 33.3% | PASS on one repo, "mechanically, not statistically" |
| 3 extension | v1 vs v5 on two more repos | 8 vs 8; 4 vs 5 | 0 vs 0 | 0% | FAIL both; pooled v1 1/29, v5 1/16 |
| 5 | v1 vs v5, three repos, pooled | 28 vs 7 | 1 vs 1 | 3.6% vs 14.3% | v1 FAIL; v5 INSUFFICIENT, 7 decided findings below the 10 minimum |
| 6 | v5 vs v6, two repos, pooled | 32 vs 25 | 10 vs 9 | 31.2% vs 36.0% | both PASS; scored under the recalibrated v3 judge with clone detection live, not comparable to runs 1 to 5 |
Three things in that table are the gate doing its job rather than the scanner. Run 1's zero findings passed every secondary check and failed the primary one, because a scanner that goes silent has not become precise. Run 5's INSUFFICIENT verdict exists because the gate was rebuilt to say that seven decided findings is not evidence, which is E4 in another repo. And the v1 arm reproduced its 6 percent baseline across three runs and two tool commits: the measurement is stable, which everything else depends on.
The retraction is the part I would show anyone who thinks the gate is overhead. An injected-defect corpus reported recall of 0.75, then 0.875. A later instrument caught the ground-truth manifest being packed into the root analysis unit, where the model read it: twelve findings in one corpus came from that unit, citing defect files it did not contain. Both figures were retracted in the same document. The first honest measurement was 0.125, and its decomposition was the real finding: same-unit defects 2 of 8, cross-unit defects 0 of 8, a recall ceiling built into per-unit batching that no prompt could see. Human labelling later put true precision at 20 percent for v1 and 60 percent for v5 on a small sample; the evaluator's agreement with the human, Cohen's kappa 0.365 before recalibration and 0.658 after, is marked in-sample and not to be cited as validated until a held-out round.
References
All ten are in private repositories; each document is quoted where used in the text.
- spark-trials,
docs/ws5-acceptance-thresholds.md. Written 2026-08-04; change control, waivers and the eval close are recorded in the same file. - spark-trials,
docs/ws5-eval-spec.md. Two-layer design, three suites, cost model. - spark-trials,
docs/GATE-REPORT-round-1.mdanddocs/GATE-REPORT-round-2.md. - spark-trials,
docs/RESULTS-round-2.md. Round-2 deltas and errata. - spark-trials,
docs/FINDINGS-instance-drift-2026-08-07.md. - spark-trials,
docs/MODEL-SELECTION-CLOSE-2026-08-08.md. Decision of record and re-open triggers. - spark-trials,
docs/SPEC-continuous-eval-2026-09-04.md,docs/tier-gates.md,docs/PLAN-continuous-eval-stage2-2026-09-05.md,docs/PLAN-continuous-eval-stage3-2026-09-06.md. - spark-trials,
docs/FINDINGS-self-trial-2026-09-06.md. - repo-gardener,
docs/PRIORITIES.md. Charter, 2026-08-09. - repo-gardener,
docs/EXPERIMENT-PRECISION-BASELINE.md.