Two models on the same box, the same day, the same harness. One decoded at 106.9 tokens a second. The other decoded at 29.8. Both are mixture-of-experts models quantized to 4-bit weights, both activate about three to four billion parameters per token, and both ran on kernels I had checked were the fast path. The gap is 3.6x, and no single fact I measured explains all of it.
I ran the round on 2026-08-31 to test a claim I kept reading in the DGX Spark community and had not seen measured under control: that small-active-parameter MoE models at 4-bit weights with speculative decoding are decisively the fastest class on a single GB10. A mixture-of-experts (MoE) model splits its feed-forward layers into many expert blocks and routes each token through only a few, so the model is large on disk but does a small model's worth of arithmetic per token. Active parameters are the ones actually read to produce one token; for the models here that is about 3 to 4 billion out of 26 to 35 billion total. NVFP4 is NVIDIA's 4-bit floating-point weight format, roughly half a byte per parameter, executed by dedicated FP4 kernels on Blackwell-class GPUs. Multi-token prediction (MTP) is a form of speculative decoding in which a small extra head drafts several next tokens and the main model verifies them in one pass, so one read of the weights can yield more than one accepted token.
The theory is plausible because a GB10 decodes at the speed it can read memory. The community-measured sustained bandwidth is about 220 GB/s. Reading three billion parameters at half a byte each is 1.5 to 2 GB per token, which caps a non-speculative decoder somewhere near 110 to 146 tokens a second. Fewer active parameters, fewer bytes per parameter, and more than one token per read are the three levers that move that ceiling.
The reason to measure rather than believe: the public figures for one of these checkpoints on this hardware spanned about 2.5x across sources, 86, 101 and 218 tokens a second, and most published Spark results are one machine, one operator, one run.
What I ran
All of it ran in one day on the second of my two Sparks, with production on the first untouched throughout. Four models. Qwen3.6-35B-A3B in NVFP4 with MTP drafting three tokens. Ornith 1.5 35B-A3B in NVFP4 with MTP drafting one. Qwen3-Coder-Next, an 80B MoE in FP8 with no speculative decoding. Gemma 4 26B-A4B in NVFP4 with no speculative decoding. Each ran a context ladder of 4K, 32K, 50K and 100K prompt tokens single-stream, a concurrency curve at 4K, and a tool-calling probe. Every cell was one discarded warm-up plus five measured runs, medians reported with ranges. Every model ran the full ladder twice with a fresh server restart between passes. 441 result rows (370 measured, 71 discarded warm-ups), zero request errors.
The verdict, on speed
On speed, the claim held for the four recipes I ran. The two 4-bit MoE models with MTP beat the FP8 80B MoE by 2.1x to 3.2x single-stream, and beat the 4-bit MoE without MTP by 2.8x to 3.8x, at every context length on both passes. Qwen3.6 decoded at 106.9 tokens a second at 4K and 91.7 at 100K; Ornith at 89.3 and 69.3; Qwen3-Coder-Next at 42.6 and 29.1; Gemma 4 at 29.8 and 24.7. Fifteen of the sixteen single-stream cells reproduced within plus or minus 2.6 percent across the restart. The sixteenth, Qwen3.6 at 100K, dropped 11 percent on the second pass with non-overlapping ranges. I flag it; I do not explain it.
MTP was the largest single lever I measured. On Qwen3.6 at 4K, turning it off dropped decode from 106.9 to 77.3 tokens a second on the benchmark's filler prompts, and from 102.9 to 78.1 on natural prose: a 32 percent gain in the natural-text cell. Those are medians of five sequential runs in one cell of one model. They support a range, not a confidence interval, and I would not generalize them without interleaved trials. They also do not explain why Gemma 4, with no MTP, decoded at 29.8 while Qwen3.6 with MTP switched off decoded at 77.3. Same weight format, similar active size, same box, both on the fast kernel path. That residual is the open question in this dataset.
The amendment, and its correction
The amendment came out of the day and got corrected two days later. "Engine" here means the serving stack: the vLLM build, its kernels, and its launch flags. Four things happened on the day that had nothing to do with model class. The stock vLLM container failed to load these MoE NVFP4 checkpoints at all, with a missing scale-tensor error, per the upstream runbook I was replicating; every SparkBench golden number for them runs on a community-built image. Older vLLM builds load the same weights and then run them in emulation at a reported one token a second, so every launch in my round checked the server log for the FP4 kernel dispatch before a single measurement counted. Gemma 4's public "with MTP" figure of 54.9 tokens a second, from an issue thread on the community image's repository, did not appear on the NVIDIA checkpoint the way I launched it: the checkpoint ships no MTP head and vLLM says so when asked. A public external drafter exists; with it I later measured 47.9 on 2026-09-03, still short of 54.9. And Gemma 4's public tool-calling failure was a missing parser flag; with the right parser named, it passed 5 of 5 on both days.
Then the correction. On the day, I noted that Qwen3-Coder-Next measured 42 percent faster on the new image than the 30.0 I had recorded for the same weights a month earlier, and wrote it up as evidence that the engine matters as much as the class. That was an association across different days, harnesses and prompts, and I labelled it that way. That earlier round had already measured 42.8 on the old engine before an instance-drift event dropped it to 30.0. Two days later I ran the controlled version: same box, same day, same prompts, prefix cache off, old image against new, five runs each with ranges under 0.4 tokens a second. The old image was faster, 47.4 against 42.3 tokens a second at 4K. The 42 percent claim is withdrawn. What survives is narrower: the engine decides whether you get the class's speed at all, and two of the four items above are upstream reports, not things I demonstrated. It does not follow that a newer engine is a faster engine.
The caveats, in plain words
This was a throughput round. It says nothing about output quality, and the quality round that followed two days later says the opposite of what the speed suggests: under gates I had pre-registered after the speed numbers were in but before any quality run, none of the fast models passed, and my slower incumbent stayed in every lane.
My numbers do not compare to the public SparkBench figures I set out to replicate. Their metric puts prompt processing and queueing time in the denominator of a multi-turn session; mine measures decode alone. Run under their definition on my own serving, Qwen3.6 reads 97 tokens a second at 4K and 14.5 at 100K, not 106.9 and 91.7. The "long-context cliff" I thought I had failed to reproduce was two metrics being compared, and the comparison should not have been made.
The kernel check classified every row's launch log as the FP4 fast path and found no emulation marker on any launch; the FP8 model has no FP4 path, so its label is nominal. The check does not certify that the kernels were the best available, and each recipe chose its own MoE backend, so the cross-model comparison inherits those choices. It also does not make the serving images trustworthy: both come from personal GitHub accounts, not NVIDIA's registry, and a production lane would need re-serving on a vetted image.
The benchmark's fill prompts are repetitive template sentences over a small vocabulary, which speculative decoding likes. A natural-prose control put the inflation from that at about 4 percent, and Ornith was faster on natural text than on filler. Power and temperature are as the GPU exporter reported them, not wall-calibrated.
What it changes for one box
If you have one GB10-class box and want the fastest local decode, start the shortlist where these four recipes point: a MoE with a few billion active parameters, 4-bit weights, and a draft head the checkpoint ships or a public drafter the recipe names. Then choose the engine that runs that checkpoint's kernels, and read the server log to prove it did, because the failure mode is silent and about a hundred times slower. Expect around 100 tokens a second single-stream at 4K on the best of these, and, on the hybrid-attention models, decode that drops 14 to 23 percent from 4K to 100K tokens of context.
Do not promote on speed. Run your own tasks through the model first, with the output budget and the response format your consumer actually needs. A model that decodes at 107 tokens a second and cannot hold a JSON contract is a fast way to make work for yourself.
The model matrix
Four models, four recipes, two serving images. Sizes on disk are the checkpoint downloads as staged; active-parameter counts follow the vendors' naming (A3B, A4B) and are approximate.
| Model | Total / active params | Weights | On disk | Speculative decoding | Serving image | Notes |
|---|---|---|---|---|---|---|
| Qwen3.6-35B-A3B (NVIDIA NVFP4 pack) | 35B / ~3B | NVFP4 | ~23.5 GB | MTP, 3 draft tokens | Community vLLM image, wheel commit 7a9993878, FlashInfer 083012d6 |
MoE backend marlin, MTP path triton; FP8 KV cache; gpu-mem 0.65; qwen3_coder tool parser |
| Ornith 1.5 35B-A3B (vendor NVFP4 pack) | 35B / ~3B | NVFP4 | ~20 to 25 GB | MTP, 1 draft token | Second community image (W4A16 overlays) plus two bind-mounted kernel mod files | MoE backend b12x; gpu-mem 0.85; max 24 sequences; qwen3_xml tool parser |
| Qwen3-Coder-Next (FP8) | 80B / ~3B (per its base model's naming) | FP8 | 75 GB | none | Community vLLM image, same wheels | FlashInfer attention; gpu-mem 0.85; the parked candidate from my 2026-08-08 close |
| Gemma 4 26B-A4B (NVIDIA NVFP4 pack) | 26B / ~4B | NVFP4 | ~17 GB | none (checkpoint ships no MTP head; public external drafter tested later, 2026-09-03) | Community vLLM image, same wheels | gemma4 tool parser added; gpu-mem 0.7 |
All three NVFP4 checkpoints were pinned to a Hugging Face revision and carry no custom Python; the policy was to launch without --trust-remote-code and add it per model only if a load failed, and no load failure was recorded. The Ornith vendor is new to me; its pack is MIT-licensed with no custom code, and its quality claims (SWE-bench Verified 79.0) are vendor numbers I have not measured. Both serving images are community builds: the first from a reviewable Dockerfile plus prebuilt wheels, the second an opaque pulled image pinned by digest. More on that under the supply-chain flag below.
Measurement protocol
The harness is my own, written for this round. The discipline below was frozen in the plan that morning; the decode formula as written here first appears in the mid-round review.
- Decode tokens per second is
(completion_tokens - 1) / (t_last - t_first)on the streamed response. Time to first token is recorded separately and is not in the decode denominator. - 400 output tokens requested. Sampling as the recipes ship it: temperature 0.6, top_p 0.95, and top_k 20 on Qwen3.6. That lowers speculative acceptance versus greedy, so it cannot be the reason the numbers are high.
- Fill targets 4,096, 32,768, 50,000 and 100,000 prompt tokens; the billed prompt-token count was verified against the target for every run.
- One warm-up request per configuration, discarded. Five measured runs per cell. Median plus min and max.
- Fresh server restart between materially different configurations, with cold-start time recorded. The entire ladder ran twice (labelled day 1 and day 2 in the data, both on 2026-08-31) with a fresh restart between passes.
- Kernel path recorded per row as
cutlass-fp4orfallback; a run onfallbackis invalid by rule. - Power, temperature and memory joined from Prometheus range queries against the existing GPU and node exporters, because
nvidia-smireturns N/A for memory and power on GB10. - Prefix caching on in every cell, including the natural-prose control, because the upstream recipes have it on and the public methodology assumes it. Prompts are seed-distinct; the shared prefix is under ten tokens.
The plan I wrote that morning said "prefix caching disabled everywhere", and my recipe document defines the cross-model control cells as cache-off. Every row that day ran with it on, the control included, and the state is recorded per row.
Single-stream decode, per cell
Median decode tokens per second with the min to max range in parentheses, both passes, and the day-2 change against day-1. The 4K rows for the two MTP models are the recipe's MTP setting (3 for Qwen3.6, 1 for Ornith).
| Model | Context | Day 1 median (range) | Day 2 median (range) | Day 2 vs day 1 |
|---|---|---|---|---|
| Qwen3.6-35B-A3B NVFP4, MTP 3 | 4K | 106.9 (104.7 to 118.2) | 105.6 (97.4 to 136.8) | -1.2% |
| 32K | 98.6 (96.0 to 106.3) | 100.8 (91.7 to 113.5) | +2.2% | |
| 50K | 92.9 (89.1 to 96.2) | 94.8 (93.5 to 113.6) | +2.0% | |
| 100K | 91.7 (82.2 to 102.5) | 81.4 (79.5 to 82.1) | -11.2% | |
| Ornith 1.5 35B-A3B NVFP4, MTP 1 | 4K | 89.3 (85.1 to 90.5) | 88.2 (86.6 to 89.9) | -1.2% |
| 32K | 83.4 (79.3 to 83.9) | 81.5 (77.8 to 83.5) | -2.3% | |
| 50K | 76.9 (75.7 to 78.0) | 77.2 (76.4 to 80.4) | +0.4% | |
| 100K | 69.3 (67.3 to 72.4) | 71.1 (68.7 to 72.8) | +2.6% | |
| Qwen3-Coder-Next FP8, no MTP | 4K | 42.6 (42.6 to 42.8) | 42.1 (42.0 to 42.2) | -1.2% |
| 32K | 36.9 (36.8 to 37.0) | 36.3 (36.3 to 36.5) | -1.6% | |
| 50K | 34.6 (33.3 to 34.7) | 33.9 (33.9 to 34.0) | -2.0% | |
| 100K | 29.1 (29.0 to 29.1) | 28.5 (28.5 to 28.5) | -2.1% | |
| Gemma 4 26B-A4B NVFP4, no MTP | 4K | 29.8 (29.7 to 29.8) | 29.1 (29.1 to 29.3) | -2.3% |
| 32K | 27.0 (26.9 to 27.0) | 26.5 (26.5 to 26.6) | -1.9% | |
| 50K | 26.3 (25.8 to 26.3) | 25.9 (25.4 to 25.9) | -1.5% | |
| 100K | 24.7 (24.7 to 24.8) | 24.4 (24.3 to 24.5) | -1.2% |
Two shapes are visible without statistics. The non-speculative models have ranges of a few tenths of a token per second; Qwen3.6 with three draft tokens has ranges up to forty; Ornith with one, two to six. Speculative decoding trades a fixed cost per token for an acceptance rate, and acceptance varies with the text. The Qwen3.6 100K day-2 drop is the one cell where the ranges do not overlap (82.2 to 102.5 against 79.5 to 82.1). Restart-to-restart MTP variance at extreme fill is my label for it, not my explanation.
Cross-model ratios from the same cells, recomputed from the summary CSV for this article: Qwen3.6 over Qwen3-Coder-Next runs 2.5x at 4K to 3.2x at 100K (day 1) and 2.9x (day 2); Ornith over Qwen3-Coder-Next 2.1x to 2.5x; Qwen3.6 over Gemma 4 3.3x to 3.8x; Ornith over Gemma 4 2.8x to 3.1x. My findings document of the day quoted 2.1 to 3.1x and 3.0 to 3.7x; the wider ranges here include the day-2 cells and the Ornith 50K and 100K rows against Gemma.
Time to first token tracks fill and is the number a long-context user actually feels. At 4K it ran 630 to 690 ms for the three fast models and about 1.1 s for Qwen3-Coder-Next. At 100K it was 30.1 s for Qwen3.6, 31.4 s for Ornith, 31.9 s for Qwen3-Coder-Next and 99.1 s for Gemma 4, whose prefill rate at 100K (about 1,020 tokens a second) was a third of the others' (3,150 to 3,350). Qwen3.6, Qwen3-Coder-Next and Gemma 4 ran on vLLM main at commit 7a9993878 (2026-08-26, since shipped in v0.29.0); Ornith ran on the second image. v0.30.0 moved GB10's prefill for linear-attention layers to a FlashInfer kernel, so newer builds may start faster. Peak memory per model (85 to 88 GiB for Qwen3.6, 113 to 114 for Ornith, 109 to 111 for Qwen3-Coder-Next, 95 to 96 for Gemma 4) reflects each recipe's GPU-memory-utilization setting more than the model's footprint.
Aggregate throughput at concurrency
Aggregate decode tokens per second at 4K fill, day-1 medians; the single-stream column repeats the table above for scale.
| Model | 1 stream | 2 | 4 | 8 | 16 |
|---|---|---|---|---|---|
| Qwen3.6-35B-A3B NVFP4, MTP 3 | 106.9 | 132.8 | 195.2 | 193.2 | 189.8 |
| Ornith 1.5 35B-A3B NVFP4, MTP 1 | 89.3 | 108.1 | 145.2 | 199.7 | 247.0 |
| Qwen3-Coder-Next FP8 | 42.6 | 55.9 | 73.5 | 73.8 | 73.6 |
| Gemma 4 26B-A4B NVFP4 | 29.8 | 57.3 | 92.8 | 93.1 | 93.2 |
The Qwen3.6 recipe caps the scheduler at 4 sequences, which is why its curve is flat from 4 streams. The Ornith recipe allows 24, and its curve was still climbing at 16. These cells measure each recipe's serving shape, not what either model could do under a different scheduler setting. Day-2 aggregates were within a few percent of day-1 everywhere.
The MTP sweep
Qwen3.6 at 4K, single-stream, varying the number of draft tokens, filler prompts, plus the natural-prose control (C1) run later the same day.
| Draft tokens | Prompt style | Median tok/s | Range |
|---|---|---|---|
| 0 (MTP off) | filler | 77.3 | 76.5 to 77.3 |
| 1 | filler | 98.4 | 94.2 to 100.7 |
| 2 | filler | 107.5 | 104.6 to 127.0 |
| 3 (recipe) | filler | 106.9 | 104.7 to 118.2 |
| 4 | filler | 111.8 | 97.2 to 133.0 |
| 0 (MTP off) | natural prose | 78.1 | 78.0 to 78.6 |
| 3 (recipe) | natural prose | 102.9 | 93.0 to 111.7 |
The plateau is at two to three draft tokens. The n=0 arms match the bandwidth model: 77 to 78 tokens a second is 50 to 70 percent of the 110 to 146 ceiling for a 3B-active NVFP4 decoder on 220 GB/s, which is a normal efficiency. The n=3 arms exceed the single-token ceiling, which is exactly what speculative decoding is for: several tokens verified per weight pass. Filler inflation, measured as the natural-versus-filler difference at n=3, is about 4 percent. Ornith's natural-prose cell at its recipe setting (one draft token) was 95.9 against 89.3 on filler, so for that model the filler prompts were the pessimistic case. Arms were sequential, not interleaved; n=5 supports medians and ranges only.
Kernel-path validation
The plan pre-registered a per-launch gate: the server log must show the FlashInfer CUTLASS FP4 dispatch for the quantized layers, or the launch is invalid and nothing from it is recorded as a result. The trigger for the rule is a documented trap in this stack: an older vLLM loads NVFP4 weights, misses the kernel-selection path, and runs the model in emulation at a reported 1.1 tokens a second. The separate failure, the stock vllm/vllm-openai container refusing the MoE NVFP4 checkpoints with a KeyError on an expert input-scale tensor, is loud and self-diagnosing; the emulation trap is silent, and a benchmark that does not check for it will publish numbers a hundred times too low without knowing.
Result: every one of the 441 rows (370 measured, 71 warm-ups) carries kernel_path: cutlass-fp4; zero carry fallback. I recounted that from the raw JSONL while writing this article rather than quoting the findings document. Two limits on that count. The classifier labels a launch by the first matching substring in its log, and the FP8 model's label comes from a backend line that lists the CUTLASS option among its candidates; that model has no FP4 path, so its label is nominal. And only the evidence lines the classifier kept travel with the rows, not the full server logs.
The clean count does not establish that the kernels dispatched were the best available for each model: the Qwen3.6 recipe uses a marlin MoE backend with a triton path for the MTP draft, the Ornith recipe uses a vendor-specific b12x backend from the second image with two mounted kernel modification files, and Gemma 4 and Qwen3-Coder-Next each use their own recipe's defaults. The cross-model comparison is therefore recipe-against-recipe, each on its own validated fast path, not model-against-model on one kernel, and the 3.6x should not be read as an architecture-only result.
Reproducibility across the restart
Fifteen of sixteen single-stream cells moved by 2.6 percent or less between the two passes (the deltas are in the per-cell table above). The non-speculative models moved 1.2 to 2.3 percent, all downward, which is small enough that I am not going to build a story on it. The MTP models moved in both directions. The single outlier is Qwen3.6 at 100K.
Both passes are the same date, operator, image and weights: restart reproducibility, not cross-day or cross-machine. The planned day-3 reruns did not run before the quality round took the box.
Tool calling
The probe is a fixed nine-tool set run five times per session, scored as a rate rather than one-shot, because the public one-shot tests for these models flip between pass and fail on exactly this.
- Qwen3.6: 37 of 40 valid across eight sessions on the day (92.5 percent); five sessions 5 of 5, three sessions 4 of 5. All three failures were in MTP-enabled sessions (27 of 30 with MTP on, 10 of 10 with MTP off). I flagged that as a possible MTP interaction with tool-call formatting. In the quality round two days later, paired probes at n=40 gave 38 of 40 with MTP on and 39 of 40 off, and the no-thinking configuration gave 40 of 40 either way. The pattern did not survive; I withdraw the flag.
- Ornith: 15 of 15 across three sessions on the day, 40 of 40 in the later paired probes.
- Gemma 4: 5 of 5 on both days once the
gemma4tool-call parser was named on the launch line. The public dataset'stool_ok: falsefor this model came from a recipe with no parser. - Qwen3-Coder-Next: 5 of 5.
A second-model factual review caught an arithmetic error in my first count the same day, along with nine other findings I accepted; 37 of 40 is the corrected enumeration.
Against the two public datasets
I set out to replicate two public GB10 datasets whose methodologies do not agree with each other: the SparkBench context ladder and the dgx-spark-bench concurrency harness (my notes call it Field Notes). What I learned was mostly about the methodologies.
SparkBench's golden cells recorded, for Qwen3.6 (vLLM 1ea84d74b, 2026-08-04), 86.3, 78.8 and 31.5 tokens a second at 4K, 50K and 100K fill, and for Ornith (2026-08-20) 70.8, 36.5 and 27.2. My decode-only cells on wheel 7a9993878 read 106.9, 92.9 and 91.7, and 89.3, 76.9 and 69.3. On the day I wrote that my harness "does not reproduce the long-context cliff" and offered two explanations: engine improvement in the hybrid linear-attention and mamba cache path, or a definitional mismatch. The interim critical review rated the mismatch a live threat and asked for a control. Source reading of the vendored harness that evening, and an emulation of its metric run on my own serving two days later, settled it: their per-rung number is a multi-turn agent session, non-streaming, with prefill and queueing wall-clock in the denominator and a characters-divided-by-four fill estimate. Under that definition on my instance, Qwen3.6 reads 97.0, 30.4 and 14.5, and Ornith 73.0, 18.6 and 8.1. The cliff is the metric: at 100K the 30-second prefill dominates any session-level rate. The engine-improvement explanation is no longer needed; I have not excluded it, and the 100K session's tool round-trip failed once in a single session, so that emulation is itself n=1.
The rule I adopted: decode-only numbers and session-rate numbers never appear in the same comparison, and every row in my data is labelled with which it is.
The dgx-spark-bench reconciliation turned out to be about attribution as much as numbers. The Gemma 4 figure of 54.9 tokens a second single-stream "with MTP" that my notes on the day credited to that harness is not in its repository; it comes from an issue thread on the community image's repository. It did not appear on the NVIDIA NVFP4 checkpoint the way I launched it: the checkpoint ships no MTP head, and vLLM raises NotImplementedError when asked for MTP without a drafter. The community recipe uses a public external drafter model, google/gemma-4-26B-A4B-it-assistant, and that was the missing piece. With that drafter, on 2026-09-03, Gemma 4 decoded at a median 47.9 tokens a second at 4K over three scored runs: 61 percent over my 29.8, and 87 percent of the 54.9.
The engine correction, with numbers
On the day I had one clean-looking control: Qwen3-Coder-Next, same FP8 weights I had staged a month earlier, no speculative decoding, no new checkpoint. It measured 42.6 tokens a second at 4K on the new image against 30.0 recorded during the previous round on the older image. That round had already measured 42.8 on the old engine before an instance-drift event dropped it to 30.0. I called that +42 percent from the engine alone and made it the amendment to the theory, while labelling it an observed cross-run association with a control (C3) pending.
C3 ran on 2026-09-02: same box, same day, twenty minutes apart, prefix cache off, single-stream, identical prompts, five runs each. The older image (vLLM 0.24.0, the standby's pinned build) decoded at 47.4 tokens a second at 4K and 44.6 at 16K. The newer community image (its wheel reports 0.26.1rc1) decoded at 42.3 and 39.3. Ranges were under 0.4 tokens a second on every arm. The older engine is about 12 percent faster on this model. Fills above 16K could not be compared because the standby is pinned at a 32K model length.
So the +42 percent was a comparison between a cell from this round and a figure from the previous round, taken on a different day with a different method, and its sign was wrong. What the day left standing about engines is the list of four in the executive layer. Two I demonstrated: the NotImplementedError without a drafter, and the missing tool parser. Two are upstream reports I did not reproduce: the stock container's KeyError, and the emulation trap.
What the quality round did with the speed
Two days later the same two fast models ran against quality gates pre-registered on 2026-08-31, after the speed numbers were in but before any quality run. The gates were: a structured-output suite (Q1, threshold 0.90 and at least the incumbent's score), a frozen agentic subset (Q2, threshold 0.60), and native tool reliability (T1, 95 percent).
| Candidate | Q1 structured suite | Q2 frozen agentic subset | T1 native tools | Verdict |
|---|---|---|---|---|
| Qwen3.6, thinking, MTP 3 | 28/45 = 0.622 | 0.196 | 0.95 on / 0.975 off | fail |
| Qwen3.6, thinking off | 38/45 = 0.844 | 0.188 | 1.0 / 1.0 | fail |
| Ornith 1.5, MTP 1 | 30/45 = 0.667 | 0.417 | 1.0 / 1.0 | fail |
| Incumbent (30B-A3B, bf16, first Spark, live) | 45/45 = 1.000 | 4/49 = 0.082 on the full suite (subset not reported) | not configured | kept |
On the full agentic suite, every candidate beat the incumbent's 4 of 49 by 6 to 25 points, and none reached the 0.60 floor on the frozen subset. A post-verdict check found that all seven of the no-thinking Qwen3.6 structured-suite failures were a grader artifact (a verbatim-quote rule that the incumbent passes by echoing tab characters and a paraphrasing model fails by returning spaces); under a whitespace-normalized rule that arm would have been 45 of 45. It still fails Q2 and the variance gate, so the round's decision stands, and I did not re-grade after the fact. The incumbent's perfect structured-suite score should be read as "matches the grader it was built alongside", not as a quality ceiling.
Two operational findings from that round belong next to the speed numbers. Served with thinking on, these reasoning models could not drive my plain chat-completions consumer, which parses JSON from content: the reasoning tokens are invisible to it, and the thinking arm failed 76 units across my code-scanner corpus with malformed completions. Served with thinking off, both fast models completed the corpus in about 38 minutes with zero failed units, and each fabricated one citation past end-of-file, which my zero-tolerance class catches.
Power, thermal, cold start
Exporter-reported, not wall-calibrated: idle 13 W, peak 93 W under load, peak temperature 83 C. The box logged 424 seconds of ordinary software power-cap throttling across the day and zero hardware brake events. Cold starts from a fresh container to a serving endpoint: Qwen3.6 172 to 177 s, Ornith about 150 s on one pass and 332 s on the other, Qwen3-Coder-Next 267 s, Gemma 4 326 to 331 s.
Supply-chain flag
Both serving images come from personal GitHub accounts, not NVIDIA's container registry. The first is built on the box from a reviewable Dockerfile with prebuilt wheels from that account's GitHub releases; the second is an opaque image pulled on a moving tag and pinned by digest at pull time. The kernel modification files mounted into the second are small Python and were read before mounting. Mitigations I applied: review the Dockerfile and build scripts before building; record exact wheel commits and image digests at build and pull time; keep the serving containers LAN-only with bearer auth and no WAN egress at runtime; run them only on the benchmark box, never on the production Spark. Residual risk accepted for a benchmark box. Any promotion of a winner into a lane that other systems depend on requires re-serving on a vetted image, as a separate gated step.
What followed
The library the next round draws from was re-screened on 2026-09-06 with the class as the filter. Staged as Tier 1: NVIDIA's Nemotron 3.5 Lightning 30B-A3B in NVFP4, which ships an official DGX Spark recipe and a Spark-tuned 967M-parameter drafter (mean accepted length 3.75 at draft length 7), and the FP8 twin of the Qwen3.6-35B-A3B pack, to test whether 4-bit is actually faster than 8-bit on the same model, which my notes say is not universal on this hardware. Dense 27B to 31B models were ruled out of staging as bandwidth-bound; a third-party guide my notes cite puts them near 7 to 10 tokens a second on this box, a figure I have not measured. Lightning's external claim of about 420 aggregate tokens a second at 8 streams against my 184 to 193 for Qwen3.6 is a hypothesis for a matched run, not a result.
References
- spark-trials (private repository; the document is quoted where used),
docs/PLAN-ws6-nvfp4-mtp-2026-08-31.md: the pre-registered plan, experimental discipline, record schema, and gates. - spark-trials (private repository; the document is quoted where used),
docs/FINDINGS-ws6-nvfp4-mtp-2026-08-31.md: the day's verdict, headline table, findings against the public datasets, and binding caveats. - spark-trials (private repository; the document is quoted where used),
docs/CRITICAL-REVIEW-ws6-interim-2026-08-31.md: the mid-round threats-to-validity review (filler inflation, metric mismatch), the bandwidth sanity check, and the same-day post-review update. - spark-trials (private repository; the document is quoted where used),
docs/ws6-summary.csvandbaselines/ws6-*.jsonl: 71 per-cell summary rows and the 441 raw result rows, including the per-row kernel path. - spark-trials (private repository; the document is quoted where used),
docs/RECIPES-ws6.md: pinned checkpoint revisions, serving images and wheel commits, launch flags, upstream reference numbers, and the supply-chain flag. - spark-trials (private repository; the document is quoted where used),
docs/FINDINGS-ws7-2026-09-02.mdanddocs/GATE-REPORT-ws7-2026-09-02.md: the quality round, the C2 metric emulation, and the C3 engine A/B that withdraws the +42 percent claim. - spark-trials (private repository; the document is quoted where used),
docs/MODEL-SELECTION-CLOSE-2026-08-08.md: the standing recommendation this round re-opened, and the pre-registered re-open trigger. - spark-trials (private repository; the document is quoted where used),
docs/RESEARCH-library-candidates-2026-09-06.md: the Tier 1 candidate list that followed, with its own second-model review. - spark-trials (private repository; the document is quoted where used),
docs/RESEARCH-spark-gaps-2026-09-02.md: the correction that re-attributes the Gemma 4 54.9 figure to the community image's issue thread and names the external drafter. - spark-trials (private repository; the document is quoted where used),
docs/FINDINGS-ws8-diag-2026-09-03.md: the Gemma 4 plus external drafter measurement, 47.9 tokens a second at 4K. - SparkBench, the context-ladder dataset and recipes replicated here: https://github.com/shawnmarck/sparkbench
- dgx-spark-bench, the concurrency harness my notes call Field Notes; the Gemma 4 54.9 tokens a second figure is not in this repository: https://github.com/jvr0x/dgx-spark-bench
- spark-vllm-docker, the community vLLM image build for GB10; its issue thread #343 is the source of the Gemma 4 54.9 tokens a second figure, and its Gemma 4 recipe names the external drafter: https://github.com/eugr/spark-vllm-docker
- Checkpoints: https://huggingface.co/nvidia/Qwen3.6-35B-A3B-NVFP4, https://huggingface.co/ornith-ai/Ornith-1.5-35B-A3B-NVFP4, https://huggingface.co/nvidia/Gemma-4-26B-A4B-NVFP4