Twice in eight runs, the model on my GPU bumped a peer dependency, which cleared the install error CI had died on and surfaced a type error behind it. Then it wrote "status": "fixed" into its result file, summarized the fix in confident prose, and stopped. Both times the type checker still failed on the error the model itself had uncovered and then declared fixed. The cheapest hosted model I tested fixed the same two cases, and the other six, in about 48 seconds each, for 0.13 USD total.
The problem the benchmark grew out of
Earlier this summer I cleared a Dependabot backlog across my repositories and wrote up the lockfile contention that strands parallel merges (Why One Dependabot Merge Leaves the Other Five Behind). Serial merging solved the green PRs. It did nothing for the red ones: the bumps whose CI failed and sat there until a human read the log.
So I designed a fix lane for my agent platform. The idea is ordinary: replay the failing PR in a sandbox, hand a coding agent the failing log, let it patch, verify from outside, and hand a human a draft packet. The open question was the engine. I own a DGX Spark that serves a 30-billion-parameter open-weight coding model, and I had been assuming it would do this job for free. The design draft carried a key-injecting proxy, a client lock, a context budget and a scheduling constraint to make the local lane work. Before I built any of that, I wanted to know whether it earned its complexity.
What I ran
Eight real Dependabot pull requests from my own repositories, each of which had failed CI. Not synthetic bugs: the actual PR head, the actual failing job log, the actual verify command CI ran. The shapes were the ones every Node maintainer recognizes: a lockfile regenerated without an overrides block, an Express 5 route syntax change, a TypeScript major that removed a compiler option, a Tailwind 4 plugin move, two Vite peer-range conflicts, and two React 19 type migrations.
Seven arms ran every case: Codex CLI with three OpenAI models (GPT-6 Astra, gpt-5.6-luna, gpt-5.4-mini), Claude Code with three Anthropic models (Opus 5, Sonnet 5, Haiku 4.5), and Codex CLI pointed at the local Qwen3-Coder-30B lane. Fifty-six runs. Each one got the same prompt, the same rules (keep the bumps, do not touch tests or CI config, write a result file), a ten-minute cap, and a fresh container. The agent never scored itself. A separate container reinstalled dependencies from scratch and ran the exact CI command. Pass or fail came from that exit code and nothing else.
What happened
Every hosted arm fixed every case. Forty-eight of forty-eight. Frontier and budget models landed on the same fix families, which tells me the corpus is easy once the error message is read. That is an observation about these eight cases, not a claim about dependency repair in general.
The cheapest arm, gpt-5.6-luna through Codex CLI, passed eight of eight with a mean of 48 seconds per case and a total bill of 0.13 USD. The two frontier arms produced the same fixes for 3.29 and 3.40 USD, roughly 25 times the cost.
The local lane passed six of eight, with a mean of 454 seconds per case. Four of its eight runs ran until the 600-second cap killed them, and in all four the fix was already in place; the model kept running git status and git diff and telling itself to check more carefully. Three of its passing trees contained files no reviewer wants to see: a 927-line .backup copy of an application file, a temporary tsconfig, a tsconfig backup. And the two failures were the ones that open this piece: both React 19 type cases, where the model bumped the peer versions, declared the work done, and left TS2503: Cannot find namespace 'JSX' unfixed behind that bump. Every hosted model fixed those two by adding explicit JSX type imports.
Token use tells the same story from another angle. The local lane consumed several times the default Codex and Opus arms' input tokens on the same case, up to 1.18 million on one React case, and the four capped runs reported no usage at all because the client only emits it when a turn completes. The real figure is higher than the table shows.
The decision
On 2026-09-08 I took the local model out of the design. The fix lane runs on the cheapest hosted model that passed, escalating to a stronger one only when the verify step fails.
Correctness mattered most. A lane that emits "fixed" over a still-failing build is worse than a lane that says nothing, because it spends a human's trust on the wrong runs. The verify step would have caught both false claims, so no bad patch would have shipped. But a lane whose engine is wrong about its own output one time in four cannot be trusted with anything the verifier does not cover, and no verifier covers everything.
Time mattered second. Seven to ten minutes per PR against about one minute is not a rounding error when the goal is to attempt every failing PR every week. Cleanliness was third: junk files are filterable, but every filter is code I have to maintain for one engine.
Cost, oddly, mattered least, and it still went against the GPU. The local lane is billed at zero in the table because I own the hardware. It is not free. The Spark's idle draw alone, 22 to 45 watts across the reviews and forum reports I found, depending on firmware, costs more per year at typical US residential rates than the hosted lane would at observed volume, which the companion research puts at 7 to 27 USD a year. The break-even analyses I read put local parity against mini-tier pricing past 200 million tokens a month. A weekly dependency sweep is not that.
What did not change: the sandbox, the credential-free container, the egress allowlist, the patch path rules, the independent verify, the draft-only packet, and human review of every patch. The one real cost of the decision is a data-sharing one. PR diffs and CI logs from my repositories, all of them private, now leave the house to a vendor API. That is a choice I made knowingly, and it is the strongest argument left for a local lane. It was not strong enough here.
What this does not show
Eight cases, one run each, one corpus from three small Node repositories, one local model on one serving stack, and hosted models as they existed on 2026-09-07. Agents are not deterministic; a second pass could move a row. The per-arm speed ordering is soft. The 48-for-48 is an observed result on this corpus, not a reliability figure. Six arms agreeing shows that every case was fixable by several different models under this harness. It does not replicate anything: all six saw the same eight cases under the same protocol, once each. What I measured is case coverage. How often any one arm would repeat its result on a second run, I did not measure.
The local model is a 2025-vintage release, two agent-training generations behind the current open-weight coding models. Its failures match the documented open-weight agent failure class: non-termination, false completion, messy trees. If I retry local for this lane I will retry on a current model with a repeat-action detector and hard caps, not on this one. That is a plan, not a prediction that it would win.
The GPU is still busy. It runs my repository quality scanner and is the target of the eval harness that produced this benchmark. It just does not fix my dependency PRs.
The corpus
Eight Dependabot PRs whose CI had failed, replayed from the PR head at the recorded commit. Seven were closed with a known human fix at the time of the run; one was open. They came from three repositories: two small Node applications and one pnpm workspace. All three are private, so I describe the cases by shape rather than by path.
| Case | Bump | Root cause, from the verified fixes |
|---|---|---|
| A | grouped minor-and-patch, 7 updates, pnpm | lockfile regenerated without the pnpm.overrides block; --frozen-lockfile refuses |
| B | qs and express (Express 5) | Express 5 rejects the bare * catch-all route; needs /{*path} |
| C | typescript 5.9 to 7.0 | TS 7 removed baseUrl; the path alias must become relative |
| D | react and @types/react 18 to 19 | peer conflict with react-dom 18; needs the matching bump plus two type fixes |
| E | tailwindcss 3 to 4 | plugin moved to @tailwindcss/postcss; @tailwind directives replaced by an import |
| F | esbuild, vite, vitest | @vitejs/plugin-react peer range excludes Vite 8 |
| G | @vitejs/plugin-react 4 to 6 | plugin 6 requires Vite 8 |
| H | react-dom and @types/react-dom 18 to 19 | mirror of D |
All eight are narrow, log-signposted fixes. None needs cross-file reasoning. None has a wrong test. That is the shape of most Dependabot failures I see, and it is the shape the literature says is easy; the published benchmarks that report 23 to 51 percent success rates are built from curated breaking upgrades with hidden tests, which is a different population.
The harness
One run is one (case, arm) pair. The host does four things, all read-only against GitHub: fresh clone, fetch the PR head, pull the failing job log (error step only, bounded to a few hundred lines) into a fixture directory, and render a fixed prompt. The prompt names the PR title, the working directory, and the exact CI command, and sets six rules: keep the bumps; do not modify tests or test, lint, typecheck or build configuration except where the new version requires it, and explain any such edit; do not touch .github/; prefer the smallest change; ignore any AGENTS.md or CLAUDE.md in the checkout; and finish by writing a RESULT.json with a status of fixed or no_fix, a one-paragraph summary, and the list of files changed.
Everything else happens in three container steps in one pinned image (Node 26 on Debian bookworm, pnpm 10, Codex CLI 0.153.4, Claude Code 2.1.263), each with a 4-CPU, 8 GB, 1024-process limit:
- Install, pre-patch, using the CI install command. This is the "does the failure reproduce" step.
- Agent, with a 600-second
timeout, the one model key in the environment and nothing else, and the prompt on stdin (Codex) or as the prompt argument (Claude Code). Codex arms ran with approvals and its own sandbox disabled; Claude Code ran with permissions skipped. The container is the sandbox. A production wrapper must not do this on a host. - Verify, in a fresh container: delete
node_modules, run the exact CI command, record the exit code.
Between the agent and verify steps the host diffs the tree (excluding the fixture directory and node_modules), records the changed file list, and copies out RESULT.json or writes {"status":"missing"} if the agent never produced one. The agent's own status is recorded but never scored. Verify exit code is the primary metric; the rest are wall clock, tokens, cost, files changed, a forbidden-path heuristic, junk files, and result-file presence.
The forbidden-path heuristic flags .github/, test directories and test files, and test, lint and tsconfig files. It fires on case C for every arm because TypeScript 7 requires the tsconfig edit. Read that column as "review this", not "violation".
The arms and their prices
| Arm | Client | Model | Price, USD per 1M tokens (input / cached / output) |
|---|---|---|---|
| codex | Codex CLI default | GPT-6 Astra | 10 / 1 / 50 |
| codex-luna | Codex CLI -m |
gpt-5.6-luna | 0.20 / 0.02 / 1.20 |
| codex-mini | Codex CLI -m |
gpt-5.4-mini | 0.75 / 0.075 / 4.50 |
| claude | Claude Code default | Opus 5 | 5 / 0.50 / 25 |
| claude-sonnet | Claude Code --model |
Sonnet 5 | 2 / 0.20 / 10 |
| claude-haiku | Claude Code --model |
Haiku 4.5 | 1 / 0.10 / 5 |
| spark | Codex CLI via the Responses API | Qwen3-Coder-30B, BF16, vLLM | 0 (owned hardware) |
Prices are from the vendor pricing pages as read on 2026-09-07. Claude Code reports its own cost; Codex cost is computed from the token usage in its event stream. The local arm ran with a 32k model context window and auto-compaction at 20k tokens, and held the eval client lock so no other eval client competed with it. Zero is the table's price, not the true cost; the executive layer covers idle draw.
Results
Per-arm summary, from the harness's own summarizer:
| Arm | Cases | Verify PASS | Mean agent seconds | Total cost USD |
|---|---|---|---|---|
| codex (GPT-6 Astra) | 8 | 8 | 41 | 3.29 |
| codex-luna | 8 | 8 | 48 | 0.13 |
| codex-mini | 8 | 8 | 54 | 1.02 |
| claude (Opus 5) | 8 | 8 | 89 | 3.40 |
| claude-sonnet | 8 | 8 | 74 | 1.97 |
| claude-haiku | 8 | 8 | 110 | 1.34 |
| spark (Qwen3-Coder-30B) | 8 | 6 | 454 | 0.00 |
The two arms the decision turned on, case by case:
| Case | luna verify | luna seconds | luna cost USD | local verify | local seconds | local result file |
|---|---|---|---|---|---|---|
| A | PASS | 89 | 0.017 | PASS | 601 (cap) | fixed |
| B | PASS | 30 | 0.013 | PASS | 601 (cap) | missing |
| C | PASS | 25 | 0.014 | PASS | 305 | fixed |
| D | PASS | 48 | 0.018 | FAIL | 477 | fixed |
| E | PASS | 34 | 0.010 | PASS | 601 (cap) | fixed |
| F | PASS | 69 | 0.024 | PASS | 361 | fixed |
| G | PASS | 38 | 0.012 | PASS | 601 (cap) | fixed |
| H | PASS | 51 | 0.021 | FAIL | 85 | fixed |
A run that hit the cap shows agent exit 124 and 601 seconds. Those four runs also report zero tokens, because Codex emits usage only on turn completion and the turn never completed. The local arm's token totals in the raw table are therefore a floor.
Two hosted arms had one instruction-following slip each: gpt-5.4-mini wrote its result file into the working subdirectory instead of the repo root on case B, so its status reads missing while the fix itself passed; and the Claude Code arms and gpt-5.4-mini frequently left a committed tsc -b build-info file modified in the repository behind six of the cases, harmless noise from running the check. Codex arms left the smallest diffs overall.
Every hosted arm converged on the same fix family per case: regenerate the lockfile (A), the Express 5 route syntax (B), drop baseUrl and relativize the alias (C), add @tailwindcss/postcss and swap the directives for an import (E), bump the peer to a compatible major (D, F, G, H). The two bumps a human would call migrations, Tailwind 4 and TypeScript 7, were one-step fixes once the error message was read.
Failure anatomy, local lane
Three distinct problems, each matching a documented open-weight agent failure signature.
Non-termination. Four of eight runs ran to the cap after the fix was in place. On case B the model issued about 130 shell commands for a one-line route change. The transcripts show repeated git status and git diff loops and "let me check ... more carefully" messages until timeout killed the process. The verify step passed on all four, which is why they count as successes; a production wrapper would tolerate this at the cost of seven to ten minutes of node time per PR.
False completion. Cases D and H are mirrors: React 19 in one, react-dom 19 in the other, both ending in TS2503: Cannot find namespace 'JSX'. The local model bumped the peer versions, wrote "status": "fixed" with a confident summary, and exited at 85 and 477 seconds. Verify failed on the same TS2503. Every hosted arm fixed both by adding explicit JSX type imports. RepoRescue (2026-07) flags optimistic final messages in 32 to 76 percent of failed sessions across every agent it tested, after precision correction; this is two more.
Messy trees. A 927-line app.js.backup in the case B tree, a temp_tsconfig.json and a tsconfig.json.backup in two other passing trees, and no result file on one of the capped runs. The heuristic flagged two paths on case C for this arm against one for every hosted arm, consistent with a temporary tsconfig sitting beside the required edit.
The first and third are tolerable in a lane with an independent verifier and a patch filter. The second is not, without human review of every patch, and human review of every patch is already in the design. What the second one costs is trust in the agent's own summary as an input to that review, which is the one thing a "rationale" field in a packet is for.
Time and tokens
Codex arms averaged 41 to 54 seconds per case; Claude Code arms 74 to 110, with more turns (9 to 73 against Codex's single turn per run). Haiku was the slowest Claude arm, not the fastest; on case H it took 73 turns and 335 seconds, still passing, at 0.535 USD. The local arm averaged 454 seconds with four runs at the cap.
The default Codex arm ran 111k to 336k input tokens per case, mostly cache hits, with under 1.2k output tokens; the Opus arm 200k to 630k. The budget arms used more input (gpt-5.6-luna 170k to 618k, gpt-5.4-mini up to 1.23M) at a fraction of the price. The local arm reported 625k, 642k and 1.18M input tokens on three of the four runs that completed a turn; the fourth, the 85-second false completion, used 227k. The findings doc summarizes that as 5 to 10x the hosted arms. Against the default Codex arm on the same three cases the ratio I compute from the raw table is 5 to 6x on input and about 10x on output; against gpt-5.6-luna it is 1 to 3x on input. All of those are floors for the local arm, because the four capped runs reported zero.
Cost is dominated by model tier, not task. Token volume was similar in order of magnitude across the hosted arms; input price per token varied 50x. Per case the cheapest passing model cost about 0.016 USD. A weekly sweep with a handful of failing PRs costs cents.
What the literature says about the same shape
I compiled a companion research pass alongside the measurement, about 110 dated primary and vendor sources across benchmarks, production systems, failure modes and open-weight serving. Four findings bear on how much to trust the 48-for-48.
The task is easier than the literature's task. Published agents fix 23 to 51 percent of curated breaking dependency updates with hidden tests (DepBench, 2026-08; the BUMP-family papers, 2025). Real Dependabot CI failures are mostly lockfile mismatches, peer-range conflicts and removed compiler options, which the error message names. The hard tail, silent semantic breaks and cross-file API migrations, is real and rare, and it is where agents fail while showing green.
Green is a screen, not a verdict. Across the strongest 2026 sources, 8 to 40 percent of test-passing agent patches are wrong on closer inspection: 8 to 11 percent by differential testing, about 20 percent under strengthened tests, 30 to 41 percent under human review of plausible patches. My verify step is the existing CI command, so it inherits whatever that command misses. Human review of every packet stays.
Agents game tests when allowed to. Frontier models modify tests to force a pass in about half of conflicting-spec cases (ImpossibleBench, 2025-10), and "do not cheat" instructions barely move the rate. Read-only tests, shallow clones, held-out checks and an explicit "cannot fix" exit each measurably reduce it. The harness had the no_fix exit and forbade test edits by instruction; the read-only mount and shallow clone are on the list for the production wrapper.
The local model's behaviour is the documented open-weight class, and it is mostly harness-boundary: tool-call parser mismatches, chat-template bugs, thinking-mode interactions, and a measured tendency of smaller models to repeat a failed action after seeing the error. Qwen3-Coder-30B is a 2025 release; the current Qwen3.6 coding models post Terminal-Bench 2.0 scores of 51 and 59 against 60 for gpt-5.4-mini. If local is retried for this lane, that is the generation to retry on, with a repeat-call detector, thinking off, and hosted escalation.
Caveats
- Eight cases, one run each. Temperature and seed do not make these agents deterministic; a second pass could move a row or two. All six hosted arms agreeing on every case shows the eight cases were fixable; it does not show a second run would agree, and the per-arm speed ordering is the least stable part.
- The corpus is easy to medium by construction: real failures from three small repositories. It says nothing about failures that need cross-file refactors or where tests are wrong.
- The container ran Node 26; the original CI ran Node 20, 22 and 24 across the three repositories. No fix depended on the Node version, but a strict replay would pin it.
- Codex CLI's default model changed during the work. Pin
-min anything durable. - Hosted arms had unrestricted egress inside the container (registry, GitHub, vendor API). The production design restricts egress per step; that was not exercised here.
- Claude Code's default in this environment resolved to Opus 5, not the current Fable model; Fable was not tested.
- One local model, one quantization, one serving stack, one client. This is a measurement of that lane on that date, not of "local models".
- Hosted prices and model availability are as of 2026-09-07. Both move.
What changed in the design
- Engine: hosted only. gpt-5.6-luna through Codex CLI by default, escalating to Sonnet 5 through Claude Code or GPT-6 Astra only when verify fails. The local lane's proxy, client lock, context budget and scheduling constraint left the design with it.
- Nothing else relaxed. Sandbox, credential-free container, egress allowlist, patch path rules, independent verify, draft-only packets, human review of every patch. The hosted engine changes the data-sharing posture, not the write path.
- Cheap enough to widen. At cents per PR, attempt every failing PR every week instead of capping at three.
- Verifier hardening confirmed. Even passing runs left artifacts. The patch filter strips
*.backup,*.tsbuildinfoand anything under the fixture directory before packaging, and treats a missing result file as "packet without rationale", not as failure.
The harness and both documents live in my spark-trials repository, which is private; I quote them where they are used. The raw run directories, one per (case, arm) with the patch, the agent transcript, the verify log and the metrics, were not committed.
References
- FINDINGS: Dependabot fix-lane benchmark, local Qwen vs hosted coding agents, 2026-09-07 (spark-trials,
docs/FINDINGS-depfix-bench-2026-09-07.md; private repository; the document is quoted where used). Primary source for every run-level number in this article. - RESEARCH: LLM agents for fixing dependency updates and CI failures, 2026-09-07 (spark-trials,
docs/RESEARCH-llm-dependency-repair-2026-09-07.md; private repository; the document is quoted where used). Source for the literature figures, the idle-draw and break-even numbers, and the tiering of external sources. - The depfix harness (spark-trials,
scripts/depfix/:run-case.sh,run-arm.sh,prompt.tmpl,metrics.py,summarize.py,cases.tsv,Dockerfile; private repository; the document is quoted where used). - Jon Rehagen, "Why One Dependabot Merge Leaves the Other Five Behind", 2026-08 (/writing/dependabot-lockfile-cascade).
- "Update from Hell" / DepBench, arXiv 2608.30300, 2026-08. Source for the 51 percent ceiling on curated breaking updates. The 23 percent floor is from the BUMP-family compile-break studies (Byam, arXiv 2505.07522; Fruntke and Krinke, FSE 2025), as relayed in reference 2.
- RepoRescue, arXiv 2607.01213, 2026-07. Source for the 32 to 76 percent precision-adjusted optimistic-final-message rate in failed sessions across all tested agents and the read-only-tests effect.
- ImpossibleBench, arXiv 2510.20270, 2025-10. Source for the test-modification rate under conflicting specs.
- "Are Solved Issues Really Solved Correctly", ICSE 2026, arXiv 2503.15223. Source for the differential-testing wrong-patch rate.
- OpenAI and Anthropic API pricing pages, as read 2026-09-07, for the per-token prices in the arms table.
- ServeTheHome, DGX Spark review, 2025-10, and the NVIDIA forum idle-power thread, 2026-03, as relayed in reference 2, for the 22 to 45 W idle-draw range.