Installing the CGraph plugin made Claude Code 7.1% more expensive on 32 SWE-bench Verified tasks, solved the same tasks within noise, and the agent never called a CGraph tool in any of the 96 runs that had it. The gate we set before the run, billed cost no higher than baseline at an equal or better solve rate, fails. This post is the harness, where the 7% comes from, the three follow-up runs that tried to close the gap, and the one lever that actually moved the bill.
The gate, and why billed dollars
Every earlier CGraph number was about tokens: 82% fewer tokens than grep-and-read on a context-gathering task, a graph query answering in about 10 ms. Tokens are the wrong unit for the question a user actually asks, which is does installing this make my coding agent cheaper or better. So the ticket set a gate in the unit the user pays in:
Gate (CGR-29): billed cost no higher than baseline, at an equal or better solve rate.
The harness (bench/billed-cost/, merged in #190) runs Claude Code headless on SWE-bench Verified tasks, once with no plugins installed and once with claude plugin install cgraph@cgraph, and scores both with the official SWE-bench Docker harness.
| Held equal across arms | |
|---|---|
| Claude Code binary | 2.1.296, confirmed by the init event in every run |
| Model | claude-sonnet-5-5 |
| Prompt | the issue text plus "edit non-test source, do not install or run tests" |
| Tasks | 32 SWE-bench Verified instances rated "15 min - 1 hour" (django 10, sympy 8, scikit-learn 6, sphinx 6, xarray 2), plus a 13-task "1-4 hours" set |
| Repeats | 3 per task per arm; arms interleaved under the same load |
| Isolation | fresh HOME and config per run, a one-commit checkout at the base commit, no path that names the arm |
| Cost | total_cost_usd from the final result event, Claude Code's list-price bill |
| Statistics | per-task means, paired differences, 95% percentile bootstraps over tasks |
The Repeats row matters more than it looks. Claude Code has no sampling seed, so a "seed" is an independent repeat, and the baseline arm alone moved from 82/96 solved in one batch to 85/96 in the identical re-run below. Any solve-rate difference smaller than that is noise, and we treat it as noise below.
The result
| Main set, 96 runs per arm | Baseline | CGraph plugin | Paired difference, 95% CI |
|---|---|---|---|
| Billed cost per run (mean) | $0.0529 | $0.0567 | +$0.0038 [+$0.0025, +$0.0050] |
| Billed cost, ratio of totals | 1.000 | 1.071 | [1.048, 1.094] |
| Tasks where CGraph cost less | 4 of 32 | ||
| Solve rate | 85.4% (82/96) | 87.5% (84/96) | +2.1 points [0.0, +5.2] |
| Runs that called any CGraph tool | n/a | 0 of 96 |
The hard set says the same thing: 6.6% more expensive (95% CI 0.7% to 12.9%), 13/39 solved against 14/39, and 0 CGraph calls in 39 runs.
The agent did not use the plugin once, so it could not win anything back. There were 0 graph-tool calls, 0 Skill calls and 0 ToolSearch calls across all 135 plugin runs. Claude Code 2.1.296 lists MCP tools as deferred: the plugin's 12 tool schemas are about 4.1k tokens, but the first request grows by only 751, because the model sees tool names and skill descriptions, and must decide to search for a tool before it can call it. On these tasks it never did. Sonnet 5.5 finds the code with one to three grep calls, 3.4 tool calls and 4.4 turns per run on average. There is very little navigation left for a graph to replace.
Where the 7% comes from
All of it is the plugin's always-loaded prompt text. The cgraph and cgraph-enrich skill descriptions (1,326 characters) plus the 12 deferred tool names (490 characters; the split is an estimate from the plugin files) add about 751 tokens to the prefix of every request, on 32 of 32 tasks. Under prompt caching, that prefix is written to the cache once per run and then read back on every turn.
| Main set, per run (mean) | Baseline | CGraph | Δ | Δ in $ |
|---|---|---|---|---|
| First-request prefix tokens | 17,573 | 18,323 | +751 | |
| Cache-write tokens | 8,918 | 9,743 | +825 | +$0.0033 |
| Cache-read tokens | 70,665 | 75,638 | +4,974 | +$0.0005 |
| Output tokens | 1,015 | 1,011 | -4 | -$0.00004 |
Cache writes are 67% of the baseline bill, so a token that gets written to the cache is the most expensive kind of token in this workload. The gap is 88% cache writes and 13% cache reads; output tokens are unchanged. The results file calls this the "fewer tokens is not a lower bill" effect, seen from the other side: a few hundred always-loaded tokens are a measurable share of a five-cent run.
Telling the agent to use it does not fix it
A diagnostic arm added one sentence to the prompt: use the cgraph tools to locate and understand the relevant code before reading or editing files. It does not count toward the gate, because its prompt differs from the baseline, but it answers the obvious objection.
| Directed arm | Main (32 x 3) | Hard (13 x 3) |
|---|---|---|
| Runs with any CGraph call | 1 of 96 | 5 of 39 |
| Cost ratio vs baseline | 1.069 [1.032, 1.104] | 1.043 [0.981, 1.107] |
Even when told to, the agent made at most one graph_query per run, then went back to grep and Read. The first query on a cold checkout took 9 to 10 seconds on django while the daemon built the index (in django-16263 seed 1, the query went out at 17:10:15.281 and came back at 17:10:24.451). The answer was a symbol list, which the agent then re-read with Read. In the runs that called CGraph it added turns instead of replacing them.
The lever that moved the bill
The follow-up list in the results file starts with the only lever that changes cost when the tools go unused, which was 100% of runs: shrink the always-loaded footprint. #195 did three things: the MCP server lists a short default tool list instead of twelve, the cgraph skill description is one line, and cgraph-enrich is no longer model-invocable. The re-run below measured the footprint branch at three default tools (graph_query, graph_context, graph_impact), where the prefix delta is +113 tokens; review added graph_status back before merge, and the merged plugin's probe reads +133. The same harness, tasks, seeds, binary and model, re-run on that branch:
| Main set, 96 runs per arm | Plugin 0.9.1 | Footprint branch (three tools) |
|---|---|---|
| First-request prefix, CGraph minus baseline | +751 tokens | +113 [111, 114] |
| Billed cost, ratio of totals | 1.071 [1.048, 1.094] | 1.006 [0.973, 1.038] |
| Tasks where CGraph cost less | 4 of 32 | 10 of 32 |
| Solved, baseline / CGraph | 82/96 / 84/96 | 85/96 / 83/96 |
| Runs that called a CGraph tool | 0 of 96 | 0 of 96 |
The overhead is about a sixth of what it was and the cost ratio's confidence interval now spans 1.0. It is not zero: the median paired difference (+$0.0009) still excludes zero, and CGraph cost more on 22 of 32 tasks, which is the 113-token prefix paid on every request. The gate as written still does not pass, because the agent still made 0 calls in 96 runs, so the plugin has nothing to save on these tasks.
Bigger tasks do not change the agent's mind
The natural next hypothesis was task size: on a one-file fix, three greps are enough, so pick tasks where navigation is the cost. #198 took every SWE-bench Verified instance whose gold patch edits two or more files, 71 tasks across 10 repositories, and ran the same three arms.
| 71 tasks x 3 seeds | cgraph vs baseline | directed vs baseline |
|---|---|---|
| Billed cost, ratio of totals | 1.035 [0.997, 1.078] | 1.070 [1.028, 1.117] |
| Solved (baseline 102/213) | 112/213 | 100/213 |
| Runs that called a CGraph tool | 0 of 213 | 16 of 213 |
Zero of 213 undirected runs called it. The cgraph arm's higher solve rate (+4.7 points, CI touching zero) came with no CGraph call, so the only difference from baseline was about 133 tokens of prompt, the merged four-tool plugin's prefix; it did not reproduce with the same plugin a day later (-1.4 points), which is what noise looks like.
The 16 directed runs that did call CGraph are the most useful rows in the whole study, because they show what a call costs when it happens:
| Directed runs that called CGraph | Runs | Cost vs the same task and seed's baseline (median) |
|---|---|---|
Loaded the cgraph skill first | 8 | +75% |
| Called the tool without the skill | 8 | +39% |
Loading the skill put 7,866 tokens into the context that every later turn re-reads. The graph_query answer itself added about 1,400. When the agent does use CGraph, the skill is the cost, not the query. #196 cut the skill body to 950 tokens and moved the full guide into a separate reference file.
The hook that never fired, then fired and changed nothing
The plugin also ships a pre-edit hook: before the agent edits a file, it asks the daemon which other files are likely to change with it. The multi-file set is exactly where that should help. The baseline agent's patch misses at least one gold source file in 72.4% of runs on the 70 of these tasks with two or more gold Python files (mean gold-file recall 0.573), so there is a lot for a hook to do.
It never ran. The hook asks only a daemon that is already running and never starts one, and no daemon existed in any of the 309 plugin runs across the multi-file set and the footprint re-run. So cgraph-mcp was changed to start the daemon when the session opens in a git repository, and the set was run again with a daemon present in 213 of 213 plugin runs.
| 71 tasks x 3 seeds | Hook silent | Hook live |
|---|---|---|
| Plugin runs with a daemon | 0 of 213 | 213 of 213 |
| Runs whose patch misses a gold source file, baseline / CGraph | 72.4% / 72.4% | 72.9% / 72.9% |
| Paired difference | +0.0 [-4.8, +4.8] points | +0.0 [-3.8, +3.8] points |
| Billed cost, ratio of totals | 1.035 | 1.022 [0.994, 1.051] |
Nothing moved, because the hook almost never fired even with a daemon to ask. It is a PreToolUse hook on Edit|Write|MultiEdit, and these agents edit through Bash with python3 or sed scripts: 801 Bash calls against 22 Edit calls across the 213 runs, and only 20 runs used an edit tool at all. Even a hook that always named the missing file could move the miss rate by at most 9 points, short of the 10-point gate. Starting the daemon was necessary and not sufficient; the hook needs a trigger these agents actually hit.
A read hook was the last attempt: after a Read or a Bash command that names a repository file, inject the same message for up to two files per call. That one reaches the agent. It also does not help, and it costs about 4%: the share of runs missing a gold file was 75.7% for the baseline and 73.8% with the hook (paired -1.9 points, CI -6.2 to +2.4), while the cost ratio was 1.042 [1.017, 1.069], with the interval excluding 1.
What we take from it
None of this says a code graph is useless to a coding agent. It says something narrower and more useful: on SWE-bench Verified tasks of this size, with this model and this harness, the agent does not reach for one, and anything the plugin loads into every request is pure cost until it does.
- Measure billed dollars, not tokens. An always-loaded prefix is paid as a cache write and then re-read on every turn. A tool that saves 1,000 tokens on the one call the agent makes and costs 750 on the forty it does not is a net loss. The prefix delta is measurable with
prefix_probe.pywithout a full run. - An agent that solves the task in three greps has nothing for you to save. The benchmark's own tool-call count is the ceiling on what a navigation tool can win. Check it before building the harness.
- One call has to beat the greps it replaces. A nine-second cold query that returns a symbol list the agent re-reads with
Readadds a turn. Pre-build the index and return the source span. - A hook needs a trigger the agent actually hits. Ours was keyed to edit tools that these agents use in 9% of runs.
- Directed prompting is a diagnostic, not a fix. Told to use the tool, the agent used it once and went back to
grep.
The study cost $25.25 against an approved ceiling of $400 for the original batches, with the footprint and multi-file re-runs reported separately in the results file. The run logs for the main, hard and directed arms (405 runs) are a release asset, and analyze.py reproduces the committed main and hard results byte for byte from them; the multi-file run logs are not published.
For completeness: the retrieval benchmarks
Billed cost is one of three public measurements CGraph now carries, and the other two are mixed rather than negative. On Agent Retrieval Bench at bin-v0.9.1, Recall@20 is 0.6362 against RepoMap's published 0.6333, a difference inside the noise floor the results file measures at about 0.006, so level, not a win; BCY@8k is 0.4319 against 0.3788, well outside it. On ContextBench Lite, the default context bundle's file precision is 0.100 (95% CI 0.089 to 0.112) against a gate of 0.5, and the results file says plainly that this is not a noise call. Both have the same shape as this post: pinned data, the benchmark's own metric code, a one-command run.sh, and a stated gate that was set before the run. They live in bench/arb/ and bench/contextbench/.
Reproduce it
cd bench/billed-cost
W=/var/tmp/cgr29 # work dir; no CLAUDE.md in any ancestor
python run_ab.py templates --work $W
python run_ab.py run --work $W --tasks tasks.json --seeds 1 2 3 --jobs 6
python run_ab.py run --work $W --tasks tasks.json --arms cgraph_directed --seeds 1 2 3 --jobs 6
python evaluate.py --work $W --tasks tasks.json --jobs 8
python analyze.py --runs $W/runs --tasks tasks.json --out results/main-cgraph.json
Swap tasks.json for tasks-hard.json or tasks-multi.json for the other sets; gold_recall.py scores the multi-file runs against their gold files. The steps that come before these, the bare repository mirrors to clone and the OAuth token the runs need, and the full write-up with per-task tables and the pins for every binary and image, are in bench/billed-cost/RESULTS.md. If you ship an MCP server or a plugin for a coding agent, the probe is the part worth copying: measure what you add to the first request before you measure anything else.