Our test-impact benchmark was lying — so we built an oracle that could not

Blastline picks the tests a diff actually reaches by querying a CGraph code graph. To know whether that selection is any good, we replay history: for each of the last twenty commits touching a repo, we take the tests the author changed in the same commit and check that our selection — computed from the code changes alone — contains them. A co-changed test we miss is evidence of a blind edge in the graph.

On tokio, that number was 6 of 12. Half the co-changed tests, missed. It read like a recall crisis, and it was the reason Blastline shipped Rust as advisory — always fail open to the full suite — rather than trusting its subsets.

Then we questioned the benchmark itself, and the 6/12 fell apart.

"Co-changed" is not "depends on"

The safety proxy has a quiet assumption baked in: that a test the author touched in the same commit depends on the code that commit changed. Most of the time that holds. But it breaks exactly when it matters — on the cross-cutting sweeps that are noise for test selection.

Consider tokio ac6869a4, "remove unstable cfgs leftovers." The code change is cosmetic: a deleted doc-comment line and a renamed unused parameter. The commit also touches six test files — but every one of them changed only #[cfg(tokio_unstable)] attribute removals, part of a repo-wide cleanup. Those tests do not depend on the code change in any way. Blastline correctly excluded five of them. The proxy scored that as five misses.

So we replaced the proxy with something that cannot lie: a dependency-filtered oracle that measures real behavior, independent of our own graph.

The oracle: revert the code, keep the tests, re-run

The method is embarrassingly direct. For each test the proxy flagged as a miss:

1. Check out the commit.
2. Revert only the code hunks — leave the tests exactly as the author wrote them.
3. Rebuild and re-run those tests.

A test that flips from pass to fail depended on the code change; it is a true dependent, and missing it is a real recall gap. A test that still passes was swept along — noise, correctly excluded. No graph, no heuristic, just the compiler and the test runner as ground truth. (It shipped as scripts/bench-deps.ts in Blastline; you point it at a bench.ts run and a repo clone.)

We ran it across six Rust repos. The results were stark, and consistent:

• serde, proxy 0/2: both "missed" tests still pass when the code is reverted. The code change was a clippy::empty_enum → empty_enums lint rename. Zero true dependents.
• regex, proxy 4/9: all five "misses" still pass. The commits were pure style — inlining format arguments, {} → {name} in format strings. Zero true dependents.
• tokio, proxy 6/12: of the six missed tests, exactly one flips to failing when the code is reverted.

Across every repo we could build, every proxy "miss" we checked behaviorally was a lint rename, a formatting sweep, or a cfg cleanup — co-changed, not depended-on. Blastline's real recall was far higher than the proxy claimed. serde's alarming 0/2 was two non-dependents. The tokio "6/12" was one genuine miss, not six.

That one miss is where it got interesting.

Following the one real miss into the graph

The genuine tokio miss was dd344a55, which completes zero-length reads on the in-memory duplex stream. It changes SimplexStream::poll_read_internal, and its new test exercises exactly that. Blastline missed it, so the reverse walk from the changed method must dead-end somewhere. We built the graph and traced it.

poll_read_internal had zero incoming callers — nothing in the graph called it, not even the method sitting right above it in the same file that plainly does. The tracked issue had hypothesized an exotic cause (async trait-object dispatch through AsyncRead). The real cause was more mundane and far more consequential.

SimplexStream::poll_read — the method that calls poll_read_internal — is wrapped in a cfg_coop! { ... } macro. And tree-sitter, like most parsers, treats a macro body as an opaque token blob. Every fn, impl, and struct inside it is invisible. It is not a parsing bug; it is what "we don't expand macros" means. But tokio gates most of its code behind these cfg_*! item-wrapper macros — cfg_rt!, cfg_io_util!, cfg_coop!, and dozens more, 317 sites in tokio/src alone. All of it, dark to the graph.

That single fact explained everything: tokio's call-resolution rate was the lowest of any repo we measured, its test-to-code reachability sat at 0.59 while comparable crates were above 0.85, and poll_read_internal had no callers because the method that calls it does not exist as far as the parser is concerned.

The fix: unwrap the macros before parsing

The elegant part is what we didn't have to do. No macro expansion engine, no re-entrant parsing, no fragile AST surgery. Just a source pre-pass, run before tree-sitter sees the file: find each cfg_*! { ... } wrapper and blank the cfg_NAME! { prefix and its matching } with spaces — brace-matched, aware of strings, chars, and comments so it never miscounts a '}' inside a literal.

Blanking with spaces preserves every byte offset. So impl Stream { cfg_coop! { fn poll_read(...) {...} } } becomes impl Stream { fn poll_read(...) {...} }, and tree-sitter parses poll_read as a real impl method — at its true line number, inside its true impl, with its calls resolved — with zero changes to the rest of the extractor. When a file has no such macros, the pass is a no-op and touches nothing.

The measurements, on tokio, before and after:

Reachability jumped from barely-above-floor to the same class as crates that never hid code behind macros. The 452 new nodes and 2,761 new edges are dependency relationships that were simply absent before. And the recall — the number that started this whole thread — went from 6/12 to 11/12, with the one real miss now caught. Repos that don't use cfg_*! macros (clap, regex, ripgrep, serde) were unchanged: this is a targeted win for macro-heavy async Rust — tokio, tower, hyper — and inert everywhere else.

Two hypotheses died on the way

It is worth being honest about the wrong turns, because they are the point. The issue's original hypothesis — generic trait-bound dispatch — we implemented in full and measured: it added one edge across three large repos. Inert. And our own prediction, once we understood the macro problem, was that unwrapping macros alone couldn't close dd344a55 without also modeling async futures. The replay disproved us: it went straight to 11/12.

Both were plausible. Both were wrong. The only thing that told us so was measuring against reality instead of reasoning about it — the same discipline that turned a 6/12 "recall crisis" into a single genuine miss, and then closed it.

That is the lesson we keep relearning: a benchmark proxy is a hypothesis, not a verdict. When yours conflates "changed together" with "depends on," it will send you chasing ghosts. Build the oracle that can't.

Blastline and CGraph are both open source and MIT-licensed. The dependency-filtered replay is in the Blastline repo; the macro-unwrapping pass is in CGraph. If your codebase leans on item-wrapping macros, your code graph is probably darker than you think — and now there's a way to check.