The benchmark oracle that scored every miss as noise

git show --name-only prints nothing for a merge commit. Git suppresses the combined diff by default, and that single fact quietly broke the instrument Blastline uses to check whether its own benchmark is telling the truth. On a pull-request merge history, the oracle saw every commit as changing no code, reverted nothing, re-ran the tests, watched them all pass, and declared every missed test to be benchmark noise — a clean bill of health computed from zero evidence. This post is that bug, the guard that caught it, and the second mistake I made immediately afterward while assessing the damage.

What the oracle is for

A test-impact benchmark that replays history and asks "did we select the tests the author changed alongside this commit?" measures a proxy. Authors touch test files for reasons that have nothing to do with dependency: a lint sweep, a formatting pass, a rename across the tree. Counting those as misses makes a selector look worse than it is.

So we built an oracle: for each supposed miss, revert the commit's non-test code in a scratch worktree, re-run the missed test, and see what happens. If it breaks, the test genuinely depended on that code and the miss was real. If it still passes, it never depended on anything and the "miss" was noise. Behavior, not correlation.

The oracle exists specifically so a proxy metric cannot lie to us. It was lying.

Three lines that show the bug

Any repo with merges will do. Here is serde, at merge 6d6e9a11:

$ git show --name-only --format="" 6d6e9a11 | grep -c .
0
$ git diff --name-only 6d6e9a11~1 6d6e9a11 | wc -l
5

Zero files, then five files, for the same commit. And the command is not broken in general — on serde's most recent non-merge commit, git show --name-only reports its 2 files correctly. It fails on exactly one input class, silently, by returning an empty list rather than an error.

Now follow that empty list through the oracle. No files matched the revert filter, so nothing was written to the worktree. The re-run was therefore byte-identical to the baseline. Every missed test passed both times. Every miss was classified noise.

The two ranges

The part I find most instructive is that the benchmark and its oracle disagreed about what a commit even was.

bench.ts has always selected on <sha>~1..<sha> — the first-parent diff, which does capture what a merge brought in. The oracle used git show --name-only. So on merge commits the proxy measured one thing and the oracle reverted a different thing, and the two halves of one benchmark were reading different definitions of "this commit's changes." Both now use the same range:

// `git show --name-only` prints NOTHING for a merge commit (git suppresses the
// combined diff), so on a PR-merge history every commit looked like it changed
// no code, nothing was reverted, and every missed test re-passed as "noise".
// Diff against the first parent instead — the same range bench.ts selects on
// (`<sha>~1..<sha>`), so the oracle reverts exactly what the proxy measured.
const changedFiles = git("diff", "--name-only", `${sha}~1`, sha)

There is a second defect in the same function, and it is the same shape: the revert filter was hardcoded to /\.rs$/. Point the oracle at any non-Rust ecosystem and every commit matched no code files, taking the identical silent path to a confident all-clear. --code-re and --test-path-re now parameterize it, with defaults that reproduce the previous Cargo behavior exactly.

The guard is the load-bearing part

Neither fix is what makes this safe. This is:

if (changedFiles.length === 0) {
  // Nothing to revert means the re-run is identical to the baseline, so every
  // missed test would pass and be scored "noise" on no evidence. Refuse.
  console.error(
    `  ${row.commit}: no code files matched --code-re (${codeRe.source}) — cannot revert, skipping`,
  );
  for (const t of row.missed) {
    classified.push({ commit: row.commit, test: t, verdict: "inconclusive" });
  }
  continue;
}

An empty revert set is not a result. It is the absence of a result, and the only honest verdict is inconclusive. A third bucket alongside real-miss and noise costs almost nothing and converts a whole class of silent false-clear into a loud complaint.

That guard is how the bug was found. Pointed at a Java repo, the oracle printed "no code files matched" on five of stleary/JSON-java's six missed commits. Five refusals in a row is not a subtle signal. Without the guard, the same run would have printed a confident, wrong all-clear and I would have believed it.

What it cost

Same inputs, same repo, before and after the fix:

Java's dependency-adjusted recall on JSON-java was 8/13 (0.615), not the 8/9 (0.889) the broken run implied. Five genuine dependency misses had been invisible — and they were not scattered. They clustered in one subsystem: XMLTest three times, plus XMLConfigurationTest and JSONMLTest. That cluster was a concrete lead rather than diffuse noise, and following it produced the interface-dispatch and receiver-scope work in CGraph#68 and #69. After those landed, JSON-java's replay moved 8/14 → 14/14 and Java was promoted to replay-verified.

So the bug was worth more than it cost. But only because the guard turned it into a question instead of an answer.

The second mistake

Here is the part I would rather not write.

Having found that the oracle was broken on merge commits, I concluded that the previously published Rust oracle figures — in the README and in the dependency-oracle post — needed a re-run before they could be cited again. That reasoning was wrong, and I had to correct it after the fix had already merged.

The error was treating merge-heavy history as the exposure. It isn't. Only commits that carry a miss ever reach the oracle at all, so the exposure is the intersection of "has a miss" and "is a merge." In Rust, that set is empty:

serde's recent history is roughly a quarter merge commits and it is still entirely unaffected, because the commit that carried its miss happens not to be one. tokio and serde were then re-run end-to-end against real cargo test builds with the fixed script, and both reproduced their published verdicts exactly. Every published Rust claim stood. No correction was needed anywhere.

Checking that took about ten minutes, and I announced the conclusion before doing it. The lesson is not "be more careful" — it is that finding a bug in your instrument tells you nothing about which of your results it touched, and the second question deserves the same evidence standard as the first. A broad "everything is suspect" claim feels like the conservative, honest move. It isn't; it is just a different unverified claim, and it damages the credibility of the numbers that were fine.

What to steal

Two things, if you maintain a benchmark that anything depends on.

Give your instrument a refusal path. Any code that computes a verdict from a collection should ask what happens when the collection is empty, and whether that answer is distinguishable from a real one. "No evidence" and "evidence of no problem" render identically in most result formats, which is precisely why the confusion survives.

Test your instrument against a shape you have not seen. Both defects here were invisible for as long as the only input was single-parent commits in Rust repos. They surfaced within minutes of pointing the same script at a Java repo with a merge-based history. The generalization was not a feature; it was a test.

Blastline is MIT-licensed and the oracle is a single script — scripts/bench-deps.ts — worth reading precisely because it is small enough to audit and was still wrong in two ways.