Dogfooding test impact: Blastline on CGraph's own PRs

Every pull request to CGraph that touches src/ or tests/ builds a code graph of CGraph and hands it to Blastline, which comments on the PR with the smoke tests that diff can actually reach. It is 55 lines of workflow and it has been running for two weeks. The honest scorecard across the eleven PRs that have triggered it: six produced a real test subset, five said "run the full suite" — and all five of those had the same single cause. This post is the setup, the comments it actually posts, and what the failures taught us, including one on the PR I merged yesterday.

The whole loop is one workflow file

.github/workflows/test-impact.yml, in full but for the header comment:

on:
  pull_request:
    paths:
      - "src/**"
      - "tests/**"

jobs:
  test-impact:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - name: Install cgraph (prebuilt release binaries)
        run: |
          gh release download bin-v0.1.0 --repo Nxtsoft/CGraph \
            --pattern cgraph-linux-x64.tar.gz --output cgraph-bin.tar.gz
          mkdir -p "$HOME/.local/bin"
          tar -xzf cgraph-bin.tar.gz -C "$HOME/.local/bin"
          echo "$HOME/.local/bin" >> "$GITHUB_PATH"

      - name: Build the self-graph
        run: cgraph --root . --out /tmp/self-graph

      - uses: Nxtsoft/blastline@v0
        with:
          graph-path: /tmp/self-graph/graph.json
          base-graph-command: cgraph --root . --out "$BLASTLINE_BASE_OUT"
          github-token: ${{ github.token }}
          ignore: |
            \.md$
            ^openspec/
            ^\.github/
            ^integrations/
            ^assets/
            ^docs/

Three details are doing the real work.

fetch-depth: 0 is not cargo cult. base-graph-command runs inside a detached worktree of the range's base commit and writes a second graph for the pre-change tree, which is how a deleted symbol can still be mapped to the tests that used to reach it. Without full history there is no base commit to check out, and deletion mapping degrades.

The ignore list is a claim about relevance, not a convenience. Each regex asserts "changes here cannot affect which tests should run." A .md file, the OpenSpec directory, the workflow directory itself — fine. Getting this list wrong in the other direction is how a test-impact tool starts lying, so it stays short.

Nothing in the job runs a test. It posts a comment. ci.yml still runs the full suite on all six of its {ubuntu, macos} × {default, release, sanitizers} combinations, every time, plus a seventh job for the fuzz smoke targets. That is the point: the selection is advisory here, a claim to be checked against reality, not a gate that skips work. You cannot learn whether your test-impact tool is honest by trusting it.

What the comment says

On a good day, an explicit list of tests plus a collapsible blast radius of the symbols the diff reaches:

### blastline — test impact for `25d82d95…..HEAD`

**Impacted tests (run at least these):**

- tests/smoke/configured_extractors_test.cpp
- tests/smoke/cpp_extractor_test.cpp
- tests/smoke/daemon_persistence_test.cpp
  … 17 more

<details>
- function run_one_shot (src/engine/pipeline.cpp:30)
- function run_build (src/cli/main.cpp:94)
- function note_unextracted_change (src/engine/incremental_update.cpp:100)
  …
</details>

computed from source tree `sha256-merkle-v1:59b44afa…`

That last line is the part I would copy into any tool that makes claims about a repo. The comment is stamped with the merkle content root of the tree it was computed from, so a selection is tied to an exact file state rather than to a commit sha that a rebase can make meaningless. A reader can tell whether the advice describes the code in front of them.

Eleven PRs, honestly

The suite grew from 70 to 76 tests over that span, so the real subsets ran roughly 3% to 38% of it.

Five fail-opens, one cause, and it is not a bug in any of the obvious places. tests/smoke/CMakeLists.txt is a CMake file. CGraph does not extract CMake, so the file has no node in the graph, so nothing in the graph can say what depends on it, so Blastline fails open rather than guessing and says run everything. Every link in that chain is behaving correctly — it is the same refusal-over-guessing rule that governs what a code graph will not infer, applied one layer up.

The uncomfortable part

Look at which PRs failed open. #62, #64, #68, #69, #71 all added a new test file — and in CMake, adding a test means registering it, which means editing tests/smoke/CMakeLists.txt.

So the loop goes blind exactly on the pull requests that add tests. Which is to say: on the changes most likely to be doing something the existing tests do not cover. A test-impact tool that works beautifully on refactors and gives up on new-behavior PRs has an inverted usefulness curve, and no amount of graph quality fixes it — the blocker is a build-system file that a code graph has no business claiming to understand.

The tempting fix is to add ^tests/smoke/CMakeLists\.txt$ to the ignore list and make the problem disappear. That would be a lie of exactly the kind the ignore list is supposed to prevent: editing that file changes which tests exist, which is about as relevant to test selection as a change can be.

The honest fix is narrower and we have not built it: teach the selector that a change to a test-registration file selects the tests it declares. add_test(NAME cgraph_import_disambiguation_test …) names a target; a diff that adds that line is a diff that should select that test and, absent other evidence, nothing else. That is a build-system-aware rule per ecosystem — CMake, Gradle, Cargo — which is real work and a real maintenance surface, and it is the difference between "run 76 tests" and "run the 1 you just added."

Until then the behavior is right and the ergonomics are bad, and it is worth saying which is which.

The pin, since we were already looking

The workflow installs bin-v0.1.0. That release was cut from commit 1408c8f, which is now twelve commits behind main, and bin-v0.2.0 has been the latest release for a week. Every self-graph since then has been built by an older extractor — the same freshness trap that makes stale index proofs worth having.

So we measured it rather than assuming. Both binaries, same tree, CGraph's own repo:

Twenty-six more calls resolve, and the graph gains exactly one edge. f6e0c06 is eleven commits past the pin; most of them were Rust, Kotlin, and Java extractor work that a C++ self-graph never touches, but one — a fix stopping a same-file member call from binding to a free function — changed shared resolution, and that is where the movement comes from. The pin is stale in a way that is real but currently almost free.

Still worth un-pinning. "Almost free" is a property of this month's commits, not of pinning, and the next person to copy this workflow will copy the pin along with it.

What to copy

If you want this on your own repo, the Action can do the graph build itself — set graph-root instead of graph-path and it installs CGraph and runs it, no prebuilt-binary step to maintain:

- uses: Nxtsoft/blastline@v0
  with:
    graph-root: .
    base-graph-command: cgraph --root . --out "$BLASTLINE_BASE_OUT"
    github-token: ${{ github.token }}

Two pieces of advice from running it on ourselves. Keep the full suite running alongside it until you have watched the comments for a few weeks — every interesting thing in this post came from reading what it said and asking whether that was true, which is impossible if the selection is silently gating CI. And read the fail-open reasons rather than counting them: five identical messages looked like noise until we noticed which PRs produced them.

Both projects are MIT-licensed. The workflow above is the whole implementation, and the Action's inputs document the turnkey path.