A daemon listening on a local socket is still a server with a trust boundary, and a green test suite says nothing about what happens at that boundary. This week we audited CGraph's resident graph daemon and found four latent defects — an unbounded wire allocation, missing socket timeouts, a data race, and a destructive persistence fallback — all sitting behind 64/64 passing tests. This post walks through each one, plus a fifth gap in the ingest contract and a self-healing fix for a scheduler that had silently stopped running.
CGraph's daemon, graphd, holds a resolved code graph warm in memory and serves queries like impact analysis over a local socket in about 10ms. That architecture is the whole point — but "resident process serving a socket" comes with server-class failure modes, even when the only client is supposed to be your own tooling on the same machine.
Why the tests were green
None of these bugs crash the happy path. Every one of them lives in a case the test suite never exercised: a hostile frame header, a stalled peer, two threads racing on a counter, a rename that fails. That's the general lesson before the specific ones — tests verify the behavior you thought to specify, and a trust boundary is precisely where the inputs you didn't think of arrive.
Bug 1: the 4-byte header that allocates 4 GB
CGraph's wire protocol is length-prefixed: a uint32 length, then the frame body. The daemon read that length off the socket and allocated the buffer directly from it. A single corrupt or hostile header — four bytes, 0xFFFFFFFF — forced a ~4 GB allocation before any body was read. The send side had a size cap; the receive side had none.
The fix is the boring, correct one: a shared kMaxFrameBodyBytes cap (64 MiB) defined once in the protocol header and enforced by both encode_frame and the daemon's read_frame. An oversized declared length is now rejected before allocation. If your protocol caps only one direction, it isn't capped.
Bug 2: one stalled client freezes everything
The daemon served each connection inline on its single serve-loop thread, with no receive or send timeouts on the socket. A client that stalled mid-frame didn't just hang its own request — it froze queries, the file-watcher poll, graph persistence, and idle shutdown, indefinitely.
Every accepted connection now gets SO_RCVTIMEO and SO_SNDTIMEO (5 seconds). A stalled peer is dropped and the loop moves on. Timeouts on a localhost socket feel paranoid right up until a client process gets wedged with a half-written frame — which, given the clients are AI coding agents and editor tooling, is a when, not an if.
Bug 3: the status query that raced its own writers
The daemon's status operation read the enrichment counters and iterated an internal map while the enrichment-refresh, ingest, and rescan paths wrote to them under a different lock. That's undefined behavior, and not a theoretical kind: the enrichment drainer polls status constantly by design, so the race ran in routine operation.
The fix gives that state one dedicated mutex. Every writer takes it, and status snapshots the values under the lock and builds its response from the copies. Data races don't announce themselves in tests — they announce themselves later, somewhere else, as corruption you can't reproduce.
Bug 4: the fallback that deletes your last good snapshot
This was the worst one. Graph persistence wrote a temp file and renamed it over graph.json. On rename failure, the fallback path deleted the existing graph.json and retried the write — so a double failure left the daemon with no graph at all. The recovery path was the only path that could destroy data.
Now persistence performs an atomic rename only. If the rename fails, the last-known-good graph.json is left untouched, the orphaned temp file is removed, and the failure is reported. This is rule number one of crash-safe file writes, and it's exactly the kind of rule a "just in case" fallback quietly breaks. A fallback that can be more destructive than the failure it handles is a bug with extra steps.
The fifth gap: validation that checked shape, not meaning
CGraph's enrichment step lets a host LLM contribute semantic fragments — prose-derived nodes and edges — that merge into the deterministic graph. The contract says malformed fragments are rejected with the graph unchanged. But validation was shape-only: a schema-valid fragment whose edges pointed at nodes that existed nowhere — not in the fragment, not in the graph — merged its dangling edges silently.
Ingest now resolves every edge endpoint against the fragment's own nodes and the current graph snapshot, and rejects the whole fragment atomically if any endpoint resolves against neither. Schema validation answers "is this well-formed?"; referential integrity answers "does this mean anything?" — a boundary needs both.
And one operational bug: the scheduler that wasn't running
The same day's second change was about residency rather than the wire. CGraph's enrichment drainer runs as a macOS LaunchAgent, installed once and then — as we discovered — reconciled by nothing. On the machine we checked, the plist existed but launchctl reported "Could not find service": the agent had silently fallen out of the launchd domain, and a 210-chunk enrichment backlog sat frozen with no signal that anything was wrong.
Two fixes shipped together:
- Self-healing residency. The daemon supervisor's reconcile pass now checks the drainer: if it's installed but its service isn't loaded, it re-bootstraps it. The decision is a pure function of two booleans — plist exists, service loaded — so the logic is unit-testable even though
launchctlitself isn't. Reconcile only restores an agent you explicitly installed; it never installs one for you. - A cadence that can actually drain. The drainer's default interval dropped from daily to every 4 hours. At the old cadence, that 210-chunk backlog would have taken about three weeks to clear; now it's a few days.
The pattern generalizes: anything you install into launchd, systemd, or cron will eventually fall out of it, and if nothing reconciles the desired state against the loaded state, the failure mode is silence.
What a green suite is worth at a boundary
The takeaway isn't "write more tests," exactly. Each fix here did land with tests — a truth table for the residency decision, cap and timeout assertions, a race-free status check. The takeaway is where to point your suspicion: the places where input arrives from outside your process (the wire, the fragment), the places where two threads meet (the counters), and the paths that only run when something else already failed (the persist fallback, the re-bootstrap). Those are exactly the paths a happy-path suite never walks.
All of this shipped in CGraph, spec-first — each change landed as an OpenSpec proposal with the failure modes written down before the fixes. If you're running a resident daemon of your own, steal the checklist: cap both directions of the wire, time out every socket, give shared state one lock, never let a fallback destroy the thing it's falling back from, and reconcile whatever you install.