Green on Mocks Is Not Done

The safety cap fired during the exact incident it was built to prevent — and silently did nothing.

I’d built it into a multi-agent system over a few days. The cap had one job: detect an overload spike and kill the runaway process before it took down the box. Unit tests green. Mutation tests — where a tool like Stryker deliberately breaks your code to see if your tests notice — all caught. Seven rounds of adversarial review, clean. Merged.

Then I hit the live box. The real incident happened. And the cap produced output that looked like a kill command but was actually a log line — it had printed the right words to stdout without ever calling the function. The test had checked the mock’s return value, which was the right string. The mock said it worked.

That was the moment the pattern became impossible to ignore. Every layer had declared itself done with honest, verifiable evidence. Design, unit, mutation, review — each gate passed in sequence. And yet the next, more real layer found bugs the previous layer could not have seen.

Design said “flawless.” The first implementation pass found non-executable SQL and that safety cap — a cap that silenced itself during the incident it was meant to catch. Units said “mutation-proven.” An independent review found two critical and a dozen major bugs the unit’s own tests sailed right past. Review said “sound, committed.” The live box found integration bugs no unit test could catch — a component that couldn’t read the real dependency at all, a process check that inspected the wrong process.

The pattern only ever broke when verification touched something more real: a different model, the live machine, a human’s eyes.

What “green” actually meant

Here is the uncomfortable part. Every “green” was honest. The tests really passed. The mutations were really caught. The reviewer really approved.

But “green” meant consistent with my own assumptions — not matches reality. The bugs all lived at the boundary between the system and the real environment: the real command’s exact output shape, the difference between a tmux window and a tmux session, the difference between a shell process and the program running inside it, a directory two agents quietly shared without either knowing.

Those are exactly the places mocks hide. My unit tests injected fakes — stand-in objects that return canned responses instead of talking to real systems. So they verified the logic against the same wrong assumptions the code was built on. “Mutation-proven” proves a test catches a regression against the mock’s shape. It says nothing about whether the mock’s shape is right. At a boundary between your code and something external — a database, a subprocess, the filesystem — it usually isn’t.

The verification chain was allowed to terminate on a mock. That was the whole bug.

The ladder

The fix is a ladder, and a rule: a unit is not done until it has been exercised against the most-real substrate available right now, and every rung it hasn’t reached is named out loud.

I didn’t invent this. It’s Google’s testing pyramid read from the other end. Most teams over-trust the bottom — the fast, cheap, mocked unit tests. The lesson is that the boundary needs the top: a real browser, a real database, a real process on a real machine.

the mechanism — why boundary mocks structurally fail and how to gate against it give me the detail

Why mocks can’t catch boundary bugs. A mock is a contract written by the same person who wrote the code. At a process boundary — spawning a subprocess, calling a real CLI, querying a live database, inspecting a running process — the actual output shape is owned by the external system, not by you. When the real tool diverges from the mock (different JSON key, extra newline, a session vs. a window, a shell pid vs. the program’s pid), every test in the suite stays green and every mutation is caught. The test suite is sound within the mock’s world. It’s just the wrong world.

Concrete example: process inspection. Say you check whether a program is running by inspecting /proc/<pid>/cmdline or using pgrep. Your mock returns a predictable string. The real shell might return the interpreter path, not the script name. A real-I/O acceptance test exposes this immediately; the mock never can. The same class of bug hits every boundary: tmux list-panes vs. list-windows, pg_isready vs. an actual query, curl -o /dev/null vs. parsing the response body.

The gate that makes the discipline stick — stamp every unit with its reality rung before merge. One lightweight pattern using shell and CI:

# In your acceptance test, tag the test file with its reality level:
# @reality-rung R1
# Then in CI, refuse to merge a boundary unit that has no R1+ tag:
grep -rL '@reality-rung R[123]' tests/integration/ && \
  echo "FAIL: boundary units missing reality-rung tag" && exit 1

Tools that help at each rung: Playwright or any drive-the-real-Chrome harness for real-browser R1/R2; testcontainers to spin a real Postgres/Redis/Neo4j instance instead of an in-memory stub; Stryker for mutation coverage (necessary, not sufficient); a cross-vendor model review pass (e.g. build with one frontier model, review with another) for the design layer.

Mocks may verify logic. They may never close a unit that touches an external boundary.

“Works on the live box” is the only claim that proves reality. Everything else — the TypeScript compiler says clean, tests pass, mutation-proven, reviewed sound — is necessary and never sufficient for anything that calls a real command, spawns a process, or talks to another machine.

The discipline

Three rules carry it:

Every boundary unit needs at least one acceptance test against real I/O — not injected seams — before it’s done. A suite that is 100% fakes — where every dependency is a mock you wrote — has verified nothing about the boundary.
The reviewer is a different substrate. A different model, the live machine, a human. Not the builder’s own fixtures grading the builder’s own code.
No silent mock-termination. If a thing was only checked at R0 — logic-green, never touched real I/O — say so. “Logic-green, reality-unverified” is an honest status. “Done” is a claim you have to earn with reality-contact.

The words done, green, sound, shipped applied to a boundary unit should make you ask one question: against what? If the answer is “a mock I wrote,” it isn’t done. It’s a hypothesis that happens to be green.

How I wire it in

The discipline only holds if it’s mechanical. The shape I use:

A unit’s “done” transition is gated on an R1+ artifact — a real-I/O test, a live-run log, a screenshot — not just a passing unit suite. For UI, that means actually opening the page in a real browser and looking at it — a “drive the real Chrome” approach, not asserting a 200.
The reviewer is a different model. When I build with Claude Code, the adversarial review pass runs on a different frontier model — a model from a different AI lab — than the builder. A cross-vendor check catches the blind spots a model shares with itself. (This is also why mutation testing — Stryker and friends — is necessary but not sufficient: it proves the test catches a regression against the mock, never that the mock is real.)
Every status carries its reality rung explicitly: R0 logic-green, R1 real-I/O, R2 live, R3 soak — meaning it’s survived hours or days of real traffic. An un-reached rung is named, never silently omitted.

None of it is heavy. It’s one gate and one habit: don’t let the verification chain terminate on something you wrote to make it pass.

Built on: the verification ladder is just Google’s testing pyramid read from the other end — most teams over-trust the bottom; the lesson is that the boundary needs the top. Adversarial cross-model review uses Claude Code with a different-vendor reviewer. Mutation testing via the Stryker family.