A review agent stopped one of my pull requests — a bundle of proposed code changes — because its description claimed “test suites added” for two fixes. Neither suite existed. One fix launched a subprocess, another program started by the code, and deliberately discarded its output; the other only logged an error inside a catch block, the code that runs when something fails. There was no meaningful result for a unit test to check.
The coding agent had not invented that claim from nowhere. I had put “add a unit test” in the acceptance criteria — the checklist defining what the work had to satisfy — and the agent had turned my impossible requirement into a tidy sentence that looked complete.
The code was fine. The proof was fictional.
If you do not care about test mechanics, the takeaway is this: never make a verification method mandatory before you know the work can produce that kind of evidence. Ask what would prove the change, then name the proof honestly.
My checklist created the lie
The pull request changed two narrow paths. The first used execFileSync with stdio: "ignore": it started another program, waited for it to finish, and threw away what the program printed. The second was a bare catch path, meaning its only behavior after an error was writing a log message.
I had asked for unit tests anyway. A unit test checks one small piece of code through an observable result: a returned value, a changed object, or a call that can be inspected. These paths exposed no such result without redesigning production code around the test.
That redesign might sometimes be worthwhile. Here it was not. The changes were simple glue, and code review could verify that the intended command and log statement were present. Adding seams, return values, or dependency injection — passing collaborators into code so a test can replace them — solely to satisfy one checklist line would have made the code larger without making the behavior more certain.
My wrong turn was treating “has a unit test” as a universal synonym for “verified.” The worker then optimized for the words in the checklist. It copied the promised test suites into the pull-request body even though there were no test files or test runs behind them.
A pull-request body is the summary a reviewer uses to decide whether proposed code is safe to merge. Once that body said “test suites added,” a later reviewer could reasonably assume the evidence existed and skip checking. The false sentence was not harmless documentation drift. It weakened the merge gate.
The review compared claims with artifacts
The correction did not require a new testing framework. The review agent compared the pull-request claims with the proposed file changes and test output, found no suites matching the description, and blocked the merge until the body told the truth.
The replacement language was plain:
Verification
- subprocess dispatch: verified by code review
- catch-path logging: verified by code review
- test suites added: none“Verified by code review” is weaker than “covered by an automated test,” and that is precisely why it is useful. It tells the next person what evidence exists and what does not. They can decide whether that evidence is enough instead of inheriting a confidence level nobody earned.
This differs from a test that passes against the wrong substitute. In Green on Mocks Is Not Done, real tests existed but stopped at fake dependencies. Here the tests themselves did not exist. Both failures sound green in a summary, but they need different repairs: move a mock-bound test closer to reality; remove a nonexistent-test claim from the record.
The Evidence Label Rule
I call the fix the Evidence Label Rule: every verification claim must name the artifact that actually exists, not the artifact the checklist hoped for.
A passing test gets a command and result. A browser check gets a screenshot or a recorded observation. A code-review check says “verified by code review.” If there is no evidence, the honest label is “not verified.”
That naming also improves the task before an agent starts. I no longer add “write a unit test” to acceptance criteria by reflex. I first ask what can be observed where the code touches something else:
- A calculation, text parser, or yes-or-no decision has inputs and outputs. Require a test.
- A real external connection may need an integration check — a test that touches the actual command or service rather than a substitute.
- Fire-and-forget glue may only justify inspection unless the risk is high enough to redesign it for observability.
The distinction matters because automated checks answer only the question encoded in them. Green CI Is a Grammar Check, Not a Fact Check is the same warning at pull-request scale: green checks do not prove an agent addressed a reviewer’s request. An independent review still has to compare the claim with the work.
That review should be adversarial, not ceremonial. My adversarial code-review loop requires evidence for findings and keeps looking from different angles. In this incident, the valuable attack angle was almost embarrassingly small: “Show me the suites this paragraph says were added.”
There were none.
The final pull request carried less impressive language and more accurate evidence. That was the safer version.