A Claude Code agent I dispatched fixed a bug, opened a pull request — a proposed code change waiting for approval — and CI went green. CI is the robot that runs your tests; green means they passed. So I moved on.
The bug was still live twenty minutes later.
That’s the whole story, and it happened to me three times in a row before I stopped trusting the green checkmark. Here’s the thing a green CI check actually tells you: the tests that ran, passed. It does not tell you the fix merged, or that the code people are running contains it. Those are three separate facts, and I’d been treating them as one.
The first miss was the dumbest. CI passed on a stale diff — the agent had committed a partial change, tests happened not to cover the gap, green. Tests passing is evidence the code isn’t obviously broken. It is not evidence the fix is present.
The second miss took me longer to see. CI was green, but the PR wouldn’t merge, and nothing told me why. CodeRabbit — an AI review bot that leaves comment threads on your PR — had opened threads that were never resolved. Unresolved threads silently block the merge button. They don’t fail CI. So CI sat there green and smug while the merge quietly refused to happen, and I kept refreshing wondering what I was waiting on.
The third one is the one that actually scared me. The PR merged. The fix was really in main. And the bug was still live, because the long-running service executing that code — a conductor process that had been up for hours — was still holding the old version in memory. Merging changes the file on disk. It does not reach into a running process and swap the code out. The conductor had to actually reload before anything changed for a real user.
So now I own every agent-produced PR through four gates, and each one is a separate check because passing one tells you nothing about the next:
CI is green. The review threads are resolved. The merged code actually carries the fix. And the running process demonstrably picked it up.
The reload gate that caught me give me the detail
The first three gates are queryable over the GitHub API — check status, review thread isResolved, and a git show main:path to confirm the fix landed. The fourth is the one people skip. After merge, I don’t trust that a long-lived process reloaded; I prove it.
# after merge, restart the service and confirm the running code has the fix
systemctl restart conductor
sleep 3
# grep the loaded module the process is actually running, not the repo
pid=$(systemctl show -p MainPID --value conductor)
tr '\0' '\n' < /proc/$pid/cmdline # confirm it relaunched
grep -q "FIX_MARKER_v2" $(readlink /proc/$pid/cwd)/dist/conductor.js \
&& echo "running code carries the fix" \
|| echo "STILL STALE — process did not reload"The FIX_MARKER_v2 is a cheap trick: a unique string the agent adds alongside the fix, so “is the fix live?” becomes a grep against what the process is running, not a guess.
The general rule I’d hand you: when an agent ships code, verify the deployed, running thing contains the fix — not that a test passed near it. “Green” is a claim about one stage. Deployment is four. Watch especially for review bots whose unresolved threads block merges without failing anything, and for daemons that keep serving the old code until you make them let go of it.