Last week one of my Claude Code agents finished patching a pull request — a PR, the bundle of proposed code changes another person or bot reviews before it goes live. All four CI checks went green. CI is the robot that automatically compiles the code and runs the tests; green means nothing broke. My cursor was already on the merge button.
I didn’t press it. And I’m glad, because green CI was answering a question I hadn’t asked.
Here’s the thing green CI actually proves: the code compiles and the tests pass. That’s a grammar check. It tells you the sentences are well-formed. It tells you nothing about whether the agent did what the reviewer asked.
Because here’s what had happened. A reviewer — in my case CodeRabbit, an AI that reads every PR and leaves comments like a senior engineer would — had left four review threads. My agent went through them, edited files, and pushed a new commit with a tidy message: “Addressed all review comments.” Tests passed. Everything looked done.
But “addressed” was the agent grading its own homework. An agent that half-fixes a comment will still write “done” in the summary with total confidence — not because it’s lying, but because it genuinely can’t see the gap between what it changed and what was asked. Self-verification has a blind spot exactly where you need it most.
The fix that CI can’t give you is a second reader. So my standing rule now: after the agent pushes, I wait for CodeRabbit to re-review the new HEAD commit — the latest version of the code. Not the old comments. The new ones, posted after my fix landed. If CodeRabbit comes back APPROVED, or with zero new actionable comments, I merge. If it opens three fresh threads because the agent missed something, I just caught it before production instead of after.
The trigger to merge is an approving review, not a check status. Those are different facts about different questions.
Gating merge on a fresh review, not check status give me the detail
The trap is polling CI. What you actually want is a review posted after your fix-push SHA. Grab the head SHA, then look for a review whose commit matches it:
SHA=$(gh pr view "$PR" --json headRefOid -q .headRefOid)
gh api "repos/$REPO/pulls/$PR/reviews" \
--jq "[.[] | select(.commit_id==\"$SHA\" and .user.login==\"coderabbitai[bot]\")] | last | .state"Merge only when that returns APPROVED. Also pull inline comments (/pulls/$PR/comments) — CodeRabbit puts real objections in the review body and inline, and an approving body can still sit above unresolved inline nits. Make waiting the default; require an explicit --skip-review flag for a human to override.
If you let agents auto-fix your PRs, add one hard gate: green CI never means merge-ready. It means the code is syntactically alive. Whether it’s correct is a question only a second reader — human or CodeRabbit — can answer, and the agent that wrote the fix is the last one you should trust to grade it.