The Comment Count Was Lying to Me

A pull request merged one night with two unresolved CodeRabbit MAJOR threads still open — the kind of comment (“this can throw on null”) you’re supposed to fix before shipping, not after. Nobody overrode anything. The gate just didn’t see them.

Here’s the setup, for anyone who isn’t knee-deep in GitHub. One of my automations writes code and opens pull requests — a “PR” is just a proposed change waiting to be accepted. A second Claude Code agent runs a night-shift audit over those PRs: it reads the code, checks the automated review bot’s comments, and posts a summary so I can approve the merge in the morning. I’m the human at the end of the chain, half-asleep, saying yes or no async.

The bug was in how the audit agent counted. It asked GitHub’s REST API for the PR’s /comments — and got a number. Zero unresolved-looking comments on the snapshot it took. I skim-read that snapshot as “clean,” treated it as a merge ack, and the PR went in.

The problem: that count doesn’t mean what it looks like it means.

GitHub’s REST comment count tallies top-level comments on the pull request. It says nothing about review threads — the little resolvable conversations attached to specific lines of code, the ones with a “Resolve conversation” button. A thread can be wide open, blocking, unaddressed, and still not move that REST number at all. It’s like checking whether a restaurant is busy by counting cars in the lot when all the diners took the train. Wrong instrument, confident answer.

The right instrument is GraphQL’s reviewThreads, which actually knows each thread’s state. You filter on isResolved == false AND isOutdated == false — not-resolved and not-stale — and that count has to be 0. Outdated matters because a thread against code that’s since been rewritten is noise, not a blocker.

query($owner:String!, $repo:String!, $pr:Int!) {
  repository(owner:$owner, name:$repo) {
    pullRequest(number:$pr) {
      mergeStateStatus              # want CLEAN
      reviewThreads(first:100) {
        nodes { isResolved isOutdated }
      }
    }
  }
}

# blocking = threads that are neither resolved nor outdated
gh api graphql -f query="$Q" -F pr=$PR --jq \
  '[.data.repository.pullRequest.reviewThreads.nodes[]
    | select(.isResolved==false and .isOutdated==false)] | length'
# must print 0 — checked at the merge gate, not at a baseline

The deeper mistake wasn’t the wrong endpoint. It was trusting a snapshot. A baseline taken at 2am and a merge decided at 8am are two different worlds — threads open and resolve in seconds. So now the check runs at the gate, right before the irreversible action, every time.

And I changed what the agent is allowed to say. It no longer recommends “merge.” It surfaces three facts — live threads, CI, merge status — and lets me pull the trigger. If your automation is about to do something you can’t undo, make it re-verify at the last second, and make the human the one who commits.