My Own Filter Hid the One Finding That Mattered

The gate said “3 green, 0 unresolved threads, ready to merge.” I was about half a second from clicking when GitHub itself refused, and that refusal is the only reason a critical bug didn’t ship.

Here’s the setup in plain terms. I was building a little robot referee for pull requests — a “pull request” being a proposed code change waiting for approval before it goes live. My robot’s job was to decide is this safe to merge yet? It combined two signals: the automated reviews from CodeRabbit (an AI reviewer that leaves comments on code), and GitHub’s own branch protection rules (the platform’s built-in safety latch). If both looked clean, ship it. If you’re not an engineer: I built a checklist, and I trusted my own checklist over the building’s fire alarm. Feel free to skip the next bit.

The way you count “unresolved” CodeRabbit comments is you filter its review threads. I wrote what felt obvious: count the threads where isResolved == false and isOutdated == false. My reasoning was that an “outdated” comment is one stuck to a line of code that has since changed — so surely it’s already been dealt with. Filter those out, and my count came back zero. Clean.

GitHub disagreed. Its mergeStateStatus — the coarse, one-word verdict on whether a branch can merge — sat stubbornly at BLOCKED while every check next to it glowed green. That mismatch is what made me stop. A green-everywhere screen with a red final verdict is not noise. It’s the tool telling you your model of “clean” is wrong.

So I went and read the thread I’d hidden from myself. It was a 🔴 CRITICAL finding: a fail-open path in auth code — the kind where, if a check errors out, the system defaults to letting you in instead of keeping you out. And it was marked isOutdated for the dumbest possible reason. An earlier commit had added a few lines higher up in the file, every line number below shifted down, and CodeRabbit’s anchor drifted. The finding was still completely valid against the current code. “Outdated” meant the lines moved. It did not mean anyone fixed anything.

That’s the trap, and it’s worth sitting with. My filter encoded an assumption — outdated implies addressed — and because I wrote both the filter and the assumption, the filter couldn’t catch the place where the assumption was wrong. It shared my blind spot exactly. The most dangerous bug in any verification system lives in the verifier itself, because that’s the one place nothing is checking your work.

GitHub’s gate saved me precisely because it was dumber than mine. require_conversation_resolution counts every unresolved thread, outdated or not. No cleverness, no assumptions to be wrong about. It fails closed.

reviewThreads(first: 100) {
  nodes { isResolved isOutdated }
}

# WRONG — excludes valid findings on shifted lines
unresolved = [t for t in threads
              if not t["isResolved"] and not t["isOutdated"]]

# RIGHT — outdated ≠ resolved
unresolved = [t for t in threads if not t["isResolved"]]

gh pr view "$PR" --json mergeStateStatus,statusCheckRollup
# gate on this, not your filter:
test "$(gh pr view "$PR" --json mergeStateStatus -q .mergeStateStatus)" = CLEAN

The lesson I keep now: when a coarse fail-closed gate contradicts your detailed green dashboard, the gate is probably right and your cleverness is probably the bug. Count isResolved == false and nothing else. “Outdated” is a fact about line numbers, not a verdict about safety — and a filter you wrote yourself can only ever be as honest as the assumption you hid inside it.