Verification Gates for AI-Generated Changes

When an automated agent proposes a change, “the tests pass” is a dangerously weak signal for merging it. Green CI is consistent with fixing the wrong thing — the right build succeeding against the wrong line. If a human has to re-verify every automated fix anyway, the automation hasn’t actually saved the work; it has just moved it.

The fix is a deterministic verification gate that runs after the change is built but before it merges, and emits a verdict as a pure function of the artifact’s own evidence — no model in the scoring path. Useful checks include: does the change reference the same issue it was dispatched for; does the summary claim success while the logs contain failure evidence; is the diff actually non-empty and touching the relevant code; and does a reproduction check fail against the pre-fix state.

Treat “couldn’t get the evidence” as its own outcome, distinct from pass. If the transcript is missing or an API errored, the gate should return a blind spot, and a blind spot must never be silently coerced into approval. Not-run is not the same as passed.

Finally, make enforcement a tunable policy — off, advisory, or strict — per surface, rather than a blanket freeze. Start in advisory mode so the gate earns trust by commenting its verdict, then flip the load-bearing paths to strict once it’s calibrated. Safety belongs in the flow, not as a wall across it.