← All posts

Zombie Agents: When the Watchdog Isn't the One Doing the Killing

Five of my review-fix agents kept iterating after their pull requests had already merged. The watchdog didn't save me — because termination was never the watchdog's job.

  • autonomous-agents
  • claude-code
  • reliability
  • infrastructure
  • ai-engineering

I found five agent loops still grinding away on pull requests that had merged hours earlier.

Some background for anyone not knee-deep in this: I run autonomous dev workflows — I call them ADWs — where a Claude Code agent watches one of my code changes (a “pull request,” basically a proposed edit waiting to be accepted), fixes whatever a reviewer flags, and repeats until the change gets merged. Merged means done. The agent should stop.

These five didn’t stop. The pull requests were long gone, closed and shipped, and the agents were still politely asking “any new review feedback?” and burning compute answering their own question. Zombie loops.

My first instinct was that my watchdog had failed. I run a separate little daemon whose whole job is to notice stuck processes and reap them — the smoke detector in the hallway. So I went to fix the smoke detector.

That was the wrong turn. The watchdog was working fine. The problem was that I’d made it load-bearing — I’d quietly decided that stopping was someone else’s responsibility, and the someone was an external process that only wakes up on obvious pathological crashes. A merged pull request isn’t a crash. It’s a perfectly healthy agent doing exactly what I told it, forever.

When I actually read the code, three holes lined up like a Swiss-cheese diagram.

The outer loop did check whether the pull request had merged — but only between iterations. Nested inside was a ten-minute poll waiting for new review comments, and that inner loop had no idea a terminal condition even existed. So the agent would enter the poll, sit for ten minutes on a dead pull request, come back up, and only then notice the party was over. Usually it noticed. Sometimes the timing meant it re-entered before checking.

Worse: some agents had launched with no pull-request number at all. Instead of refusing to run, they’d shrugged, defaulted to “keep looking,” and looped with nothing to look for. A missing required input got treated as a reason to wait instead of a reason to stop.

And there was no hard ceiling. No line anywhere said “no matter what, you’re done after 90 minutes.”

The fix is a mindset, not a patch: a long-running agent has to be able to kill itself. Termination can’t live in a neighbor.

Making an agent self-terminate give me the detail

Three rules, applied to every ADW loop:

1. Terminal-condition check at the top of every nested loop — not just the outer one.

while not deadline_exceeded():
    if pr_is_terminal(pr_number):   # merged, closed, missing
        return
    # ...ten-minute review poll lives here...
    for _ in range(POLL_CYCLES):
        if pr_is_terminal(pr_number):  # re-check inside the poll
            return
        time.sleep(POLL_INTERVAL)

2. Fail-fast input validation at step entry — raise, never default.

def run_review_fix(pr_number: int | None):
    if pr_number is None:
        raise ValueError("review_fix requires a pr_number; refusing to loop")

An agent with nothing to work on should die loudly, not wait quietly.

3. A hard wall-clock ceiling beneath the retry budget.

DEADLINE = time.monotonic() + 90 * 60   # 90 minutes, full stop
def deadline_exceeded() -> bool:
    return time.monotonic() > DEADLINE

The retry count is the soft limit. The clock is the wall. The watchdog daemon stays — but now it’s belt-and-suspenders for real crashes, not the thing keeping compute bills sane.

The mechanism matters because external watchdogs can only recognize failures they were taught to recognize, and a healthy agent stuck in a valid-but-pointless loop looks like nothing is wrong. The only process that always knows the job is over is the one doing the job.

So if you run agents that wait on external state, don’t ask “what will stop this if it hangs?” Ask “how does this know it’s finished?” — and make it check at the top of every loop, refuse to start without its inputs, and carry a clock it can’t outrun.