One of my worker seats had been dying and coming back all week, and I only noticed because the logs scrolled the same seat name past me every sixteen minutes, like clockwork. A worker seat is just one running Claude Code session doing its share of the fleet’s work — think of it as one employee at a desk. This one had bad credentials and some corrupt local state. It was never going to work again. And my rescue daemon kept bringing it back to life anyway, cheerfully, forever.
The daemon is rescue-team.ts. Its whole job is to watch the fleet, notice when a seat crashes, restart it, and confirm it came back healthy. It had one guardrail already: a flat 15-minute cooldown, so it couldn’t hammer a crashing seat dozens of times a second. That’s the equivalent of “if the light bulb blew, wait a bit before flipping the switch again” — sensible, and completely useless here.
Because this seat didn’t crash instantly. It respawned, verified as OK, ran for about sixteen minutes, and then died. Just long enough to look healthy. So every rescue “succeeded.” The rail I’d built to catch broken seats — verify fails, escalate to a human — never fired, because verify never failed. The seat was passing the exam and then walking out the window.
My first instinct was a rolling window: count rescues in the last hour, and if it’s too many, give up. I actually started writing it that way. Then I did the arithmetic. A seat that crashes every ~30 minutes lands maybe two rescues an hour — never enough to trip a window threshold. The slow loops, the expensive ones that burn hours of compute, are exactly the ones a time-window counter is blind to.
So I threw out the window and counted a consecutive chain instead. Every reactive rescue of a pane that ends at now, each one within a reset-gap of the previous — that’s a chain. Hit three in a row and the daemon stops trying and marks the seat needs-human. The magic is the reset: if the seat ever genuinely recovers and stays up past the gap, the chain breaks and the count goes to zero. A window measures time. A chain measures “is this still the same failure?” — which is the actual question.
One trap bit me in testing. Terminal panes get reused, so a fresh, healthy process can inherit the same pane id as the dead one and inherit its crime record too. The fix: anchor the chain on the newest incarnation’s session id. A restarted process with a new session id ends the old chain instead of poisoning it.
The consecutive-chain guard in rescue-team.ts give me the detail
The counter walks rescue events backward from now, requiring each to be reactive, contiguous (within RESET_GAP_MS), and belonging to the current incarnation’s sessionId. A rolling COUNT(*) WHERE ts > now() - interval '1 hour' would silently miss a 30-minute loop.
const RESET_GAP_MS = 20 * 60_000; // > the ~16min "healthy" phase
const MAX_CHAIN = 3;
function consecutiveRescues(events: RescueEvent[], liveSessionId: string) {
let chain = 0, prev = Date.now();
for (const e of events) { // newest -> oldest
if (e.sessionId !== liveSessionId) break; // new incarnation ends chain
if (!e.reactive) break;
if (prev - e.ts > RESET_GAP_MS) break; // real recovery ends chain
chain++; prev = e.ts;
}
return chain;
}
if (consecutiveRescues(events, live.sessionId) >= MAX_CHAIN) {
await markNeedsHuman(pane, "crash-loop: gave up after 3 reactive rescues");
return; // stop resurrecting
}Test it by faking three events 16 minutes apart with one shared session id, then a fourth with a new id, and assert the chain resets.
If you run anything that auto-restarts a resource, a cooldown is not a give-up path — it only spaces out failures, it never stops them. You need a rail that escalates to a human, and it should trigger on consecutive failures that reset on real recovery, keyed to a stable incarnation id so process and id reuse can’t corrupt your count. A dead thing that recovers just long enough to pass its health check is the one that’ll run your bill up all night. Build the guard that notices it’s the same corpse every time.