Last Thursday afternoon, five of my Claude Code agents hit their rate-limit walls at the same time. I didn’t panic. The rescue daemon would catch them — it always did.
It didn’t.
I sat there refreshing tmux, watching stuck sessions and a daemon that had quietly gone dark. The thing I’d built to save my agents from rate limits had died of a rate limit. It was, in retrospect, the most predictable failure I’ve ever engineered.
Here’s the architecture, and the mistake embedded in it.
I run multiple Claude Code sessions in parallel — autonomous coding agents, each living in its own tmux pane, each pointed at a different Anthropic account. The constraint that shapes everything is rate limits. Each account gets a rolling 5-hour budget of API calls and a weekly cap. When a session burns through its budget mid-task, it’s walled — it just stops. Often in the middle of writing code, debugging something, halfway through a thought.
So I built a rescue daemon. It polls every session’s remaining quota. When one gets walled, or is about to be, the daemon grabs that session’s conversation history and restarts it on a different account that still has headroom. It uses Claude Code’s --resume flag, so the agent picks up exactly where it left off — same context, same task, fresh budget. For weeks, this was beautiful. An agent would hit a wall, blink out for maybe sixty seconds, and come back running on a new account. None the wiser.
Then came the afternoon when a bunch of sessions hit their limits at once, and the daemon didn’t save any of them. It was dead too.
The reason is almost funny, in the way that watching your own architecture collapse under a weight you built into it is funny: the rescue daemon authenticated against the same pool of accounts it was rescuing. When the pool got tight enough that sessions started getting walled — which is exactly the moment the daemon exists to handle — the daemon’s own account got walled mid-rescue. Now the stuck agents were still stuck, the rescuer was down, and the rescue attempt had burned the last scraps of headroom something else could’ve used. The system was strictly worse with the daemon than without it.
I had built a healer that shared a failure domain with its patients. When the patients got sick, so did the doctor.
The fix is obvious in hindsight, but I missed it because I was thinking about the daemon as infrastructure, not as another agent in the same constrained pool. The daemon now gets a reserved account that the worker pool never draws from, plus an explicit fallback to a cheaper, separate API substrate if even that reserved account runs dry. The startup assertion is one line — the healer’s account must not be a member of the pool it monitors.
The general pattern outlived this specific bug. It applies to any self-healing component: a watchdog, a failover controller, a circuit breaker’s recovery path, a backup job. Ask one question before you trust it: does the healer depend on the exact resource it’s trying to heal? If the answer is yes, you don’t have a self-healing system. You have a single point of failure in a rescue costume. The healer has to keep working precisely in the condition where everything it watches has failed. That’s the only condition that matters.
how the rescue daemon actually works give me the detail
The daemon polls each session’s status line, which reports the remaining 5-hour and weekly budget per account. A session counts as walled when its budget hits zero or the model starts refusing with a rate-limit notice. Recovery is a tmux respawn-pane that relaunches the same agent pointed at a different account’s CLAUDE_CONFIG_DIR, with claude --resume <session-id> so the conversation history carries over intact — same agent, same context, fresh budget.
The fix for the shared-failure-domain bug is isolation: the daemon gets a reserved account the worker pool never draws from, plus an explicit fallback to a cheaper, separate substrate for the moment even that is exhausted. The startup assertion is one line — the healer’s account must not be a member of the pool it monitors:
assert rescuer.account not in worker_pool, "rescuer shares the failure domain it heals"Run the daemon itself under systemd with Restart=always so it survives its own host hiccups — not as a thread inside the very process it’s meant to revive. And the test that would have caught this before production: drain the pool on purpose and confirm the daemon still runs. If it can’t act with the pool exhausted, it was never going to help you on the day you needed it.
Every self-healing component you build has a dependency graph. Trace it against the failure it’s supposed to survive. If they overlap, the day they overlap is the day you’ll need it most — and the day it won’t be there.