Around 2am I was rolling out little rescue daemons — background babysitters — one per box across a fleet of machines, each running a pile of Claude Code agent sessions inside tmux. tmux is the tool that keeps terminal windows alive after you disconnect; think of it as leaving your programs running in a room and being able to walk back in later. Each agent lives in one of those windows, which I call a seat.
The daemon’s whole job: scan every seat, notice when one has gone dead or hit a wall, and relaunch it. My overseer dashboard showed every daemon LIVE, green across the board. I went to bed happy.
The daemons had been blind for hours.
Here’s the part that got me. The daemons ran under systemd — the Linux service manager that starts things at boot and restarts them if they die. Under systemd they found tmux at /usr/bin/tmux. But the tmux server actually holding all my seats had been started earlier by a different tmux, the one from linuxbrew (Homebrew for Linux) at /home/linuxbrew/.linuxbrew/bin/tmux.
Two different builds of the same tool. And they cannot talk to each other — a tmux client only speaks to a server made by its own build. So when my daemon ran list-panes -a to enumerate every seat, it got back zero. Not an error. Zero. An empty universe.
To the daemon, there was nothing to rescue, so it rescued nothing and reported success. The health check only ever asked am I running? — and the process was running fine. It just couldn’t see the thing it existed to watch.
It gets worse. A follow-on “balance” routine — an autonomous bit that trims idle seats to free up capacity — looked at that same empty scan, decided three seats were idle, and killed them. Their relaunch failed. So my safety system deleted live work while wearing a green badge.
The fix was small: a systemd drop-in pinning PATH to linuxbrew first, so the daemon shells out to the same binary that owns the state.
Pin the binary, then assert the scan is non-empty give me the detail
Check the divergence directly — your shell vs. the service:
command -v tmux # /home/linuxbrew/.linuxbrew/bin/tmux
systemctl --user show-environment | grep PATH # starts /usr/bin — the trapThe drop-in (~/.config/systemd/user/rescue.service.d/path.conf):
[Service]
Environment=PATH=/home/linuxbrew/.linuxbrew/bin:/usr/bin:/binThen make the health check assert targets seen, not process up:
n=$(tmux list-panes -a 2>/dev/null | wc -l)
[ "$n" -gt 0 ] || { echo "BLIND: 0 panes"; exit 1; }A daemon that scans zero should fail loud, not report healthy.
The lesson I keep now: any watchdog that shells out to a CLI — tmux, docker, kubectl — has to invoke the exact binary that owns the state, and its health check must prove it can see something real, not just that it’s breathing. Green should mean “I found N targets,” never “I’m alive.” A blind daemon that reports healthy is worse than one that’s plainly dead — because you trust it.