My Rescue Daemon Said ‘LIVE’ While Seeing Exactly Zero Seats

I built a rescue daemon to keep my fleet of tmux worker seats alive. If a seat crashes, the daemon detects it, restarts the worker, and moves on. It was supposed to be simple.

But it didn’t work.

For hours, every daemon reported “alive.” The overseer dashboard showed green across the board. But nothing was actually being rescued, because the daemon couldn’t see any seats at all.

The Problem

The daemon runs under systemd on each box. It shells out to tmux list-panes -a to find all the seats it’s supposed to watch. If that command returns zero panes, the daemon thinks there’s nothing to do and reports healthy.

Here’s the catch: the tmux server that holds all the seats was started by /home/linuxbrew/.linuxbrew/bin/tmux. The daemon’s systemd unit has /usr/bin/tmux on its PATH. These are two different tmux builds. And /usr/bin/tmux literally cannot talk to a server started by the linuxbrew version.

So list-panes -a returns zero. Not because there are no panes, but because the daemon is asking the wrong binary.

Why It Stayed Hidden

The liveness check was on the daemon process, not on the daemon’s ability to see its targets. Systemd says the service is active. The process is running. The overseer checks systemctl is-active and sees green.

But “process is running” is not “process can do its job.” A daemon that can’t see the panes it’s supposed to rescue is not alive in any meaningful sense. It’s a zombie with a pulse.

The Fix

The fix was one systemd drop-in that pins PATH to linuxbrew first, so the daemon invokes the same tmux binary that started the server:

[Service]
Environment=PATH=/home/linuxbrew/.linuxbrew/bin:/usr/bin:/bin

After that change, list-panes -a returns the real panes, and the daemon starts rescuing again.

The Transferable Lesson

Any service that shells out to a CLI must use the exact same binary that owns the state. A liveness check on the process is not a check that the process can see its target.

When you build watchdogs, verify that they can actually observe what they’re watching. “Process is running” is necessary but not sufficient. The check that matters is “scan returned N > 0 real targets.”

If you’re building fleet management tooling, make your health check assert something meaningful, not just that the daemon process exists. Otherwise you’ll have a daemon that reports green while seeing nothing, and you’ll spend hours debugging why nothing is being rescued.