For about a day, a voice notification channel I keep for real emergencies read me the same line every two minutes: “Recovery at risk.” It was coming from every headless Linux box in my agent fleet — the machines that run Claude Code sessions with no screen attached, just processes chugging in the background. And every one of them was fine. The backups it was panicking about were current, correct, and would have restored perfectly.
Here’s the setup in plain terms. I use tmux-resurrect, a tool that snapshots my terminal sessions to disk so that if a box reboots, the work comes back exactly where it left off — like a save file in a video game. On top of it I wrote a little backstop script whose job is to notice if those saves ever stop happening and yell at me. That yell was the voice channel.
The backstop decided “saves stopped” by looking at the snapshot file’s timestamp. If the newest snapshot was older than a cutoff I called STALE_AFTER, it assumed the save machinery had died. Reasonable-sounding. Completely wrong.
What I didn’t know is that tmux-resurrect’s save_all deduplicates. Every save writes a fresh timestamped snapshot, compares it to the previous one, and if the contents are identical, it deletes the new one and keeps the old. So a stable box — one whose sessions aren’t changing — legitimately holds a snapshot with an old timestamp whose contents are perfectly current. The tool was working exactly as designed. My monitor was reading the design as a failure.
The timestamp was never the health signal. It was noise dressed up as a signal.
The fix was to judge health the way the system actually defines it: did the last save succeed (exit code 0), and does the snapshot contain real content — actual pane rows I could resurrect from? A current-but-old file passes both. A truly broken box fails the exit code. Timestamp age tells you nothing about either.
Why the exit code kept lying too give me the detail
There was a second trap. My first attempt captured the save’s return code through tmux run-shell, which runs asynchronously — it fires the command and returns its own success, swallowing the inner script’s actual rc. So even after I stopped trusting timestamps, my “did the save succeed” check was reading tmux’s happiness, not the save’s. The backstop has to run the save script directly:
# WRONG: async wrapper hides the real rc
tmux run-shell "$RESURRECT_DIR/save.sh" # always looks fine
# RIGHT: run it inline, capture the real exit code
"$RESURRECT_DIR/save.sh"; rc=$?
snapshot="$(latest_snapshot)"
if [ "$rc" -ne 0 ] || ! grep -q ':pane' "$snapshot"; then
alert "recovery at risk on $(hostname)"
fiThis regressed twice — once in PR #419, again after I thought I’d killed it, in #553 — because both fixes drifted back toward the timestamp as a convenient proxy.
If you monitor any backup, snapshot, or sync system, don’t equate an old file with a dead one. Anything that deduplicates by content will hold a stale-looking file on purpose, and it’s the healthy state, not the sick one. Ask two honest questions instead: did the last write succeed, and does the file contain what you’d need to actually recover? And check that nothing async is standing between your monitor and the real exit code — a wrapper that reports its own success is worse than no monitor at all, because it lies with confidence.