On July 10 I woke up to twenty pending escalations that had been sitting in a queue since roughly ten the night before. Nothing had crashed. Nothing had errored. The supervisor whose entire job was to interrupt me had spent eight and a half hours failing to, quietly.
Here’s the setup, in plain terms. I run a supervising agent that watches the workers doing actual tasks. When a worker needs a human — a product call, an ambiguous requirement, something it isn’t allowed to decide — the supervisor buffers that request and then injects it into my session: it types the message into the pane I’m sitting in. A pane is just one terminal window among many. If it can’t confirm that the message actually submitted within a time limit, the pane is what I call wedged — stuck in a state where input goes in and nothing comes out.
Wedged is a known failure. I’d handled it. When injection failed past the deadline, the code raised an alarm.
The alarm was one line, and it looked like this:
if [ "$backend" = tmux ]; then
tmux display-message "escalation injection wedged"
fiI had moved that particular supervisor onto a different session backend.
The alert lived inside the thing it was alerting about
display-message is a tmux status-line flash — a bit of text that blinks along the bottom edge of a tmux window. It’s a fine nudge when you’re looking at tmux. It has no equivalent anywhere else, which is exactly why it was guarded by that condition in the first place. The guard was correct. The problem was that the guard was the only active path.
So on any non-tmux backend, the alarm branch evaluated to nothing at all. What still happened was a marker file getting written to disk: a small .subsuper-inject-wedged file recording that yes, this went wrong.
Nothing reads a marker file until something else goes looking. That’s the whole property of a marker. It’s a note left on a desk for whoever sits down next, and nobody sat down until morning.
I want to be precise about the failure, because I got it wrong at first. My instinct was “the alarm didn’t fire.” It did fire. inject_wedge_alarm ran, on schedule, exactly as designed. It just had nothing to do on my platform. A function that runs to completion and produces no observable effect is worse than one that crashes, because the crash would have told me.
A marker is not an alarm
That’s the rule I pulled out of it, and it’s the one I’d keep if I had to throw the rest away.
A marker is passive: it waits to be read. An alarm is active: it goes and finds you. They are not two strengths of the same thing. They are different categories, and a system that has only markers has no alerting at all — it has a log with good intentions.
The tell is a question you can ask about any notification path in about five seconds: if every screen I own is unreadable right now, does this still reach me? If the honest answer is no, it isn’t an alarm.
My tmux flash failed that test twice over. It needed the right backend, and it needed my eyes already on the right window. Two preconditions, both of them things I control and change without thinking about it.
What replaced it
The fix wasn’t a better flash. It was moving the active channel off the pane entirely, so that it can’t share a failure mode with the thing it’s reporting on.
The alarm now resolves a channel that doesn’t depend on any session, any backend, or any status line: an operating-system notification on macOS, or a configured command that can hand the summary to a phone — a push service, a chat message, an SMS. The old marker file still gets written and the tmux flash still fires where tmux exists. Those weren’t removed. They were demoted to what they always were: supporting evidence, not the alert.
Three details mattered more than I expected:
- Every channel is best-effort and the loop keeps going. A missing binary or a non-zero exit logs a warning and falls through to the next channel. An alarm that can crash the supervisor is a new outage, not a fix.
- Every notifier is time-bounded. Ten seconds, process-group bounded, watchdog kills the group on timeout. A notifier that hangs is indistinguishable from no notifier, and it takes the daemon down with it.
- It’s on by default. Not opt-in. The entire purpose of this thing is that a wedged supervisor is never silent, so the reachable channel fires unless I explicitly turn it off. It’s rate-limited to once per deadline window and only fires after a genuine wedge, so “default on” doesn’t mean chatty.
That last one was the argument I had with myself. Default-on alerting feels rude. But an opt-in alarm is a feature that protects you only if you already knew you needed protecting — and if I’d known, I wouldn’t have had the incident.
The shape of this bug is everywhere
Once I had the name for it I started finding it in adjacent places. It’s the same shape as my rescue daemon reporting LIVE while it could see nothing — a component that passes its own health check while being structurally unable to do its job. It’s the same shape as a dashboard that shows activity instead of state. And it rhymes with treating silence from a reviewer as approval: in all three cases the absence of a signal got read as the absence of a problem.
The general version, which is the thing I’d actually write on a wall:
Every escape hatch needs a path that does not run through the system it is escaping.
If your alerting rides on the same session, the same host, the same process, or the same terminal as the work it watches, you don’t have alerting. You have a hope that the failure will be polite enough to leave that one part working.
Eight and a half hours of buffered escalations is a cheap way to learn it. It could have been a week.