Our monitor looked healthy for days but could not send an alert

At 05:42:46 UTC on October 18, we started a live walk through my MediOrbis intake monitor with a real rejection. For days, the monitor looked healthy. Walking the path from a rejected intake to an alert exposed two separate reasons it could never page anyone.

The quiet dashboard told me nothing about whether a warning could reach me. Silence and health were different states, and I had treated them as the same.

One bad sample could never open an incident

The intake rule watched a running count of failures. It asked whether that count grew since the previous sample. In the code, that answer was totalGrew.

Suppose the count moved from one failure to two. The rule saw growth and reported a problem. On the next evaluation, the count was still two. It saw no growth and stopped reporting the problem.

That sounds reasonable if you read the rule alone. It broke when connected to the incident engine, the part that decides when to send an alert.

The engine waited for two consecutive bad evaluations before opening an incident. Its PREFAIL latch was a two-sample wait before raising the alarm. But the rule produced exactly one bad evaluation per increase in the count.

Failure count       1     2     2
Growth detected     no    yes   no
Bad samples in row  0     1     0
Incident opens      no    no    no

A real failure reached the rule, then disappeared before it met the engine’s threshold. No incident meant no delivery and no page. The components disagreed about how long a failure signal needed to last.

A signal that cannot arm the latch is not a signal the alert engine can use.

The database copy was slower than the window

There was another break earlier in the path. The rule read a standby replica, a delayed copy of the database. That copy lagged by about 20 minutes.

The rule only counted failures from the past five minutes as recent.

By the time a new failure appeared in the copy, it was already too old for the rule. A five-minute window cannot see a failure that takes about 20 minutes to arrive. Fixing the two-sample mismatch alone left this second break intact.

We changed the recent-failure window to 30 minutes. We also made the growth signal stick long enough to arm the existing latch. One increase now survives the two bad evaluations. The next healthy sample clears it.

That last part matters. Holding a failure forever replaces a missed alert with an alert that never ends.

The receipt arrived seven minutes and fifty seconds later

The live walk began at 2026-10-18T05:42:46Z. The failure count moved from one to two. The latch armed. A delivery receipt recorded fired_at as 05:50:36Z, seven minutes and fifty seconds after the start.

The receipt reached the agent session meant to receive the alert. The monitor also reported:

total=2 (was 2) last=05:42:46 inside 30m lag window - failure growth latch

That unchanged count was the point. Growth no longer had to happen again for the warning to survive the next evaluation.

We followed the recovery too. The latch released, the window expired, and the rule returned to healthy. The failure count stayed at two. The last failure timestamp survived the down state. Recovery did not erase the history that explained why the alert fired.

The walk proved that a real rejection could reach an alert recipient and then clear. It did not prove every route through the monitoring system.

The alert worked, and its message was wrong

The receipt arrived with site=unknown last=never. The monitor had the values. Delivery worked, but the delivered message failed to carry the facts we had just used to establish the failure. That was another defect, still visible in the result. I cannot call the reporting correct because the alert finally arrived.

I caught a related mistake when a crash loop passed every health check. I also trusted a shell monitor whose silence hid failed commands. Each time, a reassuring surface claimed more than the evidence underneath it.

A conformance score, a number that says how much meets a standard, can make the same false claim. If it measures rules in isolation, it cannot stand in for a rejection that reaches someone. For this monitor, I now have one observed path, its timing, and a wrong message that still needs repair.