A Check You Never See Fail Is Already Dead

For weeks, a sentiment collector on my fleet reported a clean exit on every scheduled run while the Python inside it failed every single time. The wrapper was the culprit: the task ran cmd /c script >> log 2>&1, and cmd returns the exit code of the redirection, not the program. The redirect always succeeded. The job was dead and telling me it was healthy, twice a day, on schedule.

Nobody investigates a green check. That is the whole problem. A red result gets a ticket, a log dive, a fix by Friday. A false green gets trusted, and it compounds: every day of "success" is another day the gap between what I believed and what was running got wider. When I finally noticed, the data the collector was supposed to be gathering had a multi-week hole that no amount of fixing could backfill.

It gets worse, and the worse part is the lesson. I had a watchdog for exactly this: a weekly job that checked whether scheduled tasks were running on cadence. It never flagged the dead collector, and the reason is painful. The watchdog was one of the broken jobs. Its silence read as health. I had built a checker and never once seen it fail, and I took that as evidence it had nothing to report. A check you never see fail is indistinguishable from a check that is dead.

Three signals, because any one of them lies#

The rebuilt job-health system refuses to decide health from a single signal, because each signal has a documented way of lying to me.

An exit code lies the way the collector's did: the wrapper reports its own success, not the program's. So results are read with the scheduler's vocabulary in mind, where "has not run yet," "still running," and "battery refusal" are distinct non-failures, and a zero from a cmd redirect is treated as a claim, not a fact.

Configuration lies by rotting. A task can point at a script that moved, a user profile that no longer exists after a machine migration, or the Microsoft Store's python.exe stub, which exists, launches, and never runs anything. Config rot is invisible to result checks because the task often "succeeds" at running the wrong target.

The catcher for both is staleness: a job that claims success but has not actually executed within its own trigger cadence is lying somewhere, and the staleness check does not care where. It compares the claim against the calendar. Of the three signals, this is the one that would have caught the collector in week one, because no signal a broken job emits can fake the passage of time.

One signal is a guess. The triangle is a verdict.

The checker gets checked, on a schedule#

The watchdog failure taught me that layering checkers is not enough, because the layer on top inherits the same disease: I will never see it fail either. So the fleet now runs a weekly job whose only purpose is to plant a failure. It registers a deliberately broken scheduled task, asserts that the watchdog flags it, and then cleans up. If the watchdog ever stays quiet about the plant, the watchdog is the incident.

I wrote about the code-layer version of this in the Ballast post: a secrecy scanner that skipped its own source file, and therefore published the exact string it existed to suppress. Same disease, same cure. The scanner now scans itself, and the eval suite includes cases the scanner must fail. A check that cannot fail on what you care about is a green light wired to the wall.

The gate that passed while the bundle was broken#

Three-signal health still misses one version of the false green, and it bit me this month. A packaging gate on one of my APIs hard-fails a Lambda bundle that contains Windows wheels, because a Windows-built bundle once took down every function behind the API. The gate worked. Then a bundle built on modern Linux shipped wheels compiled for a newer glibc than the Lambda runtime has, and died at import with the same total outage. The gate passed that bundle without complaint. It was checking the failure mode I had already met, not the one I was about to.

"The gate passed" is only as meaningful as the gate's scope. A passing check with the wrong scope is a false green with extra steps, and the fix is the same discipline pointed at coverage instead of liveness: when a new failure mode gets past a gate, the gate grows a case for it the same week, and the case is proven by planting the bad artifact and watching the gate reject it. Both directions matter. The gate has to fail on what I fear, and I have to see it fail.

Name how you would find out#

The guardrails post ends on an open question: pre-execution guardrails protect the platform, but nothing in that pattern protects the checks themselves from rotting. This is the answer I use, and it fits in two sentences.

For every check that guards something you care about, name the specific way you would find out it died. If the honest answer is "someone would notice," schedule the plant: a deliberate failure, on a calendar, that the check must catch, so that silence stops being ambiguous.

I used to read my fleet's dashboards as reassurance, all that green stacked up like evidence. The dangerous state was never red. It is a green you have stopped deserving.