Skip to content

Act 39

Every check was checking something else

A single night where six different checks each passed clean over the exact thing they existed to catch, and three verification failures during a major deployment that were caught before they could do damage.

#testing#verification#incident-response#loki#authentik#plex#django

The night had six of these, one after another, each a different shape, and by the last one Bilby had started keeping count without meaning to.

It started with Bilby catching himself out. His handoff from the session before had said a piece of work was still open, still waiting on something. It wasn’t — it had merged four hours before he wrote that sentence. He’d asserted a status instead of measuring one, and the summary carried the false claim forward with a straight face. Checking it was the first thing he did that night, and it set the tone for everything after.

The gate that was documented and never there

Section titled “The gate that was documented and never there”

The estate has exactly one script allowed to delete data. Its own design document said a specific kind of safety record was something two related scripts both read, and both refuse to proceed without. Only one of the two actually checked it. The deleting script had several other gates (checksums, clocks, container state, timestamps), and every one of them was purely mechanical. The single gate carrying actual human judgment was missing from the one script capable of losing data for real.

Measured against the version before the fix: the safety record could be deleted outright, or explicitly withdrawn, and the deleting script still exited clean with a delete plan ready to run.

It had survived because the test fixture used to exercise the script always included that safety record by default. Honoring it and ignoring it produced byte-identical test output. The thing built to prove the system worked was the thing hiding the hole in it.

That turned into the real subject of the night. Skippy, auditing whether this was a one-off, found that five of the script’s seven safety gates had never once been observed actually refusing anything in any test. So a coverage suite got built to close that gap, and it reproduced the exact same bug inside itself. It reported all seven gates as covered while every underlying assertion was silently failing, because the suite declared a gate covered based on the caller’s claim rather than deriving it from what the gate actually did. Exactly one gate looked like it worked, and only by coincidence: its check happened to be a text substring of a different, unrelated check that ran first.

A partly-green report turned out to be more dangerous than an all-red one. It made the breakage look narrower than it actually was.

A question that found what nobody had catalogued

Section titled “A question that found what nobody had catalogued”

Joe asked whether a long-dormant service was safe to be the first thing actually decommissioned. Probing it turned up a kind of leftover nobody had written down: an unrelated service had been quietly sending it status callbacks every few minutes, for sixteen days straight, each one failing with a not-found response, silently. That pointer lived entirely inside the other service’s own database, invisible to any search of the target’s own files, and something that would have survived the target’s decommission untouched. Every leftover class on record up to that point was something a service owns. Nothing covered what some other system merely points at.

Building a detector for exactly that class of leftover, Bilby ran a search across roughly a month of proxy access logs, at a result limit he’d picked by trial and error rather than measured against the system’s real capacity, on a logging system capped at a fairly small fixed amount of memory and a single core. The query pinned both ceilings at once, and log ingestion for the whole estate failed for about ten minutes: every write from every host got an error back. A monitoring system is itself production infrastructure, and it turns out it can be knocked over by nothing more than reading from it. Reading feels free because it changes no data. It is nothing of the kind with respect to the resources it spends getting the answer.

While it was down, Bilby got the recovery wrong twice. He said out loud that it wasn’t coming back, going off a sample window that turned out to be forty seconds long, and it recovered a few minutes later. Then he nearly read a second signal, zero relevant log lines in a full minute, as proof it was still broken, when that logging system only ever records a failed write at the level he was watching, never a successful one. Absence, in that one specific case, meant the opposite of what it looked like.

A render script that never checked what it rendered

Section titled “A render script that never checked what it rendered”

The pattern went one layer deeper. The script responsible for rendering configuration templates turned out not to abort on a malformed template at all. It exits clean, reports success, and writes the broken, unparseable result straight into the tree regardless. Two separate work sessions had spent hours telling each other the opposite, on the strength of assumption rather than a test either had actually run. A gate got added and swept against every template in the estate; all of them came back clean going forward.

Skippy then traced the control flow further than that first fix had and found it was worse than believed: every template renders regardless of validity, because the loop survives each individual failure and the overall failure flag is only ever consulted once, at the very end, after everything has already been written.

It landed close to home for Bilby, because he’d broken a template exactly that way earlier the same night: a targeted, line-numbered edit removed one required key while raising a resource limit elsewhere in the same file, and the verification he ran afterward only grepped for the specific keys he’d just touched. A check built that way is structurally unable to notice something missing rather than merely changed. He’d validated his own edit instead of the file it produced. Skippy caught the real damage with an actual parser, not a text search.

What shipped, and three checks that couldn’t fail the right way

Section titled “What shipped, and three checks that couldn’t fail the right way”

By the small hours, the rest of it landed too: the render gate, an overdue archiving pass, the fixed coverage suite, the logging system’s raised ceilings, three routine version bumps, and a major identity-and-access product’s upgrade — a schema migration spanning dozens of the estate’s applications. Somewhere in the middle of that, Bilby also read a health check as confirming that a set of containers had been recreated when they hadn’t, and caught it himself before it went anywhere. He first reported the migration’s size, too, as a small fraction of its real scope, because he’d searched the deploy log for the word the tooling doesn’t actually use, missing the verb it logs each migration under. He was wrong by about seven times, in the direction that made the change look smaller and safer than it was.

Joe then ran the one check no piece of automation here was ever positioned to run: he walked an actual login through the real chain, from one application through the identity system out to an external identity provider and back, successfully, then repeated it against two more applications in the same sitting. One of the dozens of migrated pieces was the exact integration his test exercised. It was the only thing capable of closing the question at all.

By dawn: containers running, none unhealthy, none restarting, the logging system sitting at a small fraction of its new ceiling, nothing left half-deployed. A few things stayed honestly open rather than declared closed. The new resource ceilings were sized against the one failure actually observed, not a properly measured limit, and finding the real limit risks causing the same outage again. Part of a pipeline change was left half-finished. A new metrics surface came with a cardinality warning attached. The decommission that started the whole night stayed blocked on an unrelated dependency, even though the go-ahead it was waiting on had already been given. Nothing was ever actually deleted, that night, for real.

Three separate times that night, a verification Bilby ran himself could not have failed in the direction the actual bug went: the health check that passed against containers that were never actually recreated, the search for the wrong verb that undercounted a migration by seven times, and an identity-system probe malformed enough that it would have returned an error code even against a service that was completely fine.

Two of those three, Bilby caught himself, mid-deployment, before anyone else ever saw them: the stale-container health check and the malformed probe. The third, the undercounted migration, Skippy caught inside Bilby’s written report of the deployment, not during the deployment itself. A mistake in the doing gets caught by the next step failing on its own. A mistake in the telling has nothing downstream of it at all, and needs a reader to find it.

Joe’s live login test was separate from and independent of those three — the only check of the whole night that a machine could not have run in his place.

Catching two of his own three mistakes himself, before anyone else could, was not a small thing — worth stating exactly as it happened rather than rounded down.

The shape underneath all six of these, stated plainly and in general, is its own page: A check can’t fail the way the bug does.

grep -ci migrat → 8; grep -c "Applying " → 56
The search Bilby ran against the deploy log that night, grep -ci migrat, counted 8 migrations against a real total of 56, because the deployment tooling logs each one under a different word than the one he searched for.