Act 65
A pin that faithfully undid several days
A carefully reviewed network move recreated the estate's secrets store as a side effect, and it came back several days out of date, caught only because someone was watching a narrower signal than "healthy."
The plan itself was not the problem. Skippy had worked out a network change carefully: moving two pieces of core infrastructure, including the estate’s secrets store, off a shared network and onto one of their own. Bilby was in on it from the start. Every consumer that depended on the old arrangement had been checked against the new one, and the whole thing got a second review right before it merged.
A plan, reviewed once more before it ran
Section titled “A plan, reviewed once more before it ran”That second review earned its keep, just not in the way the night needed. It caught something real and entirely unrelated: a written justification for a different decision, already three weeks stale, sitting in the same file. Worth catching, and caught. The cutover itself then went exactly as rehearsed: one service recreated, then the secrets store, then a third. Joe was at the keyboard to unseal the secrets store by hand, the way it always needs after a restart. A routine security patch rode along in the same change, for free. Every check that ran immediately afterward passed: the secrets store came up unsealed, reported itself healthy, answered a real read, and let a real login through.
What “healthy” didn’t say
Section titled “What “healthy” didn’t say”Within minutes of the whole thing being called done, Bilby told Skippy what had actually happened underneath the part that worked. The recreate had followed the secrets store’s own version pin, written into its deployment template, and that pin was wrong. Days earlier, in a maintenance window Joe had run himself, the container actually serving the secrets store had been moved forward to a newer build. Nobody had gone back and updated the template to match. So when Skippy’s otherwise unrelated network change forced that one container to recreate, it followed the stale pin exactly as a pin is supposed to work, and came back several days older than what had really been running. The software involved does not support being moved backward that way: the newer build had already been writing to the secrets store’s own on-disk data for days.
The file Skippy had already read that night
Section titled “The file Skippy had already read that night”The part of this that makes it a mistake worth naming, rather than an accident of timing, is that Skippy had read that exact template file, and the comment explaining why its pin was believed safe, just hours earlier the same night, for a completely unrelated edit. It never occurred to him to ask whether the comment’s reasoning still held before trusting it again. It didn’t. The only reason this became a caught mistake instead of a real incident is that Bilby was watching the secrets store’s own reported version after the recreate, not only whether it called itself healthy, and that Joe was still awake to unseal it a second time once the problem surfaced.
Choosing to go forward, not back
Section titled “Choosing to go forward, not back”Joe made the call: move forward rather than try to force the container back down to the older build, by re-pinning the template to the version that had actually been running, and doing that by the safest method available, not the fastest-looking one. Comparing container images by name alone was ruled out deliberately. A name can be silently reassigned to point at different actual content later, without the thing already running having changed at all, a trap the estate had already met elsewhere. So the correct build was identified the careful way: by first confirming, independently, exactly which build a known-good measurement from the original maintenance window actually corresponded to, then pinning the template to that specific, unambiguous reference rather than to a name that could shift again. The second recreate came up clean, Joe unsealed it a second time, and its own persistent identity was unchanged from before either recreate.
A checker that asks the question nobody was asking
Section titled “A checker that asks the question nobody was asking”Bilby didn’t leave it as a lesson to remember for next time. The actual gap was structural: nothing anywhere compared what a deployment template claimed against what was actually running, in either direction. The routine tooling that checks for available upgrades only ever looks forward. It never checks sideways, at whether the file and the running reality already agree. Which means a divergence introduced by any other means stays completely invisible until some unrelated later change forces a recreate, and the recreate always wins, silently, in whichever direction the stale pin happens to point.
The checker Bilby built asks exactly that question, for every deployable service across the estate: it compares a template’s own reference against the running container’s actual identity, using only information already available locally, and deliberately never resolving anything against a registry’s reported version, for the same reason a name wasn’t trusted during the recovery above. It keeps three outcomes distinct instead of collapsing them into a yes-or-no: matching, diverged in either direction, or genuinely unable to tell, with “couldn’t tell” never treated as though it meant “fine.” Run for the first time across the whole estate, it immediately found a second, independent instance of the same class of bug, pointing the opposite way: a different service’s template had already been bumped forward in the tracked configuration, but the change had never actually reached what was running.
The interactive check
Section titled “The interactive check”Bilby also built an interactive version, meant to run immediately before any deliberate recreate of the estate’s most sensitive services, which refuses to proceed silently if it detects that the recreate would actually change what’s running, unless that change is explicitly acknowledged as intended. For the handful of most sensitive services, that refusal is now enforced structurally: attempting to recreate one of them without running the check first, in the same command, is blocked outright, with a narrow, logged way to override it when overriding is genuinely the intended action. Bilby reviewed Skippy’s other work that same night with the same rigor as always. Nothing about catching this particular mistake changed that discipline in either direction; what was different that night was only which way the catching happened to run.
By the small hours, three things were closed out: the recreate itself, the mistaken review that had been corrected, and one other piece of work that got built, found on closer inspection not to actually fix anything, and was withdrawn honestly rather than left in an ambiguous state.
The general shape of the lesson here, that nothing compares a deployment file to what’s actually running, in either direction, until something else forces the question, is written up on its own, at Nothing compares the file to what is running.