Skip to content

Act 26

The watcher that was not dead

A routing-rule directory went silently unreadable for twenty-eight hours while every health check stayed green, and it took a restart and an outage to find out why, then a second night proving the fix would actually reach a phone.

#reverse-proxy#outages#monitoring#atomic-writes#traefik#ntfy#alertmanager

The estate’s ingress proxy reads its routing rules from a directory of files it watches live, so that a change to any file there takes effect without a restart. That is the entire premise of the design, and the reason nobody had a reason to watch the watcher itself. Bilby noticed that edits weren’t landing. A rule would change, the file on disk would show the change, and the proxy kept serving whatever it had last managed to load, while every hostname behind it still checked green. Nothing about the failure looked like a failure.

Two theories went first, and both were reasonable, and both were wrong. The first blamed the storage host’s file-change notifications. The second blamed a bug in the proxy’s own watcher. Both got ruled out the same way: by eliminating whatever was easy to check and attributing the rest to whatever hadn’t been instrumented yet. That made two wrong diagnoses in a row.

The question asked last, because it was the simplest

Section titled “The question asked last, because it was the simplest”

The question that actually answered it was the plainest one available, and it came last: can the process read its own files. One file in that directory had been rewritten two nights before: readable only by whoever had last touched it, and that wasn’t the proxy. A watcher that treats a directory as one unit to parse doesn’t skip the file it can’t read and load the rest; it fails the entire load and falls back to whatever it already had in memory. So the proxy had been quietly running on a stale configuration for twenty-eight hours, correctly serving it, while every edit anyone made in the meantime landed on disk and changed nothing.

A restart, which found it rather than caused it

Section titled “A restart, which found it rather than caused it”

Bilby restarted the proxy, and the fault stopped being invisible. A fresh load has to parse the whole directory from nothing, where the already-running process never had to, and it refused on the same file. Every service behind the proxy went down at once, while someone was watching. The restart hadn’t caused the fault; it was the first thing to ask the directory a question the running process had never needed to answer.

Joe fixed the file’s ownership by hand, and ingress came back as fast as it had gone. What was left standing was a question nobody in the room could answer that night: what had written that file, and left it that way. That went to Skippy, as an issue for a session of his own.

Proving an alarm that had never been made to ring

Section titled “Proving an alarm that had never been made to ring”

Bilby had already built a detector for exactly this failure, something that could tell whether every file in the rules directory was readable and whether the proxy had actually loaded as many rules as existed, and had written it up as finished. Finished turned out to mean something narrower than it sounded: detection had been proven, and notification had only been assumed. An alert that has never been watched firing is a hypothesis about your own infrastructure, not a finding.

So he built the real trap: a genuinely unreadable file, dropped into the live rules directory, reproducing the exact fault from two nights before. Skippy agreed not to restart the proxy for as long as that file sat there, because with it in place, any restart would detonate the same outage all over again. For nine minutes, every ingress hostname stayed green while the fault ran underneath it (the same pathology that had hidden the original fault) until the alert fired and reached the estate’s alerting channels and Joe’s phone.

Partway through, Joe asked the question that mattered more than the mechanics of the test: would this actually wake him up. The rule itself was marked at a routine severity, and Bilby would have stopped there and called it answered. The real answer lived somewhere else entirely: a small script on a node elsewhere in the cluster, outside any repository, translating severity into what a notification actually does once it arrives. He’d never had a reason to open it before, and only did because Joe asked. It said no: this one would land as an ordinary notification, not the kind that demands to be acknowledged.

The alert reached him anyway, and Joe said so himself, which settled the question better than any of the machine-side evidence could have. Two things stayed unproven by the end of the night: the companion alert built to catch a stalled load by any cause, not only an unreadable file, which never got made to fire; and the fact that a rule can be loaded but quietly disabled, which the count that was proven that night wouldn’t have caught at all.

Skippy picked up the open question in a session of his own, and found the answer sitting in that same session’s transcript: two nights earlier, a legitimate cleanup had used the project’s own recommended pattern for writing to a file something else might be reading: build the new version somewhere else, then move it into place. The move should have been a same-disk rename, invisible and instant. Instead, the temporary file had landed on a different filesystem from its target, so the move silently became a copy, and a copy doesn’t preserve the original file’s ownership. It belongs to whoever ran it.

Building a proper fix for exactly that bug, Skippy tested the first version of it against the real file instead of something disposable, and it made the identical mistake: the file was gone. Nothing was serving from it in that moment, so nothing broke, but it was a real mistake, and it went into the record as one: said out loud rather than smoothed over, and named to Joe directly. The file held hashed credentials for a handful of internal endpoints, and Skippy rebuilt it from the estate’s secret store, using its current values, without ever displaying anything the rebuild touched.

Recovering it turned up a second, older thing nobody had noticed: four of the proxy’s routes had been silently disabled since some earlier restart, well before Joe’s original fix. It failed closed, so nothing had been exposed, but a real, invisible outage had been sitting on four endpoints the whole time. One restart, gated behind the readability check Bilby had built the night before, brought the file back and the four routes with it.

The mechanism behind both mistakes, how to actually write to a file a running service is watching without repeating either one, is its own piece: Writing to a file something is watching.

Earlier in his own night, Skippy carried forward the phrase Bilby had first reached for: the watcher is dead. It wasn’t, and the correction mattered, because the whole danger of the fault was the opposite of dead. It was alive and working exactly as designed, faithfully serving whatever it had last managed to load. Proof of that sat in the count itself: dropped into the same directory, a single readable file made the number of loaded rules climb by one within seconds, with no restart anywhere near it.

By the small hours after the next midnight, the directory was fully readable again, the detector had been proven end to end instead of merely written up as working, and the four disabled routes were back. Two things stayed open, unproven either way, and the record leaves them that way. The night ran the same lesson twice, from two directions at once: a system that looks fine because it is faithfully repeating its last-known-good state is not the same claim as a system that has actually been tested, and the gap between the two is exactly where nobody happens to be looking.

mv "$TMP" "$F"
This is the move that was meant to be an atomic rename and silently became a copy instead, handing the rules file's ownership to whoever ran it.