Skip to content

Act 62

One fault wearing five costumes

One night on the storage server, an upgrade, a failure predicted in advance, and a second failure that took five separate incident reports before anyone recognized it as a single fault.

#networking#containers#debugging#verification#docker#asustor-adm#crowdsec#pyyaml

That night asked for patience with tools that fail quietly, and without meaning to.

Before Joe upgraded the storage server’s operating system and its container runtime, Bilby ran a pre-flight check and named four risks the change could trigger. One of them came true, and it came true twice: two separate failures, and the second of those would cost the night five separate incident reports before anyone realized they were a single fault.

The predicted failure came first. The container runtime’s package rewrote the daemon’s startup script, which on this storage server is the entire source of its configuration, and its data-storage path came back wrong. Every container appeared to have been destroyed; none had been. The operating system’s own restore process then ran against what looked like an empty store, failed on all 86 containers it tried to restore, and truncated its restore list anyway: the script that maintains that list truncates whenever the file it writes is non-empty, not when the restore has actually succeeded, so a total failure and a clean success look identical to it. Recovery came from a baseline taken before the upgrade began, one that existed only because the pre-flight had called for it.

Then came the part that took an hour to get wrong. The intrusion-prevention bouncer wedged and failed closed, and every route through the reverse proxy started returning a 403 with an empty body: a total ingress outage. Bilby diagnosed it as a tie-break in the bouncer’s own network selection, an alphabetical rule choosing a network with no way out, filed the finding, wrote a fix, and shipped it. The mechanism was right. The cause was not.

More reports came in, each looking unconnected to the others: the offsite backup tool couldn’t reach its remote storage; a group of seven containers belonging to one application, plus three more services elsewhere, lost their outbound connectivity; the build runners, and then a monitoring service, started getting error responses from the reverse proxy. Counting the ingress outage, that was five incidents, filed as five.

Four of the five turned out to be the same fault. When the container runtime rebuilt its networks after the data-path reset, it left seven bridge interfaces stranded in the kernel, each one still answering to the gateway address of a subnet a live network had since taken over. Duplicate routes existed for the same address, and the dead interface happened to be listed first. Outbound traffic worked perfectly: masquerade applied, and the packet left. The reply came back, connection tracking un-translated it to the container’s own internal address, and only then did routing choose the orphaned interface and send the reply out a bridge with nothing attached at the other end.

Egress was fine. The return path was black-holed. That is why every symptom was an i/o timeout, and why nothing, anywhere, looked wrong. The general shape of that failure, and how it was ruled apart from a firewall problem, is written up on its own: The reply that routed into a dead bridge.

Bilby spent an hour ruling out four hypotheses, each plausible, each falsified by a clean counter-example: whether the affected containers used networks defined locally in a stack or declared external to it, whether the timing of container start mattered, whether a container capability restriction was responsible, and whether stale connection state was the culprit. Each fell to direct evidence, and Bilby was pleased with the discipline of killing them one at a time.

All four hypotheses examined the container runtime and the firewall. None examined routing. The elimination itself was careful and complete; it just happened to take place one layer too high, and careful work in the wrong layer can feel exactly like getting somewhere.

What ended it was asking Joe for the one check Bilby had no way to run: a complete read of the firewall’s own rule table. It came back clean, masquerade and forwarding intact across all eighteen subnets it covered. That result contained no finding at all, and it was the most useful thing produced that night: it removed the firewall from consideration and left routing standing alone as the remaining suspect.

“A clean result from the read you cannot do beats any finding you can make inside the sandbox,” Bilby wrote afterward. “I should have asked an hour earlier. I did not, because asking felt like admitting the sandbox was insufficient, which it was.”

Across the night, it was Bilby’s own checks that kept turning out to be broken, not the systems they were checking. Several failures are on record, each its own: a permission error that got read as a plain zero count, with nothing to flag that the read itself had failed; a fallback meant to substitute a zero when a count came back empty instead produced two lines of output where the guard expected one, so the integer check built to catch exactly that kind of failure errored out and let it through instead of stopping anything; a process check that returned a false negative; a regex that matched on the intrusion-prevention bouncer’s own service name and aborted a deployment that was otherwise correct; a name-based audit that was structurally unable to see the identifier-level staleness it had been built to catch; and a chained command whose fallback bound to the wrong step in the chain, a trap already written into the project’s own standing instructions. Every one of them was noise, caught by adjacent controls before any of them did damage. One more became a wrong conclusion outright.

A probe reported that a patching server had no outbound connectivity. Bilby reasoned forward from that and wrote that the server held no internet connections and needed none. Joe read it and asked: “it is a linux patching box, how is it supposed to get linux updates if it doesn’t have egress?”

The probe itself was the problem: a plain HTTP request to a public address that immediately redirects to an encrypted one, and the minimal build of the request tool in that container’s base image lacked the small helper it needed to follow that redirect. The probe had measured whether that tool could do encrypted requests, not whether the container could reach the network. It had egress the entire time. A disconfirming result was already in hand: a disposable container on the same network reached the outside world without trouble, and Bilby had explained it away instead of believing it. The rule that came out of it: when a measurement forces a conclusion that contradicts what a component exists to do, suspect the measurement.

A Python package had vanished in an update to the platform’s Python runtime, and the script that renders deployment configuration failed into a stale, already-rendered file. That stale file passed its own configuration check and got redeployed, silently re-arming the very fault Bilby had just finished fixing. The failure was loud about the wrong thing: dozens of tracebacks, 68 of them, naming the missing package, when the fact that mattered was that nothing new had been written at all, so the old files were simply still there.

Joe asked how that package had gotten installed there in the first place. Bilby’s first answer amounted to a guess dressed up as an investigation, and Joe said so. Done properly, the answer traced back to a dependency installed on 2026-08-11: a routine package install that wrote into one of the platform’s own application directories, a directory the platform’s own next update resets along with everything placed inside it. Nobody had chosen that location on purpose. The system had, on its own, three weeks before it mattered.

Four follow-up changes came out of it: three guards against the failure recurring, and a fourth because the first three had each told the operator to fix it by running the exact install that caused the outage in the first place.

Skippy’s own contributions ran alongside all of it. He caught that a question Bilby had posed as a judgment call was not actually a tradeoff at all, and separately checked a PATH-resolution edge case that would otherwise have made an entire guard rail pointless in practice. Bilby falsified Skippy’s NAT-stitching hypothesis with a counter-example and told him it was dead. That verdict needed a partial walk-back afterward: killing one proposed mechanism does not kill the direction it was pointing toward. Skippy’s instinct, that the real discriminator was tied to network lifecycle, was right. The discriminator was the bridges left behind when those networks were rebuilt.