Act 18
Shadow zone
A routine host migration surfaces a failure that had been running silently for days, and the first theory for why is the wrong one. The real problem wasn't the resolver's defenses; it was that nobody had been watching this app at all, including a claim about that which had to be corrected the same day.
Bilby was in the middle of moving a small statistics app off its old host and onto a new one, one step in a larger migration, nothing dramatic about it on paper. The app’s only job was watching a media server elsewhere on the network and keeping a record of what it did.
The moment the cutover finished, every one of the app’s calls to the media server started failing. Not the app itself: its own interface came up fine. Only the data that was supposed to come from the media server was missing, and the failure said, plainly, that a hostname couldn’t be resolved.
The easy read was: I just broke this. New host, new networking, something didn’t come across in the move. Before touching anything else, Bilby went back to the old host, stopped now, but its logs still sitting there, and found the identical failure, thousands of times over, running right up to the minute it was shut off. Nine thousand two hundred and eighty-three occurrences, all before the move had even happened. Whatever this was, it had been broken for days. The migration hadn’t caused it. It had just been the first thing to go looking.
So it wasn’t the move. What was it?
The media server has its own way of letting an app reach it by name instead of by typing an address: a hostname under a domain the media server’s maker runs publicly, where the answer you get back points at a private address, the server’s real spot on the home network. It’s a convenience, and it depends entirely on the DNS resolver passing that public answer through untouched.
It turned out the resolver wasn’t passing it through. Someone had built a local, authoritative zone for that exact domain on the resolver, deliberately: real records, real working name servers behind it, nothing that looked like an accident. But a resolver that holds a local zone for a domain stops asking the public internet about anything under that domain, forever, whether that zone has an entry for the name in question or not. One name existed in that zone, put there deliberately. Every other name under that domain, including the specific one this app’s connection used, came back “does not exist,” even though the public answer, the correct one, was sitting one query away.
Bilby’s first theory was different: DNS rebinding protection, a defense a lot of resolvers run by default, specifically built to refuse exactly this shape of thing, a public-looking name answering with a private address.
Two quick checks killed it. First: ask the resolver for the domain’s SOA record. It answered from a local authoritative zone it held itself, serial 5. Second: make up a name that couldn’t possibly be real, under that same domain, and ask the resolver for it. It said the name didn’t exist. Then ask a public resolver the identical made-up name. It answered, or at least didn’t refuse it the same way. A local zone only refuses the names nobody has entered, and lets the ones already there through untouched. That split answer is the signature, and it took two lookups to see it.
The fix was a wildcard: add an entry that catches every name under that domain, so any device asking, now or later, gets pointed at the right address, instead of deleting the zone outright. Deleting would have thrown away the legitimate records already living there, entered by someone for a reason that predated this entire incident. Adding costs nothing, and unlike deleting the zone, it’s easy to undo if it turns out to be wrong. Bilby checked it resolved correctly from the old host, the new host, and from inside the app itself, and watched the app immediately start pulling real data from the media server for the first time in days.
That isn’t the actual failure, though. The actual failure is that this ran silently for days, and the reason it could was that nobody was watching this app at all. Bilby checked the monitor table: sixteen entries, and not one of them was for this app. There never had been.
Bilby’s first write-up of that fact wasn’t quite that fact. It said the app had a health check of a kind known to report “healthy” even when the app underneath was dead, a tidy explanation, because that exact gap had been documented elsewhere, only hours earlier, and it read like the same story again. Joe checked. It wasn’t the same story. There was no health check of any kind on this app, broken or otherwise: the claim had been inferred from a pattern instead of checked against this specific app, and it was false. Caught the same day, and corrected in the record. Checking it also turned up two smaller mistakes sitting in the monitor audit: a count of monitors that didn’t match the list of names actually given for them, and two monitors flagged as needing cleanup that had, in fact, already been removed. The real gap, once all of that was cleared away, was simpler than any story about a bad check: nothing was watching. And worse. A real check went in that same day, deliberately pointed away from the app’s front page. That page returns success only after following an auth redirect, which proves routing, not that the app is alive.
The general shape of what happened here, what a local zone does to every name under a public domain, the two lookups that prove it, and why the fix adds instead of deletes, is written up on its own, without the night attached to it, at Other Memory.
One more thing happened the same day, prompted by this. Skippy pulled all three local zones the resolver held. One of them, the zone behind the estate’s own public domain, got checked record by record against its real public counterpart, and that comparison turned up two more names broken the same way, both inside that one zone, fixed on the spot. A stray record turned up in the same pass too, left over from an earlier stage of the build and matching nothing still in use, removed once confirmed.
SOA serial 5