Act 4
A floating address for the resolver
The night a floating address went in front of a two-resolver pair, six things that broke on the way to working, a verification against a real client, and the wrong turn about what a reboot would and wouldn't fix.
The case for overkill
Section titled “The case for overkill”Two days before this night, the migration plan still had one decision open: whether a secondary resolver, added purely for redundancy, needed its own floating address, or whether the cluster’s own scheduler already covered it. The scheduler could relocate the secondary’s container to any node in the cluster if the one it started on went down. But the address configured as the fleet’s second resolver was still tied to one specific machine. If that particular machine died, not just its container moving elsewhere, that entry point was gone, even after the scheduler had already put the container back up somewhere else.
The call, made two days before the build, was to treat that as a real gap to close, not a documented shortcoming to live with. “Overkill is underrated” was invoked, and accepted as the governing principle rather than a joke. A reverse proxy was floated as a shortcut: route the secondary’s traffic through it and skip building real address failover. It was rejected on the merits. A reverse proxy only handles traffic that clients explicitly point at its own address, and a DNS query goes straight to whatever address is configured, no proxy hop involved. Routing through it would also have made the secondary depend on the very machine the primary already lived on, which defeats the reason a secondary exists at all. What got scoped instead was real failover: an election between two resolvers, and a floating address sitting in front of both.
The pair goes up
Section titled “The pair goes up”The day before this night, the new resolver had already gone live for real: a full cutover, not a parallel test, with the old resolver pair stopped and kept only as a rollback net. This night was the second half of that work. A secondary instance went up on a different, lower-powered machine in the cluster, and a floating address sat in front of the pair so that anything pointed at that one address kept resolving no matter which of the two machines was up.
Six things went wrong on the way to working, each one caught and fixed before the night was over.
Three things that didn’t work on their own
Section titled “Three things that didn’t work on their own”The container platform’s own network-interface naming turned out not to be stable across a recreate for a service attached to more than one network. Confirmed three times in a row, same compose file, the two networks swapped which interface name they got each time. The fix was an entrypoint that detects which interface belongs to which network by matching its actual address range, not by trusting whatever name the platform handed it.
The floating address itself didn’t serve anything on its own. The election daemon only owns the address; the resolver runs as an entirely separate service. Getting a query to reach it took adding address-translation rules by hand, inside the daemon’s own network namespace, to forward anything arriving on the floating address to the real resolver behind it.
One of the daemon’s own settings was guessed wrong twice before the correct name turned up, not in any guide, but in the container image’s own manual.
Two silent failures, and a credential that didn’t carry over
Section titled “Two silent failures, and a credential that didn’t carry over”A setting meant to keep the side that had won an election from stepping back down turned out to silently do nothing once combined with the state both sides were told to start in. The fix was to start both sides in the same neutral state and let priority alone decide which one won.
A session token that worked against one resolver instance turned out to be worthless against the other. The fix was getting a token that belonged to the instance being used, rather than assuming one instance’s credentials would carry over to its twin.
And zone replication onto the secondary failed silently, every zone reporting no successful transfer and no data, until the secondary was added into the primary’s own zone data as a real, delegated name server. That step got done once, by hand, through the primary’s own interface. It stayed manual on purpose: automating it would have meant exposing the primary’s own admin interface to the wider network, and that’s a separate decision for another day.
What was proven, and what was only staged
Section titled “What was proven, and what was only staged”None of this counted as working until it was checked against something real. A real client, a phone, resolving over VPN rather than sitting on the network directly, was pointed at the floating address and checked with two independent signals: a packet counter on the address-translation rule that ticked up exactly during the test window, and an internal query log showing the client’s own address rewritten to one that only appears if a query passed through that translation, rather than reaching the resolver by some other path. Both transports were checked, not just the common one; a plain query and one forced over TCP both came back clean. All three of the primary’s real zones were checked, not one representative zone standing in for the rest. And the health check the election itself depends on was confirmed to be checking whether the resolver was actually listening on its port, not just whether its process happened to still be running.
The night still ended with real things open. The central setting that points the whole fleet at the floating address had been flipped, and every dependent stack re-rendered against the new value, forty-two of them, but not one container had been recreated to pick it up. All of them were still resolving directly against the primary, exactly as before this night happened. A second setting, meant to keep containers running through a daemon restart rather than stopping cold, had been staged the same way: written, not active, folded into a reboot Joe already had planned for an unrelated firmware update.
A wrong turn about reboots, and a test not yet run
Section titled “A wrong turn about reboots, and a test not yet run”That same planned reboot came up once already this night, and it produced the one real wrong turn. Skippy first said, correctly, that a plain reboot wouldn’t pick up a re-rendered compose file’s new values: a container’s configuration is fixed at the moment it’s created, not read fresh from the file on every start. Then, in the very next message, said the opposite: that the daemon coming back up after a reboot would recreate containers fresh from their compose files. It wouldn’t have. Joe caught the contradiction before Skippy did.
And the test that mattered most hadn’t been run at all. Nobody had killed the primary and watched the secondary take over end to end; everything checked that night was the pair sitting still, correctly configured, ready. Whether it would hold when the primary went away for real was, at the end of the night, still just a plan.
+93 UDP packets