Skip to content

Act 20

The firewall that was never really there

Building the estate's first CI runner meant reading its firewall the way an attacker would, and four inherited gaps fell out, none of them anyone's fault and all of them invisible until someone actually looked.

#networking#firewall-policy#ci-cd#unifi#terraform#forgejo

Two sessions ran the whole day in parallel, Bilby building forward and Skippy closing out a backlog of his own. This is Bilby’s thread.

The day’s stated goal was a continuous-integration runner: a virtual machine on its own network segment, in its own firewall zone, with a per-machine firewall underneath it at the hypervisor, registered against the estate’s code forge and executing real jobs by the end of the day.

A CI runner executes whatever is sitting in the repository it’s pointed at, faithfully, with no judgment applied in between. That means whoever writes the instructions it runs owns whatever happens next. That’s exactly why it went into a cage before it went anywhere near real jobs: every network change on the way was gated on a negative test, a proof that a boundary actually held rather than a proof that the happy path worked. One of those gates paid for itself within the hour.

Building the cage meant reading the firewall the way a hostile scanner would, instead of trusting what the documentation said about it, and four things fell out that nobody had done anything wrong to cause, all the same shape: something inherited, standing in for something actually authored.

The first was a stale file. The tagging model for one of the network’s trunked segments was documented backwards in the repository, and had already been corrected twenty-three hours earlier, in a commit message and in memory, but never in the file itself. Bilby read the file, believed it, and rediscovered the correction from scratch before Joe caught it, by asking a simple question: how had those two segments ever been trunked together at all, if the file were right? The stale copy had outranked the correct memory, for the ordinary reason that the file is what a working session actually reads.

The second was a rule that looked like a control and wasn’t one. Two zones carried an outbound allow rule for a short list of common services, with no matching block beneath it, and the network platform’s own predefined allow-everything rule, sitting underneath everything by default, was still the one actually deciding traffic. The narrow allow wasn’t wrong, exactly; it just wasn’t doing anything, while looking like it was. The runner’s own zone had the identical gap an hour earlier, and it was caught the same way the runner itself would need to catch things: by running two outbound probes from inside it that should have failed and didn’t.

The third was scale. Eight of thirteen zones could reach the estate’s gateway on any port at all, including the two networks that exist specifically because whatever’s plugged into them isn’t trusted. Real scoping, authored on purpose by a person, existed for exactly two zones, and those two worked. That was also the entire list. Everywhere else, whether a zone was open or closed to the gateway had been decided by default, by the fallback rule bundled into its zone type, not by anything anyone had actually written down.

The fourth was a blind spot in the tooling itself. The infrastructure-as-code tool reported a clean plan: no drift, nothing to apply. But a clean plan from that tool only ever covers what its own provider is capable of expressing. Most of the network platform’s policies turned out to be predefined underneath, invisible to the tool entirely. “Clean” meant everything I can see matches what I declared, which is a much narrower claim than it sounds like.

Two testing traps produced confident wrong answers before either was caught.

The first was a scanning script that took the first rule listed for the first rule that matches. A rule near the top of the list, written for one narrow protocol, made the runner’s zone look scoped down to that single protocol alone. It wasn’t: the web port fell straight through underneath it, and a live probe against that port went straight through too. The live probe was the one telling the truth.

The second was a test run from the wrong vantage point entirely. Validating the same model from a host that happened to share a network segment with the target came back “everything open,” which felt like it disproved a rule that actually worked, until it became clear that traffic between two hosts on the same segment never reaches a zone firewall in the first place. The test hadn’t found a hole. It had never crossed the boundary it was supposedly testing.

Skippy, working his own backlog in parallel that same day, hit a worse version of the second trap: a check that passed clean while, underneath itself, still using the very path it was replacing.

Getting the network platform’s provider to author policy at all took four failed applies before the shape of a working rule became clear. It got solved not by debugging the one that kept failing, but by reading a rule that already worked and copying its shape. That shape turned out to be legal on the controller and impossible for the provider to emit directly; the supported path was to create the rule live on the controller first and import it into the tool afterward, since importing can round-trip a shape that a fresh create cannot produce.

The same pass turned up a set of hardcoded controller identifiers standing in for name-based lookups, after one of them was found to have already gone stale without anyone noticing.

And two plausible explanations for why the runner’s first real job failed both turned out to be wrong: it wasn’t a missing piece of the job’s own environment, and it wasn’t a certificate that failed to verify against the forge. The actual cause was a manual, static entry pointing the runner at the forge by name, a deliberate choice made so that untrusted compute never gets to query the estate’s production name-resolution service directly. The choice was the right one. It also meant the same problem would resurface, silently, in every container that later needed to resolve that same name, since a host-level entry like that one doesn’t reach into containers running on top of it.

By the end of the day, the assertion was written and proven correct in all three of the states it needed to handle. It was not wired in. The only path available for it to reach the controller and actually run was the very hole the third finding had just described: the unauthored gap in the gateway’s own policy. Routing the check through that gap to make it pass would have buried that finding under a green checkmark, precisely the failure the runner was built to prevent.

The record stops there, with a list of open items handed to the next session.

The general shape of what went wrong here, a control nobody actually wrote standing in for one that was, and why that’s worse than having no control at all, is carried in full on Inherited is not authored.

Allow mDNS matches udp/5353, not tcp/443
The rule that caught the scan's attention was an mDNS allow matching udp/5353 only; tcp/443, the web port, fell straight through beneath it, same as the live probe did.