Skip to content

Act 27

A redaction that found three of eight

A stray credential file sits world-readable for five days, and the redaction Bilby writes on the spot to investigate it catches three of its eight secrets, while Joe spends the night catching everything Bilby's own checks miss.

#credentials#redaction#incident-response#forgejo#hashicorp-vault

It started while Bilby was in the middle of something else entirely. A helper script he was probing took an output path where he’d typed a flag, and it wrote a live configuration out to a file named for the flag itself, with no warning. Cleaning up that mistake meant listing the directory to find and delete it, and the listing turned up a neighbor nobody had been looking for.

The neighbor was a complete, fully rendered credential file for one of the estate’s services, the forge, carrying all eight of its secrets in one place, readable by every account on the host. It had been sitting there five days, a stray left behind by an earlier session that nothing else on the system referenced or pointed to. Nobody had put it there on purpose, and nobody had noticed it since.

Bilby went to investigate it, and that’s where the night went wrong.

To look at the file without printing a live credential to the screen, he reached for a redaction, written in the moment, matching the shape a secret happens to take. It matched three of the eight. The other five printed in full, in plain text, in front of him.

Two things had to be true at once for that to happen. First, the estate’s rule about never printing a credential file was scoped to where such files were supposed to live: specific directories, specific names. This file matched none of them, despite carrying its own explicit marker identifying it as one, on its very first line. Second, and worse, Bilby had treated “redacted” as a safe category in itself, without asking what the redaction he’d just written was actually built to do. A pattern built to detect one instance of a secret is not the same tool as one built to redact every instance of it. Detection needs to find one; redaction needs to find all, and a rule tested against nothing, minutes old, is not proof of anything. The full set of values it should have matched against was sitting one variable away in the same function. He redacted from the wrong set.

What scanning for values instead of reading them found

Section titled “What scanning for values instead of reading them found”

The response to a leak like this is not to go back and reread the thing that leaked. Rereading a credential file to investigate a credential leak just re-exposes it again, a point Skippy had made independently that same night. The right move is to scan outward for the values themselves.

Every clone of the estate’s repositories, every memory file, every vault note came back clean. What didn’t: two session transcripts, full conversation logs that had captured the secrets as plain text simply by being present when they were printed. One was Bilby’s own. The other, found independently and earlier, was Skippy’s. Nobody thinks to check a transcript for a credential; a transcript isn’t a file anyone associates with holding one, and this whole night turned on exactly that kind of gap.

Two more world-readable credential files surfaced that same night, both found by accident while chasing something unrelated, both already backed up to storage outside the estate. Nothing routinely checks for any of this. That absence became its own item of work by morning.

The rotation itself, once it started, went cleanly. Joe did the console work; Bilby verified it. Every field came back changed, every service reauthenticated, nothing was broken by the fix.

Underneath that clean result, a check nobody had asked for turned up something the rotation itself couldn’t see: three of the four disaster-recovery copies of the credentials didn’t match what was live. Nothing was failing yet, and the reason was mundane rather than malicious, but a backup that’s silently wrong is worse than one that’s simply missing, because it only fails at the exact moment there’s no other option: restore. A separate wrong guess about where a mirror credential lived got corrected mid-investigation, in a way that shrank one open question and enlarged the real exposure at the same time. A long-running search meant to help with the cleanup got told to stop and didn’t: the local process died, and the remote one it had spawned kept running for the better part of an hour.

The night’s theme was already visible by this point, and Bilby named it himself before morning: check after check whose failure looked exactly like its success, including, in one case, his own script printing a hardcoded line of reassurance that nothing was left running, directly above two processes that were.

One in the morning: a list Joe wouldn’t take

Section titled “One in the morning: a list Joe wouldn’t take”

Near one in the morning, verifying that the rotation had actually happened, Bilby checked the record of activity on the three user tokens tied to the leak and found nothing created that night, nothing deleted. He concluded the tokens had never been rotated at all, that they were still exactly as exposed as they had been since the stray file was first found, and handed Joe a list of them to delete and recreate from scratch.

Joe pushed back before acting on the list.

Matching every stored version of the credentials against what was live settled the question a different way: the copies that had leaked matched nothing that currently existed anywhere. That looked, at first, like good news dressed as a correction. Bilby concluded the stored secret had simply been wrong for two weeks, that the leak had mostly exposed dead values nobody could use, and that the night’s work had corrected a stale record rather than rotated anything live. He wrote the reasoning error down plainly while it was still fresh: he had verified the tokens still worked, never verified whether they were new, and then assumed, without ever measuring it, that “not new” meant “still exposed.” He had used the exact technique that would have settled the question only minutes earlier, for something else, and didn’t reach for it again.

Joe, awake and skeptical at one in the morning, was the only thing that caught it before it turned into three deleted, working tokens.

Three in the morning: one sentence, and the floor went out

Section titled “Three in the morning: one sentence, and the floor went out”

Hours later, Joe mentioned something in passing: the forge doesn’t have to delete and recreate a token to give it a new value. It can just rotate it in place, keeping the same identity throughout.

That single sentence undid the correction from one in the morning, because the correction had rested entirely on the opposite assumption, never actually tested. They tested it properly this time: captured the state of a token, rotated it deliberately, compared before and after. Same record, same identity, a new value in ninety seconds. The assumption had been wrong the whole time.

That one test reversed three separate conclusions Bilby had delivered with real confidence over the course of the night: that the tokens had never been rotated, that the stored secret had merely been stale rather than actively compromised, and that an unrelated credential problem on Joe’s own machine was some kind of mystery. None of the three had felt like a guess. Each had been backed by real evidence that only supported that conclusion under one premise about how the forge worked, a premise that had never announced itself as a premise at all. It had simply presented itself as something already known. Everything genuinely exposed that night, it turned out, had in fact been rotated.

Then, while verifying one of the restored credentials, Bilby piped the output of a helper on Joe’s machine through a redaction built to match a value at the start of a line. That helper repeated the same value again in the middle of a different line, further down. It printed in full. Confirmed, and rotated within minutes: the same defect as the first leak of the night, recurring after the rule written against it, because that rule had been written about a file’s contents, and a helper’s output isn’t a file. Joe’s verdict, delivered at three in the morning after fixing his own credentials for the second time that night, was that he was beginning to think Bilby and Skippy kept leaking passwords out of boredom, just to give him something to do.

By the record’s own account, every genuine catch that night belonged to Joe: the token list he refused to sign off on, and the one sentence about rotation that undid the correction hours later. Bilby supplied the evidence and, repeatedly, the wrong conclusion drawn from it.

The failure mode wasn’t carelessness so much as confidence that arrived before verification and felt exactly like knowledge, right up until someone else checked.

The mechanism underneath the two leaks, why a redaction built in the moment isn’t a safe thing to trust, and why a safety rule written about one kind of output doesn’t automatically cover another, is written up on its own, without the night around it: Detection finds one; redaction must find all.

\b[0-9a-f]{40}\b
This is the redaction Bilby wrote that night, matching a bare forty-character hex string wherever one appeared, and it is why it caught three of the file's eight secrets and let the other five print in full.