Act 76
The partial write that wasn't
A single call meant to add one variable to a stack erased six, because a wrapper's word for what it did wasn't a description of the layer underneath it.
That night ran on trust that hadn’t been earned yet.
One call, meant to add a single variable to a stack, deleted six. The container it belonged to hadn’t even been recreated yet when it happened, so nothing was broken that a user could see, but the write itself had already replaced the stack’s entire set of stored values with the one line it was given, and five of the six things it erased belonged to a migration that wasn’t finished. The sixth didn’t belong to that migration at all.
Two jobs, one pass through the stacks
Section titled “Two jobs, one pass through the stacks”Joe asked to fold a separate piece of outstanding maintenance work into a migration already underway: moving secrets out of hand-wired values and into a secret-management backend, driven through a container-management tool. Same configuration files, same recreate cycle either way, so the instruction was simple: don’t touch any one stack twice. Skippy picked the pilot stack for the combined pass, following a recommendation the migration’s own earlier planning had already made.
A name that didn’t match its field
Section titled “A name that didn’t match its field”Before touching anything live, the mechanism needed one more answer than earlier testing had given it. A few of the secrets being migrated had a variable name inside the container that didn’t match the field name the value was actually stored under in the backend, something the migration’s original testing had never hit, because every case it had tried before used matching names. Rather than guess, or rename a live secret’s stored field to make it match, Skippy ran the same kind of test the migration’s early work had already established: throwaway material, in a throwaway stack. A bare declaration of the backend-named variable, left sitting alongside the existing line that still referenced the container’s real variable name, worked. The injection landed early enough for the configuration’s own interpolation to pick it up, so no renames were needed anywhere.
A word that didn’t mean what it said
Section titled “A word that didn’t mean what it said”Wiring the new variable onto the stack, Skippy called a write endpoint directly with only that one variable, on the assumption that a partial payload would merge with whatever the stack already had. That assumption came from the wrapper layer used to make this kind of call: its own language described writing a value as a merge. The raw endpoint doesn’t merge. Reading the underlying tool’s own source afterward showed the function behind that endpoint deletes a stack’s entire stored variable set and then inserts exactly what the request body contained, nothing more.
What the write actually undid
Section titled “What the write actually undid”The damage was real but, for the moment, invisible: six secrets erased from the container-management tool’s own stored variables, one of them unrelated to the migration itself. Because the container hadn’t been recreated in the window when the values were wrong, nothing that depended on them had actually failed yet. Skippy found the problem by reading the endpoint’s source ahead of the next planned step, not after something broke downstream.
Fixing it without waiting to be caught
Section titled “Fixing it without waiting to be caught”The fix was a single write, built entirely from what the secret-management backend still held on record, never printed to a screen or a log along the way, restoring all six values in one pass. That closed the immediate hole. It didn’t stop the script that had been setting those values by hand from doing the same thing again on its next run, so that script got rewritten so it no longer can. What happened went into the write-up as it happened, rather than getting quietly patched over.
A second pass finds a second gap
Section titled “A second pass finds a second gap”A review from Bilby found that the mistake wasn’t only a matter of discipline. What followed was a check that runs ahead of any future write. It reads and confirms which keys already exist on a stack, with every value still masked, so a key can no longer be silently dropped by a future full-replace write. Working files that had been landing in a bare temporary directory moved into a private one instead.