Act 26
One file, seen from two sides
Two nights chasing the same broken guarantee, a note vault's history failing to reach its code forge, end on the discovery that a host and the container it feeds share a single git configuration through one mount, not two.
The estate keeps its working notes in a vault that runs inside a container, and the vault’s history is supposed to reach the estate’s code forge by git without anyone thinking about it. Two nights running, Skippy went looking for why that had stopped being true, and got the wrong answer both times before getting the right one. Each wrong answer was a real fact; it just wasn’t the fact underneath it.
A fix that didn’t survive contact with a direct check
Section titled “A fix that didn’t survive contact with a direct check”The first night opened on a fix from an earlier session that hadn’t held: the vault’s git plugin kept hanging on a credential prompt with nobody there to answer it. Skippy didn’t re-check the settings that were supposed to be right. Instead, Skippy went looking for the actual stuck process, and found one: a push sitting on that same prompt, waiting on an answer that was never going to come. The credential helper behind it pointed at a script path that existed on the host and nowhere else; from inside the container, that path meant nothing, so every push fell straight through to a prompt with no one behind it.
Joe made the call on the fix: switch to the simplest thing available, a plain stored credential, and stop asking the plugin to reach outside its own walls for one. Skippy proved it, pushing from inside the container itself and watching the commit land, then closed the issue.
It didn’t stay closed. Joe checked directly, rather than taking the proof on faith, and the proof didn’t hold up: every push that had “worked” had been triggered by hand. Nothing had shown that the plugin’s own scheduler could do the same thing unattended. Reopened.
The hang that wouldn’t move
Section titled “The hang that wouldn’t move”Skippy went back to the same instinct that had worked before, looking for the actual stuck process rather than checking the setting again, and found one: a push hung on the identical prompt, this time for four-plus hours, with every scheduled cycle behind it backed up and waiting. Killed, and the very next cycle told two different stories at once. The plugin’s own auto-commit fired on its own, unattended, for real: proof, finally, of the thing the first night’s proof had missed. Auto-push did not. A fresh hang appeared thirty-one minutes later, exactly on schedule, against the identical wall.
By the end of the night the pattern was clean enough to trust: every push triggered by hand succeeded, every single time; every push the plugin tried to make on its own hung, every single time, with no exception. That stopped looking like a credential problem and started looking structural, as if the plugin’s own push, whatever else it did, simply never consulted the credential helper it had been given, no matter how that helper was configured. The night ended on that honest read and a next step named, not a fix.
The wrong thing to be worried about
Section titled “The wrong thing to be worried about”The second night started from a premise that had felt settled after the first: the plugin’s own push was the thing standing between the vault and its history reaching the forge. It wasn’t. By design, the plugin only ever commits locally; it was never the push mechanism at all. The actual guarantee had always been a separate job, scheduled on the host itself, running every twenty minutes on a credential path that had nothing to do with the plugin’s. That job was the real backstop, and it had been failing on every single run since the previous evening: eighteen runs, every twenty minutes, identically, and silently enough that nothing had surfaced it.
The cause, once Skippy went looking, was almost embarrassingly plain: the host job’s credential helper was pointed at a path that only existed inside the container. From the host’s own execution context, running unattended with no one there to answer a prompt even if one had appeared, that path resolved to nothing, and the job failed instantly, every time, for the better part of a night.
A fix that broke the other side within the hour
Section titled “A fix that broke the other side within the hour”Skippy repointed the host job’s credential helper to a path that actually existed on the host, writing it as a persistent change to the repository’s configuration, the ordinary way to fix a setting that’s simply wrong.
Within the hour, a scheduled check from inside the container hit a completely different failure. The fix that had just gone in on the host side looked like it had broken something it had never touched. It had actually broken the same thing again, from the other direction: the host’s checkout and the container’s own copy of the vault turned out to be the same repository’s configuration, reachable through two different mounts into the same underlying file, confirmed the only way that kind of claim can be confirmed, by checking that both paths reported the same device and inode. One file, two doors: a persistent write made through either one changes what the other reads, and nothing about the write itself says so. Nobody had reason to know this until it broke something, and it had, immediately, caught by that very check rather than sitting silent the way the first failure had. The mechanism that had spent all night failing quietly didn’t catch this one; the container’s own side did, within the hour.
What actually held
Section titled “What actually held”The fix that lasted had two parts. The shared configuration went back to the value the container actually needs, restoring what the plugin had been reading all along. And the host’s job stopped writing to that shared file at all: it now carries its own credential helper setting on every invocation, clearing any inherited one first, so the host and the container can each get what they need without either one being able to overwrite the other’s. Skippy verified it end to end: a real pull, a real push, run directly, and after that the job kept succeeding on its own ordinary schedule too, no persistent write anywhere in the path.
The plugin’s own hang, separately, turned out to be a confirmed bug in the plugin itself, not a local misconfiguration: observed here twice with the identical signature, and not something fixable from this side. That stopped mattering as much as it had seemed to the night before, once the real backstop was doing its job, the plugin’s own broken push was no longer load-bearing.
One question was left standing rather than forced to an answer: why the plugin’s own git activity never seemed to consult the credential helper it had been given in the first place. An isolated test aimed at that question got contaminated by the same-night config mistake mid-run, and was left open as inconclusive rather than forced to a verdict the evidence didn’t support.
By the end of the second night, the job was succeeding on its ordinary cadence, and the configuration it depends on was no longer a place where a well-meant fix on one side could quietly undo the other. The same lesson had already shown up elsewhere in the estate that same night; this thread carried it one layer deeper, which is its own story. This one ends here; the mechanics of the one file and its two doors are their own piece, written up separately: A git configuration behind a bind mount is one file with two doors.
device:inode 2306:25307706 on both sides