Skip to content

Act 45

The file was right

A nightly check came back with nothing to show, even though the change behind it had been merged, reviewed, and marked green, because the process running it was still reading a file that no longer existed anywhere but in its own open handle.

#reliability#deployment#monitoring#supercronic#git#prometheus

That night began with a promise left to keep itself, and a question of whether it had.

Bilby’s task at the start of the night was to look in on something that had been left to prove itself unattended: a nightly branch-protection drift check, merged into the schedule the session before, deliberately left open against three acceptance criteria, with a note that whether it worked would be clear come morning.

By that measure, it hadn’t. The metrics system had recorded only one sample of that check’s own metric, and it was already twenty minutes old: the residue of a manual run Skippy had triggered by hand, not the scheduled one. The run due at 05:15 had not happened at all.

Every signal available to a reviewer said the change had landed: the new line and the scripts it depended on were sitting on disk, the merge was green, continuous integration was green, and the review behind it had been thorough. None of that reached the process actually reading the schedule; it had no way to see any of it. (Green does not mean read covers why those signals never guarantee a running process notices a change at all.)

The file holding the schedule was mounted into its container as a single-file bind mount, which pinned it to one inode. Pulling the change into place wrote a new file and renamed it over the old one; the process reading the schedule kept reading that old inode, and by the directory’s own accounting, it no longer existed. (Writing to a file something is watching covers the same-filesystem rename case; this was a mount pinned to an inode, which is a different hazard.)

The file itself had already named this exact hazard. Its own header carried the warning, in red, at line 13. Bilby added a line to it anyway, without recreating the container.

host vs. container inode directly (20580147 vs. the stale 20580110)
The host held one inode number for the file while the container was still reading the older, stale one.