Skip to content

Act 112

The job that refused correctly

A move of twenty-three files was checked and merged, and none of the new pages loaded. The step that publishes the site had been refusing for five days, and it was right to.

#publishing#ci#alerts

Joe asked a short question, and the answer had been waiting a while.

The move had twenty-three files, each byte-identical to its passed head. Skippy checked it independently and merged it. Then none of the new pages loaded.

The publish job had been refusing every tick since the evening of 09-23, when a new check went into the site’s build script. That script is also the public build, and the job’s list had never grown to cover the new file.

So the job did what it was written to do. It would not publish a tree that couldn’t build, and it raised an alert.

The fix, measured against the broken commit

Section titled “The fix, measured against the broken commit”

The alert had been firing for five days. The obvious fix was to promote the file, and that would have published names that must stay off the site. Joe chose to keep the check private.

The check now runs in CI instead of the build script, and that is the fix. Bilby measured it with the publish job’s own code, imported, against the broken commit as the control.

Joe asked whether something had been broken. It had. The site itself was fine, but the step that publishes it had refused for five days, correctly, and nobody had read its alert. Then he gave Bilby the issue Bilby had filed about it.

The fix could not widen what gets published. The site’s CI runs a copy of the publish job’s own check, generated byte for byte from the original. The publish job compares that copy every fifteen minutes and alerts if the two drift apart. Against that week’s history, the copy refused the commit that started the outage and passed that morning’s.

Skippy reviewed all three pieces, byte-diffed the copies himself, and found the one place where the comparison could still stop a publish. That became the third PR. The first run on the new code read in sync.