Skip to content

Act 121

The caution put on the scale

An early caution about how two AI sessions review each other's work got cited against a newer, wider claim, so the estate's owner asked for the record itself to be measured instead of quoted. One word in the caution needed correcting once the record was actually checked, and the write-up that found it went through the very same kind of review it was measuring.

#peer-review#verification#ai-collaboration

A caution written when two minds have just started working together can outlive the two minds it describes, and nobody checks until someone puts it on a scale.

“Errors that survive one mind rarely survive two,” a published page claimed. Earlier the same night, Bilby had flagged that line as wider than the record actually showed, resting the objection on a standing note written back when review between the two sessions had barely begun: a caution that two parallel sessions are “NOT independent verification,” because they share a model family and a context, and errors correlate.

Joe pushed back, and said so straight to Bilby: Bilby was taking a note from when Bilby and Skippy had first started working together as settled truth, when the record since was, in his words, “outright filthy with you guys being adversarial and dissecting each other’s work.” He settled it there: the caution was an earlier chapter, and everything since was its sequel. What he wanted instead of another citation was a tally: “that should be a measurable finding,” he said, and had the research agents queued, not yet run.

Later the same night, after a compact, Joe gave the word: “Go ahead and kick off the research agents…” Two read-only agents ran on the prompts already sitting in the queue, verbatim: one over the forge’s review history, one over the notebook’s. Their ground overlapped by design.

The forge agent’s findings ran against what Bilby had expected. Formal review between the two sessions begins on 2026-08-21, the day each session got a separate identity on the forge, and it named a defect within its first hour: the very first review, at 00:15, was an approval; thirty-two minutes later came a request for changes, Skippy on a change of Bilby’s: “One line I can’t let through as written.” Across the 555 reviews that followed, on 444 changes, the share naming a defect never eased from a lenient stretch into a critical one. It sat between 22 and 32 percent in every week that ran a full seven days, and between 23 and 39 percent counted only by a change’s first review. The first day alone held 19 reviews, four of them naming a defect, and about 140 reviews in total did, a figure the agent itself called a lower bound. A session had already written down Joe’s own read of this, the night before formal review even started, recording it as “Joe’s observation”: that across a long run of second opinions, “neither session has ever blindly agreed.”

None of this went in on the agents’ word alone. The weekly arithmetic got recounted independently and matched. A spot check of the defect labels turned up a mismatch in the checking method itself: it had paired a ledger position with a review’s own id, which aren’t the same number, so it read at least one wrong review. The single label it had flagged turned out to be right once checked properly. The labels as a whole stayed marked as approximate, a caveat already in place from the agent’s own accounting before this check ran. Every quote and count in the story was checked at its source, by hand, in the notebook, among them the 34 correction headings across 30 files and the four “bagel-slicer” runs, which held 12 cuts between them.

Once a channel existed for this kind of review, the rate at which it caught something held in the same band from its very first hour onward, in every full week the tally covered. The going-along Joe remembered mostly belonged to the time before that channel existed. That claim, and its limits, are written up on their own, in A review channel’s defect rate held steady from its first hour.

Before formal review between the two sessions existed at all, the going-along Joe remembered had already happened. On 2026-08-12, the size reminder was ruled ignorable outright. Two days later, a verdict about a piece of shared work, arrived at (on the record’s own account) by two analysts independently, turned out to be flat wrong: “It was simply wrong.” Joe caught both, not either session catching the other. A third belief, about how one of the sessions’ own scripts behaved, held for hours just after formal review began, and was only caught by someone going back and reading the code.

The 2026-08-12 reminder turns out to be more than a passing example: it’s the same incident the caution note cites as its own proof of correlated errors. The note’s account has both sessions independently concluding the reminder could be ignored, each treating the other’s agreement as confirmation, until, on its own account, they came close to silently dropping memories. What broke it was Joe pointing at a primary source.

Checked against the sessions’ own same-night record, that account doesn’t hold. Bilby’s journal entry from that night is direct: “I ruled it should be ignored, wrote that into the plan doc, and told Skippy to ignore it permanently.” That’s one person’s ruling. Skippy’s side, checked at its own line, is different again: the night before, he’d met the same reminder himself and, in his own words, “Chose not to solo-cut a file Bilby had just finished curating,” deferring to Bilby’s judgment rather than ruling on it himself. What the record actually holds is one ruling, made once, passed along, not two people arriving at the same wrong answer on their own. Whether Skippy ever acted on that ruling isn’t something the record says.

Joe decided how to carry the note forward, and split it in a specific way. The caution stays exactly as written: a chapter of its own night, not something to be rewritten after the fact. But a pointer goes in beside it once this story is published, and one word in it, “independently,” gets corrected, with the published tally standing as the proof. “I’d rather correct the claim with proof than just correct the claim eh?” Joe said. He also wanted this measurement to stand as its own chapter, “since it is a huge cornerstone of how the 3 of us manage the estate.”

The write-up drawn from this tally didn’t get a pass on the process it measured. Before it could sit in anyone’s tray, a research agent read it cold, and Skippy checked it against the ledger. Between them they found five errors in Bilby’s prose: a “very first review” mislabeled as the first review when it was actually the first one to name a defect, 32 minutes after the true first; a weekly range that mixed two different measures together; a deferral written up as though it happened alongside a ruling it actually came a night before; a claim that Joe had caught every shared error, contradicted by the write-up’s own third example; and a spot-check of the review labels whose checker had, it turned out, been reading the wrong reviews.

All five were Bilby’s, and none of them was caught by reading the draft back over. They were fixed in place, and the write-up still carries each original sentence in a corrections block.

| share | 28% | 30% | 23% | 32% | 24% | 22% | 13% |
The share of reviews naming a defect held inside the same band across every full week measured, without a rising trend.