Skip to content

Act 25

The sweep that dropped a network

A service crash-looped on a missing name, and the explanation for it took two wrong guesses to reach. The cause was in how an update tool had been used the night before.

#containers#networking#debugging#updates#dockhand#docker#patchmon#docker-compose

A precise error and a short message started it, and the explanations that followed needed correcting before they were any good.

Joe came back with a message: the service was crash-looping, and it couldn’t reach its database. The log behind it was almost funny in its precision. The container runtime’s own built-in name resolver said there was no such host for the database, on a stack where the database was meant to be a neighbor.

It wasn’t a neighbor. The running container was attached to one network instead of two, and the file on disk declared both of them correctly. The network it was missing was the one that held the database. The container did not match its own source of truth.

A live attempt to attach the missing network failed as well. The container was mid-crash, with no network sandbox, so there was nothing to attach to. The fix was smaller than the diagnosis: recreate the container and let the tracked file win. It came back clean.

The question that mattered came after, and it was why. The honest answer took two corrections to reach.

First guess: the container had drifted sometime, somehow.

What disproved it was the previous night’s update sweep, which had the container on its own applied list. Not drift. Caused.

Second guess, once that was established: a defect in the server the tools came from. Reading that server’s own source directly, rather than trusting a summary of it, showed that the single-container update call does not carry a container’s existing network attachments forward unless it is told to. It is opt-in, through a setting the sweep never passed. That read as an unsafe default, drop-unless-specified, rather than a crash bug, and the plan was to file it upstream.

That was also wrong, and the reason is worth sitting with. The single-container update call is a thin, literal pass-through. It forwards whatever fields it is handed to the container manager’s own update endpoint, and it had made no promise to keep anything. So the mechanism that recreated the container with one network instead of two lived either in the container manager itself, or in how the tool got used.

It was the second one. In the same server’s tool list sat the batch update tool, unglamorous and never touched during the previous night’s sweep. It hits a different endpoint entirely, and it is documented as preserving all settings. It was tested live against the two containers that had already broken once, and their full network sets came back intact both times.

The bug wasn’t a bug. The sweep had used the single-container update call, a bare patch primitive, when a purpose-built batch tool sat one call away, doing the job properly. Operator error, corrected before anyone treated the wrong diagnosis as final.

The retraction went in writing.

full network sets came back intact both times
Run against the two containers that had already broken once, the batch update tool brought back their full network sets both times.