Making a rollback boring

01The situation

Every team we meet has a rollback plan. It is usually a paragraph in a document, and it usually contains the phrase “revert the deploy”. Ask when it was last used and the answer is either “during the incident in March” or a slightly embarrassed silence.

Both answers are the same answer. A rollback you have only run during an outage is a rollback you have only run badly.

02What we found

When you actually walk through it in daylight, the interesting failures are never the deploy itself. Reverting the application container is a solved problem and takes seconds.

The failures are downstream. A migration that added a column the old version does not know about, and a second one that dropped a column the old version still writes to. A cache holding the new shape of an object. A queue full of messages only the new consumer can read. A feature flag that was flipped as part of the release and is not in the deploy at all.

None of that shows up in “revert the deploy”. All of it shows up at 2am.

03What we did

Before a system's first real release, we run the drill. Somebody deploys a deliberately harmless change to production, somebody else who did not write it rolls it back, and a third person holds a stopwatch and writes down every place the process stalls.

The rules are simple. No preparation on the day. Use the documentation exactly as written; if it is wrong, that is the finding. And the person rolling back is not allowed to ask the person who deployed.

The first run is always slower than everyone expects. Ten, fifteen minutes, most of it spent working out which of three dashboards shows whether it worked. The second run, a fortnight later, is usually under two.

A rollback you have only run during an outage is a rollback you have only run badly.

04What changed

The drill turns the rollback from a piece of folklore into a procedure with a known duration. That number changes how a team behaves: if going back takes ninety seconds and you can see it working, you release smaller things more often, because the cost of being wrong has a price tag on it.

It also changes the migrations. Once you have watched a rollback stall on a dropped column, you stop dropping columns in the same release that stops writing to them. The database work splits into two deploys, permanently, without anyone needing a policy about it.

05What we would do differently

We used to run the drill once, near launch, and tick it off. That is not enough. The system changes underneath the procedure and the document rots quietly.

Now it goes in the calendar quarterly, and the person holding the stopwatch is somebody who joined after the last one. If the newest engineer can roll back production from the written instructions alone, the instructions are true.