The alert that woke us up for three months

01The situation

We took over the running of a system that had been live for about two years. It had monitoring. It had a lot of monitoring. On the first night of on-call, the phone went twice; neither alert needed a human. On the second night, three times.

By the end of the first week we had a number: forty-one alert rules, and in the previous twelve months, six of them had ever led to somebody changing anything.

02What we found

The alerts had accumulated the way alerts always accumulate. Something breaks, everyone agrees it must never happen again, an alert gets added, and nobody ever comes back to ask whether it earns its place. Nothing in the process removes anything.

The team had adapted, as teams do. They had learned which pages to ignore. That is the real cost: not the interrupted sleep, but a team that has been trained, night after night, to assume the alarm is nothing.

03What we did

We went through every rule and asked one question: the last time this fired, what did a person do about it? Not what were they supposed to do. What did they actually do.

If the honest answer was “acknowledged it and went back to sleep”, the rule was deleted. Not tuned, not moved to a lower priority, not routed to a channel nobody reads. Deleted. Thirty-four went that afternoon.

The seven that survived got the opposite treatment. Each one got a runbook of a few lines, kept in the repository next to the code it concerns, saying what the alert means, what to check first, and who to wake if it is genuinely bad.

An alert nobody acts on is not a safety net. It is a training exercise in ignoring alarms.

04What changed

The pager went quiet, which was the point but not the benefit. The benefit was that when it did go off, the person holding it believed it. They opened the runbook instead of hoping.

The other change was cultural and slower. Adding an alert became a thing you had to argue for, in the same way adding a database table is a thing you argue for. That one shift stops the drift from starting again.

05What we would do differently

We should have measured before we deleted. We know the count went from forty-one to seven, and we know the on-call complaints stopped, but we did not have a clean before-and-after on how quickly incidents were being noticed. Next time that number comes first.

We also deleted in one pass rather than staging it. It worked, and there is a version of that afternoon where it does not. A week of “this rule is scheduled for deletion” in the alert body would have cost nothing.