An infrastructure with no monitoring is blind. An infrastructure with too much monitoring is blind too, but it thinks it can see. That’s worse.
The abandonment threshold
There’s a volume of alerts beyond which a human stops reading them. Nobody announces it, nobody decides it consciously: they simply start archiving without opening. That threshold is low. In my experience, a few messages a week and the inbox stays alive; a few a day, and it’s dead within a month.
The day an alert actually matters, it lands in a graveyard.
What I cut first
When I take over an existing monitoring setup, I start by removing:
- Alerts that resolve themselves. If the incident is over before anyone reads the message, it needed a graph, not a notification.
- Absolute thresholds on disks. “80% full” on a 20 TB volume means nothing. “Will be full in ten days at the current rate” means something.
- Duplicates. A service going down shouldn’t produce one alert per probe that depends on it.
- Anything with no action attached. If nobody knows what to do on receiving the message, the message shouldn’t exist.
The action test
My triage rule fits in one sentence: an alert must say what to do. Not what’s wrong — what to do about it.
“High load on server X” doesn’t pass the test. “The backup repository will be full in 8 days — free up space or extend the volume, procedure page 12” does.
Writing the action forces you to answer the real question: does this problem deserve to wake someone up? Most of the time, no. It deserves a weekly report, and that’s enough.
The result
On the infrastructures I run, the target volume is a few alerts a month. When it climbs, that’s not a tuning problem: something is actually wrong and needs fixing, not filtering.