Kodstigen
← Logging and Observability

Lesson 5 of 7

Alerting Without Burning Out Your Team

Collecting logs, metrics, and traces is only useful if the right person finds out at the right time when something's actually wrong, without being paged for things that don't matter. Alerting is where observability turns into action.

SLIs, SLOs, and error budgets

  • SLI (Service Level Indicator): a specific measurement, e.g. "percentage of requests completed successfully" or "p99 latency".
  • SLO (Service Level Objective): a target for that indicator, e.g. "99.9% of requests succeed over a rolling 30 days".
  • Error budget: the amount of "failure" the SLO allows, 99.9% success means a 0.1% error budget, once you're consistently spending it faster than planned, that's a signal to slow down new releases and focus on stability instead of features.

Alert on symptoms, not causes

A common mistake: alerting on every possible underlying cause ("CPU > 90%", "one Pod restarted", "disk usage > 80%") instead of on user-visible symptoms ("error rate > 1%", "p99 latency > 2s"). A single real incident can trigger dozens of cause-based alerts simultaneously, symptom-based alerts fire once, for the thing that actually matters to users.

❌ Alert: "CPU usage above 90% on node-7"
✅ Alert: "Error rate above 1% for checkout service, sustained 5 minutes"

Actionable alerts only

Every alert that pages a human should be something they can actually act on. If an alert fires and the response is always "check it, it's fine, go back to sleep", that's not an alert, that's noise, and it trains people to ignore the next one, including the real one.

  • Does this alert mean something is currently broken for users, or might be soon?
  • Is there a clear next action for whoever receives it?
  • Would ignoring this alert for an hour actually cause harm?

If the answer to any of these is "no", it probably shouldn't page anyone, it might still be worth a metric on a dashboard, just not a 3am notification.

WARNING

Alert fatigue (too many low-value alerts) is one of the most common ways real incidents get missed, not because nobody was watching, but because everyone had learned to tune out the constant noise. A team with 5 alerts they trust completely responds faster than a team with 50 they've learned to dismiss.

📝 Alerting Quiz

Passing score: 70%

  1. 1.What is an SLO?

  2. 2.The amount of allowed failure under an SLO (e.g. 0.1% for a 99.9% target) is called the error ____.

  3. 3.Alerting on user-visible symptoms (like error rate) is generally preferred over alerting on every possible underlying cause.

  4. 4.What is 'alert fatigue'?

  5. 5.Every alert that fires should always page a human immediately, regardless of whether there's a clear next action.

  6. 6.A Service Level ____ is a specific measurement, like percentage of successful requests or p99 latency.