Service Level Objectives: Choosing Alerts Around User Harm

CPU usage is high. Is that an incident? Maybe the service is efficiently handling a traffic spike. CPU usage is low. Is everything healthy? Not if every request is failing before useful work begins. Service level objectives connect reliability to an outcome users care about. They are not a replacement for infrastructure metrics; they provide context for deciding which signals deserve urgent attention. ## Start with a user journey Choose a meaningful operation, such as reading a project, completing a search, or submitting an order. Define a service level indicator that measures whether that operation succeeds within an acceptable time. Be explicit about the denominator. If you exclude errors too broadly or measure only requests that reach one internal component, the indicator may miss the failure users actually experience. Avoid averages when tail behavior matters. A small group of users can have unusably slow requests while the average looks fine. Define the population, latency criterion, and evaluation window clearly. ## Set a target that supports decisions The objective should reflect business needs and the cost of achieving it. Copying another company's reliability percentage does not establish what your customers require. The difference between perfect service and the target forms an error budget. That budget provides a shared way to discuss tradeoffs between shipping changes and reducing risk. Decide in advance what happens when the budget is consumed too quickly. The response might include slowing risky releases, addressing a recurring dependency failure, or investing in a recovery mechanism. It should not automatically mean freezing every harmless change. ## Alert on meaningful budget consumption An alert that wakes someone should indicate a condition they can act on. Rapid error-budget consumption can be more useful than a static infrastructure threshold because it reflects sustained user impact. Combine windows thoughtfully. A short window detects severe problems quickly, while a longer window helps distinguish continuing harm from a brief fluctuation. Check how low traffic affects the signal before applying a standard formula. Keep diagnostic dashboards for supporting details such as database connections, queue age, and dependency latency. Those metrics help explain an alert even when they should not independently page someone. Review objectives with product and support teams. A technically successful response may still be useless if it contains stale or incomplete results. The indicator must approximate the promise made to the user. An SLO is useful when it changes decisions. If nobody can say what action follows from missing it, the team has created another chart rather than an operating agreement. ## Further reading - [Google SRE Workbook: Implementing SLOs](https://sre.google/workbook/implementing-slos/) - [Google SRE Workbook: Alerting on SLOs](https://sre.google/workbook/alerting-on-slos/)