Overview
Site Reliability Engineering (SRE) is a discipline that applies software engineering practices to infrastructure and operations, originally developed at Google. SRE treats operations as a software problem, using automation, error budgets, and service level objectives (SLOs) to balance reliability with feature velocity. The core insight is that 100% reliability is neither achievable nor desirable — the marginal cost of each additional “nine” of availability increases exponentially while user perception plateaus. Error budgets formalize this trade-off: when the budget is healthy, teams ship fast; when it’s depleted, they focus on reliability.
Key Concepts
- Service Level Objectives (SLOs) — Target reliability expressed as a percentage (e.g., 99.9% availability). Derived from user expectations, not engineering ambition.
- Service Level Indicators (SLIs) — The metrics that measure whether SLOs are met (latency percentiles, error rates, throughput).
- Error budgets — The allowed amount of unreliability (100% minus SLO). Consumed by outages, slow responses, or errors.
- Toil — Repetitive, manual, automatable operational work. SRE aims to keep toil below 50% of team time.
- Blameless postmortems — Incident reviews focused on systemic improvements rather than individual blame.
- Capacity planning — Forecasting resource needs based on growth trends and load testing.
Trade-offs
- Requires organizational buy-in to error budget enforcement
- SLO-driven development may slow feature work when budgets are depleted
- Toil elimination requires upfront investment in automation
- Cultural shift from reactive ops to proactive engineering takes time
Related
- Monitoring provides the SLIs that feed into SLO calculations
- Observability enables the deep investigation needed for blameless postmortems
- Infrastructure as Code is a key toil-reduction practice in SRE