Overview
Observability is the ability to understand a system’s internal state by examining its external outputs — metrics, logs, and traces (the “three pillars”). Unlike monitoring, which answers predefined questions (“is CPU above 80%?”), observability enables asking novel questions about system behavior without deploying new instrumentation. This distinction matters most in distributed systems where failure modes are emergent and unpredictable. Observability tooling emphasizes high-cardinality, high-dimensionality data that can be sliced and diced during incident investigation.
Key Concepts
- Three Pillars — Metrics (aggregated measurements), Logs (discrete events), Traces (request paths across services). Each pillar provides a different lens on system behavior.
- Distributed tracing — Following a single request as it traverses multiple services. Tools like Jaeger and Zipkin implement the OpenTelemetry standard.
- Structured logging — Emitting logs as key-value pairs rather than free text, enabling machine parsing and correlation.
- Correlation IDs — Unique identifiers propagated across service boundaries to link related events.
- Cardinality — The number of unique values a dimension can take. High cardinality (e.g., user ID) enables precise debugging but costs more to store.
- OpenTelemetry — Vendor-neutral standard for instrumenting, generating, and collecting telemetry data.
Trade-offs
- Full observability generates significant data volume and cost
- Instrumenting legacy systems requires substantial refactoring
- Over-reliance on traces can miss system-level issues visible only in metrics
Related
- Monitoring is the foundation observability builds upon
- SRE uses observability to maintain service level objectives
- Microservices architectures demand strong observability due to distributed complexity