Courses › System Design › Core Concepts

Observability

Metrics, structured logs, distributed tracing, SLO-based alerting and health checks.

Observability is the ability to understand what your system is doing from the outside — to answer "is it healthy?" and, when it isn't, "why?" In interviews it often comes up at the end ("how would you know this is working?") and in operations-focused questions like Metrics & Monitoring. A strong answer names what you'd measure, how you'd alert on it, and how you'd debug a single bad request.

The three pillars

OpenTelemetry is the standard way to instrument code for all three.

Metrics

  • Measure the RED metrics for every service: Rate (requests/s), Errors (failed requests/s), Duration (latency distribution).
  • Measure the USE metrics for resources: Utilization, Saturation (queue length), Errors.
  • Report latency as percentiles (p50, p95, p99), never just the average — averages hide the slow tail that users notice.
  • Keep label cardinality low (service, endpoint, status), never user IDs. See Time-Series Databases.
  • Add business metrics — orders per minute, messages delivered, sign-ups — which often catch problems that technical metrics miss.

Logs

  • Use structured logs (JSON with fields) so they can be searched and aggregated, not free text.
  • Include a request ID / trace ID in every log line so you can find everything about one request across services.
  • Ship logs asynchronously through an agent, often via Kafka, to a central store with retention tiers (hot for 7 days, archived to S3 after).
  • Sample high-volume debug logs; always keep errors. Never log secrets or personal data.

Distributed tracing

[Gateway: 820 ms] -> [Order service: 790 ms] -> [Inventory: 40 ms]
[Order service: 790 ms] -> [Payment service: 700 ms] -> [Card provider: 680 ms]
  • A trace is a tree of spans; each span records a service, operation, start time and duration.
  • The trace context is propagated in request headers (traceparent) and message metadata, so the trace continues across queues too.
  • Traces instantly show which hop is slow — here, the card provider.
  • Trace volume is large, so use head sampling (keep 1% of all traces) or tail sampling (keep every slow or failed trace, plus a sample of the rest).

Alerting

  • Alert on symptoms users feel — error rate, latency, SLO burn — rather than every possible cause (CPU at 80%).
  • Use SLO burn-rate alerts: page when the error budget is being consumed fast enough to run out within hours; open a ticket for slow burns. See Availability & Fault Tolerance.
  • Every page should be actionable and linked to a runbook and dashboard. Noisy alerts get ignored.

Health checks and dashboards

  • Liveness — is the process alive? Readiness — can it serve traffic (dependencies reachable, warmed up)? Load balancers and Kubernetes use them to route around bad instances.
  • Keep one service dashboard per service (RED metrics, saturation, dependencies) and one business dashboard for the product's key flows.

[!INTERVIEW] How you'd know it's working "Each service emits RED metrics and structured logs with a trace ID, and we use OpenTelemetry tracing with tail sampling so every failed or slow request is kept. The SLO is 99.9% of message sends succeeding within 500 ms; we page on a fast burn rate. A dashboard also tracks messages delivered per minute, which catches silent failures that don't produce errors."