Kodstigen
← Logging and Observability

Lesson 6 of 7

Dashboards and Centralized Log Aggregation

With logs, metrics, and traces flowing from every service, you need somewhere to actually collect, search, and visualize all of it, especially once you have more than one service or more than one server.

Why centralize logs

If every service's logs only live on its own machine/container, investigating anything that spans services (which is most real incidents) means SSHing into multiple machines and manually correlating timestamps, painfully slow, and impossible once containers get recycled and their local logs disappear with them.

A log aggregation pipeline ships logs from every service to one central, searchable place:

Service A logs ─┐
Service B logs ─┼──→  Log shipper  →  Central storage/index  →  Search UI
Service C logs ─┘

Common building blocks: Fluentd/Fluent Bit or Logstash as the shipper, Elasticsearch or a similar index as storage, Kibana or Grafana as the search/visualization UI, this combination (Elasticsearch + Logstash + Kibana) is common enough to have its own acronym, the "ELK stack".

Building a useful dashboard

A good dashboard answers "is everything okay right now?" at a glance, and helps narrow down "what's wrong?" within seconds if not. A solid starting point for almost any service:

  • Request rate (traffic volume over time)
  • Error rate (percentage of failed requests)
  • Latency (p50 and p99, not just an average, which hides outliers)
  • Saturation (how close a resource, CPU/memory/queue depth, is to its limit)

This is sometimes called the "four golden signals" (from Google's SRE book), covering both "are users having a bad time" (rate/errors/latency) and "are we about to have a bad time" (saturation).

Correlating across pillars in one place

The real payoff of centralizing everything: from one dashboard, you can jump from a metric spike, to the traces active during that window, to the specific logs from the exact service and time range the trace pointed at, without re-typing timestamps or switching between five different tools' worth of context.

TIP

Don't build a dashboard with 40 panels nobody looks at. Start with the four golden signals for your most important service, and only add more once you've actually needed something else while debugging a real incident.

📝 Dashboards & Log Aggregation Quiz

Passing score: 70%

  1. 1.What does the ELK stack stand for?

  2. 2.Investigating an incident by SSHing into each service's machine individually is a scalable, recommended approach for a multi-service system.

  3. 3.The 'four golden signals' from Google's SRE book are latency, traffic, errors, and ____.

  4. 4.Why should latency be shown as p50/p99 rather than just an average?

  5. 5.A good dashboard should have as many panels as possible to show every available metric.

  6. 6.Central log storage lets you jump from a metric spike to related traces to the exact ____ from that time range, all in one place.