Distributed Tracing

A single user request in a microservices system might touch five, ten, or more services before a response comes back. When it's slow, "logs from each service" don't easily show you the full, connected picture, distributed tracing does.

Traces, spans, and context propagation

A trace represents one end-to-end request, made up of multiple spans, each span is one unit of work (a single service call, a database query) with a start time, duration, and metadata.

Trace: checkout-req-9f8a
├── span: API gateway            (2ms)
├── span: auth service check     (12ms)
├── span: inventory service call (340ms)  ← the slow one
│   └── span: inventory DB query (335ms)
└── span: payment service call   (45ms)

Each span records its parent span, building the tree above, so a tracing tool can render exactly where time was spent across the whole request, at a glance.

Propagating context across services

For this to work, a trace context (trace ID + current span ID) has to travel with the request as it hops between services, usually via an HTTP header:

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01

Every service in the chain reads the incoming trace context, creates its own span as a child of it, and passes the updated context along to whatever it calls next. Miss this step in even one service, and the trace breaks into two disconnected pieces there.

OpenTelemetry

OpenTelemetry (OTel) is the current industry-standard, vendor-neutral way to instrument code for traces (and logs and metrics), instrument your code once with OTel's libraries, and send the data to whichever backend you choose (many are compatible), instead of locking your instrumentation to one specific vendor's proprietary SDK.

import { trace } from '@opentelemetry/api';

const tracer = trace.getTracer('inventory-service');

async function checkStock(itemId) {
  return tracer.startActiveSpan('checkStock', async (span) => {
    const result = await db.query(...);
    span.end();
    return result;
  });
}