keep distributed traces continuous across every service boundary
Cross-service context propagation in microservices: practical OpenTelemetry patterns
14 min read
Broken traces rarely mean missing instrumentation—they mean a lost traceparent at an HTTP hop, Kafka header, gRPC metadata field, or background thread. Master inject, transport, and extract so every span stays on the same request journey.
Why traces break even when every service is instrumented
A user request that fans out across HTTP APIs, gRPC stubs, and message queues produces a useful waterfall only when one shared trace identity survives every hop. Teams install OpenTelemetry SDKs, then still see orphan roots: Service B starts a new trace because Service A omitted the W3C traceparent header; a Kafka consumer never sees producer headers; a cron worker rebuilds context from scratch; gRPC metadata is never wired; a batch consumer folds a hundred messages into one parentless span. Each gap raises MTTR—on-call greps timestamps instead of opening one continuous trace. Instrumentation without propagation is noise. This article focuses on the boundary patterns; collector topology and Kubernetes auto-injection are covered in our companion tracing guides.
The model: inject, transport, extract
OpenTelemetry propagation is always the same three steps. Inject writes the active span context into a carrier—HTTP headers, gRPC metadata, or messaging headers—usually as W3C Trace Context fields traceparent and tracestate, optionally plus baggage. Transport is the wire protocol that must preserve those fields unchanged. Extract rebuilds an OpenTelemetry Context on the receiver so the next span becomes a child of the upstream span. Prefer auto-instrumentation for frameworks; fall back to the propagate API only for custom clients. Configure propagators once at process start with OTEL_PROPAGATORS or set_global_textmap—defaults are already tracecontext and baggage.
from opentelemetry import trace
from opentelemetry.propagate import extract, inject
tracer = trace.get_tracer("order-service")
def call_downstream(client):
with tracer.start_as_current_span("call-inventory") as span:
span.set_attribute("peer.service", "inventory")
headers = {}
inject(headers) # writes traceparent, tracestate, baggage
return client.get("http://inventory/api/stock", headers=headers)
def handle_incoming(request):
ctx = extract(dict(request.headers))
with tracer.start_as_current_span("handle-request", context=ctx) as span:
span.set_attribute("http.route", "/api/orders")
return call_downstream(http_client)from opentelemetry.propagate import set_global_textmap
from opentelemetry.propagators.b3 import B3MultiFormat
from opentelemetry.propagators.composite import CompositePropagator
from opentelemetry.baggage.propagation import W3CBaggagePropagator
from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator
# Prefer OTEL_PROPAGATORS=tracecontext,baggage,b3multi in production.
set_global_textmap(
CompositePropagator(
[
TraceContextTextMapPropagator(),
W3CBaggagePropagator(),
B3MultiFormat(),
]
)
)HTTP and gRPC: framework interceptors first
For HTTP, use language auto-instrumentation—FastAPI, Flask, requests, httpx, Go net/http wrappers—so every outbound client and inbound server path inherits context. Manual inject is a last resort for hand-rolled clients. For gRPC, register client and server instrumentors so metadata carries the same W3C fields; do not invent a parallel header scheme. Keep tenant or region identifiers in baggage or application metadata deliberately, never as a substitute for traceparent. Log correlation should read the active span's trace_id into structured logs so dashboards can jump from log line to trace.
from concurrent import futures
import grpc
from opentelemetry.instrumentation.grpc import (
GrpcInstrumentorClient,
GrpcInstrumentorServer,
)
GrpcInstrumentorClient().instrument()
GrpcInstrumentorServer().instrument()
channel = grpc.insecure_channel("inventory:50051")
stub = InventoryStub(channel)
server = grpc.server(futures.ThreadPoolExecutor(max_workers=10))
# Register servicers, then start — interceptors are already active.Kafka and other async messaging boundaries
Queues are the most common silent break. Producers must put trace context into message headers; consumers must extract before business spans. KafkaInstrumentor wraps kafka-python producers and consumers for PRODUCER and CONSUMER spans. Consumer iterators can still drop context into user code—extract headers yourself and pass context= into start_as_current_span for the business span. Apply the same pattern to RabbitMQ properties, SQS message attributes, and NATS headers: one carrier dictionary, inject on publish, extract on consume. Prefer messaging semantic conventions (messaging.system, messaging.destination.name) so backends group async hops correctly.
import json
from kafka import KafkaConsumer, KafkaProducer
from opentelemetry import trace
from opentelemetry.instrumentation.kafka import KafkaInstrumentor
from opentelemetry.propagate import extract, inject
KafkaInstrumentor().instrument()
tracer = trace.get_tracer("order-service")
producer = KafkaProducer(bootstrap_servers="kafka:9092")
def publish_order(order):
with tracer.start_as_current_span("publish-order") as span:
span.set_attribute("order.id", order["id"])
span.set_attribute("messaging.system", "kafka")
span.set_attribute("messaging.destination.name", "orders")
headers = {}
inject(headers)
producer.send(
"orders",
key=order["id"].encode(),
value=json.dumps(order).encode(),
headers=[(k, v.encode()) for k, v in headers.items()],
)
def consume_orders(consumer: KafkaConsumer):
for msg in consumer:
carrier = {
key: (value.decode() if isinstance(value, (bytes, bytearray)) else value)
for key, value in (msg.headers or [])
}
ctx = extract(carrier)
with tracer.start_as_current_span("consume-order", context=ctx) as span:
order = json.loads(msg.value)
span.set_attribute("order.id", order["id"])
reserve_inventory(order)Baggage, threads, and span links for batches
Baggage propagates application keys such as tenant.id or feature.flag alongside the trace. Use it sparingly—every hop pays header size, and secrets must never ride in baggage. When you spawn a thread or submit a task to an executor, copy the context and attach it inside the worker; otherwise the child span becomes a new root. For batch fan-out, parent-child hierarchy is often wrong: create independent processing spans and attach span links to the originating contexts so backends show causal references without forcing a single parent.
import threading
from opentelemetry import baggage, context, trace
tracer = trace.get_tracer("checkout")
def enqueue_with_tenant(tenant_id: str):
ctx = baggage.set_baggage("tenant.id", tenant_id)
token = context.attach(ctx)
try:
with tracer.start_as_current_span("enqueue"):
captured = context.get_current()
def worker():
restore = context.attach(captured)
try:
with tracer.start_as_current_span("background-work"):
assert baggage.get_baggage("tenant.id") == tenant_id
do_work()
finally:
context.detach(restore)
threading.Thread(target=worker).start()
finally:
context.detach(token)from opentelemetry import trace
from opentelemetry.propagate import extract
tracer = trace.get_tracer("batch-worker")
def process_batch(messages):
with tracer.start_as_current_span("batch-process") as batch_span:
batch_span.set_attribute("messaging.batch.message_count", len(messages))
for message in messages:
carrier = dict(message.headers)
extracted = extract(carrier)
linked = trace.get_current_span(extracted).get_span_context()
with tracer.start_as_current_span(
"process-item",
links=[trace.Link(linked)],
) as item_span:
item_span.set_attribute("messaging.message.id", message.id)
handle(message)Prove propagation with tests and coverage signals
Add an integration check that injects on the client and asserts the downstream service continues the same trace_id—compare formatted hex strings, not raw integers against custom response headers unless you explicitly echo format_trace_id. In backends, alert on a rising share of root spans for services that should almost always have a parent (API gateways excepted). Trace pipeline metrics such as exporter failures still matter, but orphan-root ratio is the clearest propagation health signal. Keep head sampling parent-based so a sampled root keeps children; otherwise mid-chain drops look like propagation bugs.
Operational checklist for continuous traces
Standardize on W3C Trace Context everywhere; dual-run B3 only during migrations. Instrument libraries at the framework edge before writing manual inject calls. Audit every async producer and consumer for headers. Forbid PII in baggage and span attributes. Correlate logs with trace_id and span_id. Review orphan roots after each new queue or gRPC client lands. Context propagation is the invisible contract of distributed tracing—fix inject and extract at every boundary, and the next cross-service incident becomes a single waterfall instead of an archaeology project.
Cluster-wide collectors, sampling, and auto-instrumentation build on our OpenTelemetry distributed tracing guide for Kubernetes.
Export and process continuous traces through our production OpenTelemetry Collector pipeline guide.