I spent four hours on a Tuesday night debugging a 30-second API call. Four hours. The call touched 12 services — auth, inventory, pricing, three different caching layers, a recommendation engine, two legacy adapters, and a handful of internal APIs that nobody remembered writing. Logs told me nothing useful. Metrics showed elevated latency somewhere in the pricing path, but “somewhere” isn’t actionable at 11pm when your on-call phone won’t stop buzzing.

I ended up adding print statements. In production. Deploying them one service at a time, tailing logs across four terminal windows, mentally stitching together request flows by matching timestamps that drifted by hundreds of milliseconds between hosts. It took me three hours to find the culprit: a downstream service was doing a synchronous DNS lookup inside a retry loop, and the resolver was timing out intermittently. Thirty minutes to fix. Three and a half hours to find.

That was the week I stopped treating distributed tracing as a nice-to-have.

If you’ve been through something similar — and if you’re running microservices, you have — then you already know why tracing matters. What you might not know is that the tooling landscape has consolidated dramatically. OpenTelemetry won. Jaeger, Zipkin, X-Ray — they’re all backends now. OTel is the instrumentation layer, and everything else plugs into it.

This guide covers how I implement OpenTelemetry tracing in practice, with Python and Go, from auto-instrumentation through custom spans, context propagation, and backend integration. I’ve written about distributed tracing fundamentals and observability patterns before — this is the hands-on OTel piece.

Why OpenTelemetry Won

The tracing ecosystem used to be fragmented. OpenTracing and OpenCensus competed for years, each with partial adoption and incompatible APIs. Teams picked one, got locked in, and resented it. Library authors didn’t want to pick sides, so most libraries shipped with no instrumentation at all.

OpenTelemetry merged both projects and did something neither could do alone: it became the default. The CNCF backed it. Every major observability platform supports OTLP natively. AWS, Google, Azure — all of them accept OTel data. When your instrumentation library works with every backend, the argument for proprietary agents disappears.

The key insight is separation of concerns. OTel handles instrumentation and data collection. Your backend handles storage and querying. You can switch from Jaeger to Tempo to Datadog without touching application code. That’s not theoretical — I’ve done it twice.

OTel also covers metrics and logs now, but tracing is where it’s most mature and where the payoff is highest. If you’re doing monitoring and observability in any distributed system, tracing is the pillar that actually tells you why things are slow.

Setting Up the SDK: Python

I’ll start with Python because it’s where most people prototype, and OTel’s Python SDK has excellent auto-instrumentation.

The core setup is three things: a resource that identifies your service, a tracer provider that manages span lifecycle, and an exporter that ships data to your backend.

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.sdk.resources import SERVICE_NAME, Resource
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

resource = Resource(attributes={
    SERVICE_NAME: "order-service",
    "deployment.environment": "production",
    "service.version": "1.4.2",
})

provider = TracerProvider(resource=resource)
exporter = OTLPSpanExporter(endpoint="http://otel-collector:4317", insecure=True)
provider.add_span_processor(BatchSpanProcessor(exporter))
trace.set_tracer_provider(provider)

That’s it for bootstrap. The BatchSpanProcessor is important — it buffers spans and sends them in batches instead of one at a time. Don’t use SimpleSpanProcessor in production unless you enjoy adding latency to every request.

Auto-Instrumentation

Here’s where OTel shines. For Flask, FastAPI, requests, SQLAlchemy, Redis, and dozens of other libraries, you get tracing without writing span code:

pip install opentelemetry-instrumentation-flask \
            opentelemetry-instrumentation-requests \
            opentelemetry-instrumentation-sqlalchemy

from opentelemetry.instrumentation.flask import FlaskInstrumentor
from opentelemetry.instrumentation.requests import RequestsInstrumentor
from opentelemetry.instrumentation.sqlalchemy import SQLAlchemyInstrumentor

FlaskInstrumentor().instrument_app(app)
RequestsInstrumentor().instrument()
SQLAlchemyInstrumentor().instrument(engine=db_engine)

Three lines and every inbound HTTP request, outbound HTTP call, and database query gets its own span with timing, status codes, and error details. The spans are automatically linked into traces via context propagation headers.

You can also skip code changes entirely and use the auto-instrumentation agent:

opentelemetry-instrument --service_name order-service flask run

I use the programmatic approach for services I own and the agent approach for third-party apps or quick experiments.

Setting Up the SDK: Go

Go’s OTel SDK is more explicit, which fits the language. No magic, no monkey-patching — you wire things up and they work.

package main

import (
	"context"
	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
	"go.opentelemetry.io/otel/sdk/resource"
	sdktrace "go.opentelemetry.io/otel/sdk/trace"
	semconv "go.opentelemetry.io/otel/semconv/v1.21.0"
)

func initTracer(ctx context.Context) (*sdktrace.TracerProvider, error) {
	exporter, err := otlptracegrpc.New(ctx,
		otlptracegrpc.WithEndpoint("otel-collector:4317"),
		otlptracegrpc.WithInsecure(),
	)
	if err != nil {
		return nil, err
	}

	tp := sdktrace.NewTracerProvider(
		sdktrace.WithBatcher(exporter),
		sdktrace.WithResource(resource.NewWithAttributes(
			semconv.SchemaURL,
			semconv.ServiceName("payment-service"),
			semconv.DeploymentEnvironment("production"),
			semconv.ServiceVersion("2.1.0"),
		)),
	)
	otel.SetTracerProvider(tp)
	return tp, nil
}

For HTTP servers, the otelhttp middleware handles inbound tracing:

import "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"

mux := http.NewServeMux()
mux.HandleFunc("/payments", handlePayment)
wrappedMux := otelhttp.NewHandler(mux, "payment-server")
http.ListenAndServe(":8080", wrappedMux)

For gRPC, there’s otelgrpc. For database calls, there’s otelsql. The pattern is always the same: wrap the thing you want traced.

Custom Spans: Where the Real Value Lives

Auto-instrumentation gets you HTTP and database spans. That’s table stakes. The real debugging power comes from custom spans around your business logic — the stuff that’s unique to your system and impossible for a library to instrument automatically.

In Python:

tracer = trace.get_tracer("order-service")

def process_order(order_id: str, items: list):
    with tracer.start_as_current_span("process_order") as span:
        span.set_attribute("order.id", order_id)
        span.set_attribute("order.item_count", len(items))

        inventory = check_inventory(items)

        with tracer.start_as_current_span("calculate_pricing") as pricing_span:
            total = calculate_total(items, inventory)
            pricing_span.set_attribute("order.total", float(total))

        if total > 10000:
            span.add_event("high_value_order_detected", {
                "order.total": float(total),
                "approval_required": True,
            })
            request_approval(order_id, total)

        return submit_order(order_id, items, total)

In Go:

func processPayment(ctx context.Context, orderID string, amount float64) error {
	ctx, span := otel.Tracer("payment-service").Start(ctx, "process_payment")
	defer span.End()

	span.SetAttributes(
		attribute.String("order.id", orderID),
		attribute.Float64("payment.amount", amount),
	)

	if err := validateCard(ctx, orderID); err != nil {
		span.RecordError(err)
		span.SetStatus(codes.Error, "card validation failed")
		return err
	}

	return chargeCard(ctx, orderID, amount)
}

A few things I’ve learned the hard way about custom spans:

Don’t create a span for every function call. You’ll drown in noise and blow up your storage costs. Span the operations that cross boundaries or take meaningful time: external calls, database queries, queue publishes, business logic that involves decisions.

Always set span.SetStatus(codes.Error, msg) when something fails. Without it, your trace viewer shows a green span that actually errored — useless for debugging.

Span events (via add_event) are underused. They’re perfect for recording decision points without creating a whole new span. “Applied discount code,” “cache miss, falling back to database,” “rate limit approaching” — these are the breadcrumbs that save you at 2am.

Context Propagation: The Glue That Holds It Together

Tracing within a single service is just fancy logging. The magic happens when trace context flows across service boundaries — that’s what turns isolated spans into a connected trace.

OTel uses the W3C Trace Context standard by default. When service A calls service B over HTTP, the instrumentation injects traceparent and tracestate headers into the outgoing request. Service B’s instrumentation extracts those headers and creates child spans under the same trace.

This works automatically if you’re using instrumented HTTP clients. But there are cases where you need to propagate context manually — message queues, batch jobs, or custom transport layers.

Here’s manual propagation through a message queue in Python:

from opentelemetry.context import get_current
from opentelemetry.propagate import inject, extract

# Producer: inject context into message headers
def publish_event(event_data: dict):
    with tracer.start_as_current_span("publish_order_event") as span:
        headers = {}
        inject(headers)  # injects traceparent into headers dict
        message = {"data": event_data, "trace_headers": headers}
        queue.publish("orders", json.dumps(message))

# Consumer: extract context from message headers
def handle_event(raw_message: str):
    message = json.loads(raw_message)
    ctx = extract(message.get("trace_headers", {}))
    with tracer.start_as_current_span("handle_order_event", context=ctx):
        process(message["data"])

In Go, the pattern is similar — you inject into a carrier on the producer side and extract on the consumer side:

// Producer
func publishEvent(ctx context.Context, data []byte) {
	ctx, span := otel.Tracer("events").Start(ctx, "publish_event")
	defer span.End()

	headers := make(map[string]string)
	otel.GetTextMapPropagator().Inject(ctx, propagation.MapCarrier(headers))
	broker.Publish("orders", data, headers)
}

// Consumer
func handleEvent(data []byte, headers map[string]string) {
	ctx := otel.GetTextMapPropagator().Extract(context.Background(),
		propagation.MapCarrier(headers))
	ctx, span := otel.Tracer("events").Start(ctx, "handle_event")
	defer span.End()
	process(ctx, data)
}

The most common propagation bug I see: someone adds a new service to the call chain and uses a raw HTTP client instead of the instrumented one. The trace breaks silently — you get two disconnected traces instead of one. If your traces keep showing up as single-service fragments, check your HTTP clients first.

The OpenTelemetry Collector: Your Telemetry Pipeline

You can export spans directly from your application to Jaeger or Tempo or whatever backend you’re using. Don’t. Put the OTel Collector in between.

The Collector is a standalone process that receives, processes, and exports telemetry data. It decouples your application from your backend, which means you can:

  • Switch backends without redeploying applications
  • Sample, filter, or enrich spans in the pipeline
  • Fan out to multiple backends simultaneously
  • Buffer data during backend outages

Here’s a practical Collector config:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 5s
    send_batch_size: 1024
  tail_sampling:
    policies:
      - name: errors
        type: status_code
        status_code: {status_codes: [ERROR]}
      - name: slow-requests
        type: latency
        latency: {threshold_ms: 2000}
      - name: baseline
        type: probabilistic
        probabilistic: {sampling_percentage: 10}

exporters:
  otlp/tempo:
    endpoint: tempo:4317
    tls:
      insecure: true
  otlp/datadog:
    endpoint: https://trace.agent.datadoghq.com
    headers:
      DD-API-KEY: ${env:DD_API_KEY}

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [tail_sampling, batch]
      exporters: [otlp/tempo, otlp/datadog]

That tail_sampling processor is critical. It keeps 100% of error traces, 100% of slow traces, and a 10% random sample of everything else. Without sampling, you’ll either go broke on storage or your backend will fall over. Head-based sampling (deciding at the start of a trace) is simpler but misses errors that happen deep in the call chain. Tail sampling waits until the trace is complete before deciding — it costs more memory in the Collector but catches the traces that actually matter.

Backend Integration: Pick Your Storage

Since OTel standardizes the instrumentation layer, your backend choice is purely about storage, querying, and cost. Here’s what I’ve used in production:

Jaeger — the original. Great UI for exploring individual traces. Struggles at scale unless you back it with Elasticsearch or Cassandra, and then you’re operating two complex systems. Fine for smaller deployments.

Grafana Tempo — my current preference. It uses object storage (S3, GCS) for trace data, which makes it dramatically cheaper than Elasticsearch-backed Jaeger. The trade-off is that you can’t search by arbitrary span attributes — you need trace IDs, which you get from logs or metrics via exemplars. Pairs beautifully with Loki and Grafana.

AWS X-Ray — works well if you’re all-in on AWS. The OTel SDK can export to X-Ray via the ADOT Collector. The service map is genuinely useful. But you’re locked into the AWS console for querying, and the pricing gets weird at high volume.

Datadog, Honeycomb, Lightstep — commercial options that handle storage, querying, and alerting. Honeycomb’s query engine is exceptional for high-cardinality exploration. If your team doesn’t want to operate tracing infrastructure, these are worth the money.

The point is: with OTel, this is a reversible decision. I’ve migrated from Jaeger to Tempo by changing the Collector config. Zero application changes.

Connecting Traces to Logs and Metrics

Traces in isolation are useful. Traces connected to logs and metrics are transformative.

The trick is correlation. OTel automatically generates a trace ID and span ID for every active span. Inject those into your log records, and suddenly you can jump from a log line to the full trace:

import logging
from opentelemetry import trace

class TraceContextFilter(logging.Filter):
    def filter(self, record):
        span = trace.get_current_span()
        ctx = span.get_span_context()
        record.trace_id = format(ctx.trace_id, '032x') if ctx.trace_id else ""
        record.span_id = format(ctx.span_id, '016x') if ctx.span_id else ""
        return True

logger = logging.getLogger(__name__)
logger.addFilter(TraceContextFilter())
handler = logging.StreamHandler()
handler.setFormatter(logging.Formatter(
    '%(asctime)s %(levelname)s [trace_id=%(trace_id)s span_id=%(span_id)s] %(message)s'
))
logger.addHandler(handler)

For metrics, Prometheus exemplars let you attach a trace ID to a specific metric observation. When you see a latency spike on a dashboard, you click the exemplar and land directly on the trace that caused it. Grafana supports this natively with Tempo.

This is the observability flywheel: metrics tell you something is wrong, logs tell you what happened, traces tell you where and why. Without the trace ID linking them, you’re back to grepping timestamps across systems.

Production Lessons

After running OTel tracing across dozens of services for a few years, here’s what I wish someone had told me upfront.

Start with auto-instrumentation, then add custom spans surgically. Don’t try to instrument everything on day one. Get auto-instrumentation running, look at the traces, and add custom spans where the gaps hurt. You’ll know where they are because you’ll see a 3-second gap between two auto-generated spans and have no idea what happened in between.

Resource attributes matter more than you think. Always include service.name, service.version, and deployment.environment. Add k8s.pod.name and k8s.namespace.name if you’re on Kubernetes. When you’re filtering traces at 3am, these attributes are the difference between finding the problem and drowning in data.

Don’t trace health checks. Your load balancer hits /health every few seconds. That’s thousands of useless traces per hour. Filter them out in the Collector or exclude them in your instrumentation config.

Watch your span cardinality. If you’re putting user IDs or request bodies into span attributes, your backend’s indexing costs will explode. Span attributes should be low-to-medium cardinality: service names, HTTP methods, status codes, endpoint patterns. Put high-cardinality data in span events or logs instead.

Test context propagation across every boundary. HTTP is easy — the instrumentation handles it. But message queues, cron jobs, background workers, and async task frameworks all need manual propagation. I’ve seen teams run OTel for months with broken propagation through their Celery workers, wondering why their traces never showed the full picture.

The Collector is not optional in production. I know I said this already. I’m saying it again because I’ve seen three teams skip it, export directly to their backend, and then spend a weekend migrating when they need to switch providers or add sampling.

Putting It All Together

Here’s what a realistic OTel deployment looks like for a team running Python and Go services on Kubernetes:

  1. Deploy the OTel Collector as a DaemonSet (one per node) or as a sidecar. I prefer DaemonSet — fewer resources, simpler config.
  2. Add the OTel SDK to each service. Use auto-instrumentation for HTTP, database, and cache libraries. Add custom spans for business logic.
  3. Configure all services to export to the Collector via OTLP gRPC on localhost:4317 (if DaemonSet) or the sidecar endpoint.
  4. Set up tail sampling in the Collector: keep all errors, keep all slow traces, sample a percentage of the rest.
  5. Export from the Collector to your backend of choice.
  6. Add trace ID injection to your logging pipeline.
  7. Configure Grafana (or your dashboard tool) to link metrics → traces → logs.

That’s the whole thing. It’s not complicated — it’s just a lot of small pieces that need to connect correctly. The OTel documentation is dense but thorough. The community Slack is genuinely helpful.

Distributed tracing changed how I debug production systems. Not incrementally — fundamentally. That 30-second API call I mentioned at the start? With tracing, I’d have had the answer in under a minute. A waterfall view showing exactly which service, which span, which operation was burning 28 of those 30 seconds. No print statements. No log archaeology. No four-hour war room.

If you’re running distributed systems without tracing, you’re choosing to debug blind. OTel makes the instrumentation part straightforward. The rest is just plumbing.