{"slug":"observability-opentelemetry-guide","title":"Full-Stack Observability with OpenTelemetry: From Chaos to Clarity","description":"Implementing unified observability across 120+ microservices with OpenTelemetry, Prometheus, Grafana, and SLO-based alerting","date":"2025-03-10","author":"Elaph Hilful","thumbnail":"/blog-images/observability-stack.svg","tags":["Observability","OpenTelemetry","Prometheus","Grafana","SRE"],"category":"SRE","featured":false,"readTime":6,"htmlContent":"

When you're operating 120+ microservices processing 50TB of telemetry data per day, traditional monitoring breaks. Alert fatigue, disconnected tools, and 15-minute detection times become existential threats to reliability. OpenTelemetry provides the unified foundation to correlate traces, metrics, and logs into actionable insights.

\n

This is the story of building an observability platform that reduced alert noise by 92% and achieved sub-30-second mean time to detection.

\n

\"Full-Stack

\n

The Problem: Observability Sprawl

\n

Before the unified platform, our stack looked like this:

\n\n

The result? Engineers spent 40 hours/week on incident triage, MTTD was 15+ minutes, and alert fatigue was causing real incidents to be missed.

\n

The Three Pillars, Unified

\n

OpenTelemetry provides a single SDK and protocol for all three telemetry signals. Here's how we deployed it across our Go and Node.js services:

\n

Instrumentation: Auto + Manual

\n
// main.go - Go service with OpenTelemetry\npackage main\n\nimport (\n    \"context\"\n    \"go.opentelemetry.io/otel\"\n    \"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp\"\n    \"go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp\"\n    \"go.opentelemetry.io/otel/sdk/trace\"\n    \"go.opentelemetry.io/otel/sdk/metric\"\n    \"go.opentelemetry.io/otel/sdk/resource\"\n    semconv \"go.opentelemetry.io/otel/semconv/v1.21.0\"\n)\n\nfunc initTelemetry(ctx context.Context) (func(), error) {\n    res := resource.NewWithAttributes(\n        semconv.SchemaURL,\n        semconv.ServiceName(\"payment-service\"),\n        semconv.ServiceVersion(\"1.2.0\"),\n        semconv.DeploymentEnvironment(\"production\"),\n    )\n\n    // Trace exporter\n    traceExporter, _ := otlptracehttp.New(ctx,\n        otlptracehttp.WithEndpoint(\"otel-collector:4318\"),\n        otlptracehttp.WithInsecure(),\n    )\n\n    tp := trace.NewTracerProvider(\n        trace.WithBatcher(traceExporter),\n        trace.WithResource(res),\n        trace.WithSampler(trace.ParentBased(\n            trace.TraceIDRatioBased(0.1), // 10% sampling\n        )),\n    )\n    otel.SetTracerProvider(tp)\n\n    // Metric exporter\n    metricExporter, _ := otlpmetrichttp.New(ctx,\n        otlpmetrichttp.WithEndpoint(\"otel-collector:4318\"),\n    )\n\n    mp := metric.NewMeterProvider(\n        metric.WithReader(metric.NewPeriodicReader(metricExporter)),\n        metric.WithResource(res),\n    )\n    otel.SetMeterProvider(mp)\n\n    return func() {\n        tp.Shutdown(ctx)\n        mp.Shutdown(ctx)\n    }, nil\n}\n
\n

OpenTelemetry Collector: The Central Hub

\n

The Collector receives, processes, and exports telemetry to backend stores:

\n
# otel-collector-config.yaml\nreceivers:\n  otlp:\n    protocols:\n      grpc:\n        endpoint: 0.0.0.0:4317\n      http:\n        endpoint: 0.0.0.0:4318\n\n  # Scrape Prometheus endpoints from Kubernetes pods\n  prometheus:\n    config:\n      scrape_configs:\n        - job_name: \"kubernetes-pods\"\n          kubernetes_sd_configs:\n            - role: pod\n          relabel_configs:\n            - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]\n              action: keep\n              regex: true\n\nprocessors:\n  batch:\n    timeout: 5s\n    send_batch_size: 1024\n\n  # Add Kubernetes metadata to all signals\n  k8sattributes:\n    extract:\n      metadata:\n        - k8s.pod.name\n        - k8s.namespace.name\n        - k8s.deployment.name\n        - k8s.node.name\n\n  # Tail-based sampling: keep error traces, sample normal ones\n  tail_sampling:\n    policies:\n      - name: errors-always\n        type: status_code\n        status_code: { status_codes: [ERROR] }\n      - name: slow-traces\n        type: latency\n        latency: { threshold_ms: 1000 }\n      - name: probabilistic\n        type: probabilistic\n        probabilistic: { sampling_percentage: 10 }\n\n  # Memory limiter to prevent OOM\n  memory_limiter:\n    check_interval: 1s\n    limit_mib: 2048\n    spike_limit_mib: 512\n\nexporters:\n  # Traces to Tempo\n  otlp/tempo:\n    endpoint: tempo:4317\n    tls:\n      insecure: true\n\n  # Metrics to Mimir (Prometheus-compatible)\n  prometheusremotewrite:\n    endpoint: http://mimir:9009/api/v1/push\n    resource_to_telemetry_conversion:\n      enabled: true\n\n  # Logs to Loki\n  loki:\n    endpoint: http://loki:3100/loki/api/v1/push\n    labels:\n      resource:\n        service.name: \"service\"\n        k8s.namespace.name: \"namespace\"\n\nservice:\n  pipelines:\n    traces:\n      receivers: [otlp]\n      processors: [memory_limiter, k8sattributes, tail_sampling, batch]\n      exporters: [otlp/tempo]\n    metrics:\n      receivers: [otlp, prometheus]\n      processors: [memory_limiter, k8sattributes, batch]\n      exporters: [prometheusremotewrite]\n    logs:\n      receivers: [otlp]\n      processors: [memory_limiter, k8sattributes, batch]\n      exporters: [loki]\n
\n

SLO-Based Alerting: The End of Alert Fatigue

\n

The biggest transformation wasn't technical — it was philosophical. We moved from threshold-based alerting (CPU > 80%? Alert!) to SLO-based alerting (error budget burning too fast? Alert!).

\n

Defining SLOs

\n
# slo-definitions/payment-service.yaml\napiVersion: sloth.slok.dev/v1\nkind: PrometheusServiceLevel\nmetadata:\n  name: payment-service-availability\nspec:\n  service: payment-service\n  labels:\n    team: payments\n    tier: critical\n\n  slos:\n    - name: availability\n      objective: 99.95 # 99.95% availability target\n      description: \"Payment transactions complete successfully\"\n      sli:\n        events:\n          error_query: |\n            sum(rate(http_requests_total{\n              service=\"payment-service\",\n              status_code=~\"5..\"\n            }[{{.window}}]))\n          total_query: |\n            sum(rate(http_requests_total{\n              service=\"payment-service\"\n            }[{{.window}}]))\n\n      alerting:\n        name: PaymentAvailabilityBurnRate\n        labels:\n          category: availability\n        annotations:\n          summary: \"Payment service error budget burning too fast\"\n        page_alert:\n          labels:\n            severity: critical\n        ticket_alert:\n          labels:\n            severity: warning\n
\n

Burn Rate Alerts in Practice

\n

Instead of alerting on every 500 error, we alert on burn rate — how fast the error budget is being consumed:

\n
# Multi-window burn rate alert (Google SRE book approach)\n# Page if burning 14.4x budget over 1h AND 6x over 5m\n(\n  error_budget_burn_rate:1h{service=\"payment-service\"} > 14.4\n  and\n  error_budget_burn_rate:5m{service=\"payment-service\"} > 14.4\n)\nor\n# Ticket if burning 3x budget over 6h AND 1x over 30m\n(\n  error_budget_burn_rate:6h{service=\"payment-service\"} > 3\n  and\n  error_budget_burn_rate:30m{service=\"payment-service\"} > 3\n)\n
\n

This approach reduced our daily alert volume from 500+ to under 40 — a 92% reduction — while catching every real incident faster.

\n

Distributed Tracing: Following Requests Across 120 Services

\n

With OpenTelemetry's context propagation, a single trace follows a request across all services:

\n
// Node.js - Adding custom spans for business logic\nimport { trace, SpanStatusCode } from \"@opentelemetry/api\";\n\nconst tracer = trace.getTracer(\"checkout-service\");\n\nasync function processCheckout(order: Order) {\n  return tracer.startActiveSpan(\"process-checkout\", async (span) => {\n    span.setAttribute(\"order.id\", order.id);\n    span.setAttribute(\"order.total\", order.total);\n    span.setAttribute(\"order.items_count\", order.items.length);\n\n    try {\n      // Each of these creates child spans automatically\n      const inventory = await checkInventory(order.items);\n      const payment = await chargePayment(order.total);\n      const shipment = await createShipment(order);\n\n      span.setStatus({ code: SpanStatusCode.OK });\n      return { inventory, payment, shipment };\n    } catch (error) {\n      span.setStatus({\n        code: SpanStatusCode.ERROR,\n        message: error.message,\n      });\n      span.recordException(error);\n      throw error;\n    } finally {\n      span.end();\n    }\n  });\n}\n
\n

Grafana: The Unified Dashboard

\n

In Grafana, we correlate all three signals:

\n
    \n
  1. Dashboard shows SLO burn rate trending upward
    2. \n
  2. Click through to see error rate spike in metrics
    3. \n
  3. Exemplar links jump directly to the failing trace
    4. \n
  4. Trace view shows the exact service and span that failed
    5. \n
  5. Log link shows the structured error with full context
    6. \n
\n

All in under 30 seconds from detection to root cause.

\n

Results: The Numbers Speak

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
MetricBeforeAfter
Mean Time to Detection15+ minutes< 30 seconds
Daily alert volume500+< 40
Hours/week on triage404
Trace coverage20% of services100%
Telemetry data/day5TB (fragmented)50TB (unified)
Cost per GB telemetry$2.80$0.40
\n

Key Architecture Decisions

\n
    \n
  1. \n

    Collector-per-node, not per-pod: DaemonSet deployment reduces resource overhead and provides node-level visibility.

    \n
    2. \n
  2. \n

    Tail-based sampling: Keep all error traces, sample normal ones. This gives 100% visibility into failures while controlling storage costs.

    \n
    3. \n
  3. \n

    Grafana Mimir over vanilla Prometheus: Mimir provides long-term storage, multi-tenancy, and horizontal scaling that standalone Prometheus can't match at 50TB/day.

    \n
    4. \n
  4. \n

    Structured logs, not free text: Every log line is JSON with correlation IDs. Loki queries against labels are 100x faster than full-text search.

    \n
    5. \n
  5. \n

    SLOs before SLIs: Define what \"reliable\" means to users first, then instrument to measure it. Not the other way around.

    \n
    6. \n
\n

Getting Started

\n

The migration path that worked for us:

\n
    \n
  1. Week 1-2: Deploy OTel Collector and basic Grafana stack
    2. \n
  2. Week 3-4: Instrument top 10 critical services
    3. \n
  3. Month 2: Define SLOs for critical services, set up burn rate alerts
    4. \n
  4. Month 3: Roll out to all services, disable old monitoring
    5. \n
  5. Ongoing: Refine SLOs based on user impact data
    6. \n
\n

The hardest part isn't the technology — it's convincing teams to trust SLO-based alerting over their beloved threshold alerts. Let the numbers prove it.

"}