Distributed Tracing Foundations & The Jaeger v2 Architecture Shift

Series: The Jaeger Engineering Reference (Article 1 of 10)
Official Repository: jaegertracing/jaeger
Documentation: jaegertracing.io/docs
Key Upstream Sources: cmd/jaeger/, cmd/jaeger/main.go, cmd/jaeger/internal/extension/jaegerquery/

1. The Microservices Blind Spot: Why Tracing Exists#

In a classical monolithic architecture, an incoming request is serviced within a single operating system process boundary. When an endpoint encounters high latency or crashes, diagnosing the failure is relatively direct: an engineer inspects thread dumps, local application logs, or a stack trace using thread-local storage identifiers.

In modern cloud-native distributed architectures, a single end-user action (such as clicking "Place Order" or invoking POST /checkout) fans out across dozens of decoupled microservices communicating across heterogeneous network transports (gRPC, HTTP/REST, Kafka, Redis).

User            API Gateway            Order Service           Payment / Inventory            Database
                 |                   |                      |                           |                        |
                 +--POST /checkout-->|                      |                           |                        |
                 |   [Root Span: T1-S1]                     |                           |                        |
                 |                   +--gRPC CreateOrder--->|                           |                        |
                 |                   |  (W3C: T1-S1)        | [Child Span: T1-S2]       |                        |
                 |                   |                      +--HTTP /charge (T1-S2)---->|                        |
                 |                   |                      |                           | [Span: T1-S3]          |
                 |                   |                      |<--200 OK------------------+                        |
                 |                   |                      +--gRPC ReserveStock------->|                        |
                 |                   |                      |   (W3C: T1-S2)            | [Span: T1-S4]          |
                 |                   |                      |                           +--SQL UPDATE items----->|
                 |                   |                      |                           |<--OK-------------------+
                 |                   |                      |<--Stock Reserved----------+                        |
                 |                   |<--Order Created------+                           |                        |
                 |<--201 Created-----+                      |                           |                        |

sequenceDiagram
                    autonumber
                    actor User
                    participant Gateway as API Gateway
                    participant OrderSvc as Order Service
                    participant PaymentSvc as Payment Service
                    participant InventorySvc as Inventory Service
                    participant DB as Postgres / Cassandra
                
                    User->>Gateway: POST /checkout
                    Note over Gateway: Creates Root Span (TraceID: T1, SpanID: S1)
                    Gateway->>OrderSvc: gRPC CreateOrder (Inject W3C traceparent: T1-S1)
                    Note over OrderSvc: Extracts Context, Child Span (TraceID: T1, SpanID: S2, ParentID: S1)
                    par Concurrent Calls
                        OrderSvc->>PaymentSvc: HTTP /charge (traceparent: T1-S2)
                        Note over PaymentSvc: Child Span (TraceID: T1, SpanID: S3, ParentID: S2)
                        PaymentSvc-->>OrderSvc: 200 OK
                    and
                        OrderSvc->>InventorySvc: gRPC ReserveStock (traceparent: T1-S2)
                        Note over InventorySvc: Child Span (TraceID: T1, SpanID: S4, ParentID: S2)
                        InventorySvc->>DB: SQL UPDATE items
                        DB-->>InventorySvc: OK
                        InventorySvc-->>OrderSvc: Stock Reserved
                    end
                    OrderSvc-->>Gateway: Order Created
                    Gateway-->>User: 201 Created

The Three Fundamental Distributed Systems Blind Spots#

Traditional observability tooling (server metrics and isolated log collectors) fails to provide actionable insight in this environment due to three structural blind spots:

1. Network Boundary Obfuscation (Causal Erasure):
   • The network boundary strips execution context. Operating system Thread IDs and Process IDs cannot cross TCP sockets or messaging queues.
   • When Service A calls Service B, Service B has no native knowledge of who invoked it or which upstream user transaction initiated the request.
2. Independent Log Silos (The Needle in a Haystack):
   • High-throughput microservices independently write tens of thousands of log lines per second into disjoint log aggregates (Elasticsearch, Loki, CloudWatch).
   • Searching across disparate log buckets without a globally unified correlation ID requires manual timestamp cross-referencing, which breaks under clock drift and asynchronous concurrency.
3. The P99 Latency Mystery (Concurrency & Contention):
   • An API endpoint exhibiting an average latency of 80ms may suddenly spike to a 99th percentile (P99) latency of 3.8 seconds.
   • Standard resource telemetry (host CPU usage, memory utilization, network bandwidth) appears completely normal because average resource consumption is low.
   • Distributed tracing exposes the actual bottleneck: the Order Service spent 3.7 seconds waiting in an asynchronous queue or blocked on a transactional row lock inside PostgreSQL.

2. The Core Anatomy of a Trace & W3C Specifications#

Jaeger implements the distributed tracing data model originally formalized in Google Dapper research paper (Dapper, a Large-Scale Distributed Systems Tracing Infrastructure, Sigelman et al., 2010) and standardized today by the OpenTelemetry and W3C specifications.

       [Span A: Root (HTTP POST /checkout)]
                                  +-- [Span B: gRPC CreateOrder]
                                  |          +-- [Span C: HTTP POST /charge]
                                  |          +-- [Span D: gRPC ReserveStock]
                                  |                     +-- [Span E: SQL UPDATE items]
                                  +-- [Span F: Background Audit Log Event]

1. Fundamental Entities: Traces vs Spans#

• Trace: A Directed Acyclic Graph (DAG) of Spans representing an entire end-to-end transaction journey through a distributed topology.
• Span: The atomic building block of a trace. A span represents a single contiguous unit of timed work executed by a specific component or service.

2. Anatomy of a Span#

Every span encapsulates the following attributes:

3. Context Propagation & The W3C traceparent Standard#

To preserve execution context across network boundaries, services inject and extract standard metadata headers defined by the W3C Trace Context specification:

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
                             |  +--------------+-----------------+ +-------+--------+ +-++
                          version         16-byte TraceID             8-byte SpanID    flags

• Version (00): Current W3C specification version.
• Trace ID (4bf92f35...): 16-byte random hex string identifying the overall trace.
• Parent ID / Span ID (00f067aa...): 8-byte hex string identifying the calling span.
• Trace Flags (01): 8-bit bitmap field; 01 signals that the trace was recorded / sampled (FLAG_SAMPLED).

W3C Context Headers:
                  +-- traceparent : 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
                  +-- tracestate  : rojo=123,congo=456 (Vendor-specific state passing)
                  +-- baggage     : tenant=acme,tier=premium (Distributed application context)

• tracestate: Carries vendor-specific routing state across multiple tracing systems without parsing or modifying the core traceparent.
• baggage: Carries arbitrary key-value pairs (e.g. tenant_id=enterprise) accessible to downstream services across the entire trace journey.

3. The Jaeger v1 vs v2 Architectural Paradigm Shift#

Historically, Jaeger operated as a bespoke ecosystem of Go microservices designed during the early microservices era:

• jaeger-agent: A local host daemon that accepted spans over Thrift UDP packets on port 6831.
• jaeger-collector: A custom Go application that received spans, validated them, buffered them in in-memory Go channel queues, and wrote them to storage plugins.
• jaeger-query: A standalone HTTP/gRPC server hosting the Jaeger UI and querying storage backends.
• jaeger-ingester: A specialized daemon consuming spans from Apache Kafka topics to decouple ingestion from storage writes.

Architectural Bottlenecks in Jaeger v1#

1. UDP Packet Loss Under Load: The jaeger-agent relied on Thrift over UDP to avoid connection pooling overhead in client runtimes. Under network congestion or buffer saturation, UDP packets were silently dropped with zero backpressure.
2. In-Memory Channel Bottlenecks: jaeger-collector processed individual spans sequentially through Go channels. When databases slowed down, channel buffers overflowed, dropping spans (dropped_spans metric).
3. Plumbing Duplication: Jaeger maintainers had to build, maintain, and secure custom TLS handshakes, HTTP/gRPC servers, health checks, and metrics exporters, duplicating work done by the wider Cloud Native Computing Foundation (CNCF) ecosystem.

The Jaeger v2 Architecture: An OpenTelemetry Collector Distribution#

In Jaeger v2, the architecture underwent a complete re-platforming: Jaeger v2 is built directly on top of the OpenTelemetry Collector framework (go.opentelemetry.io/collector).

Jaeger v1 (Legacy Monolith & Custom Microservices):
                  [Client App] --Thrift UDP--> [jaeger-agent] --Channel Queue--> [jaeger-collector] --> [DB Storage]
                                                                                                             ^
                  [User UI] <--REST/gRPC API----------------------------------- [jaeger-query] -------------+
                
                Jaeger v2 (Unified OpenTelemetry Collector Base):
                  [Client App] --OTLP gRPC/HTTP (:4317/:4318)--+
                                                               v
                  +--------------------------------- OTel Collector Engine ---------------------------------+
                  | Receivers (OTLP, Jaeger, Zipkin) --> Processors (Batch, Sampling) --> Exporters (Storage)|
                  +--------------------------------------------+--------------------------------------------+
                                                               v
                                              [internal/storage/v2/api/tracestore]
                                                               |
                                       +-----------------------+-----------------------+
                                       v                                               v
                  [Backends: ClickHouse, ES, Cassandra, Memory]     [Extensions: jaegerquery & jaegerstorage]

flowchart TD
                    subgraph "Jaeger v1 (Legacy Custom Microservices)"
                        V1Agent["jaeger-agent<br/>(Zipkin/Thrift UDP :6831)"] --> V1Collector["jaeger-collector<br/>(Custom Channel Pipeline)"]
                        V1Collector --> V1Storage[("Custom Storage Plugins")]
                        V1Query["jaeger-query<br/>(Custom REST/gRPC :16686)"] --> V1Storage
                    end
                
                    subgraph "Jaeger v2 (OpenTelemetry Collector Architecture)"
                        OTLPSource["OTLP Ingestion<br/>(gRPC :4317 / HTTP :4318)"] --> Pipeline
                        subgraph Pipeline["OpenTelemetry Pipeline Engine"]
                            Receivers["Receivers<br/>(otlp, jaeger, zipkin)"] --> Processors["Processors<br/>(memory_limiter, batch, adaptive_sampling)"]
                            Processors --> Exporters["Exporters<br/>(jaeger_storage_exporter)"]
                        end
                        Exporters --> V2SPI["Storage SPI: internal/storage/v2/api/tracestore"]
                        V2SPI --> Backends[("Backends: ClickHouse, Elasticsearch, Memory")]
                        ExtQuery["Extension: jaegerquery<br/>(UI, REST, gRPC v3)"] --> V2SPI
                        ExtStorage["Extension: jaegerstorage<br/>(Shared Connection Pools)"] -.-> V2SPI
                    end

Key Architectural Advantages of Jaeger v2#

1. Native OTLP Ingestion: Replaces custom Thrift protocols with native OpenTelemetry Protocol (OTLP/gRPC on port 4317 and OTLP/HTTP on port 4318).
2. From Single Spans to Native Batches (ptrace.Traces): Jaeger v2 processes spans using OpenTelemetry Collector native columnar in-memory data structures (ptrace.Traces), unlocking massive bulk insert efficiencies on columnar engines like ClickHouse.
3. The Collector Extension Model:
   • jaegerstorage Extension: Acts as a centralized singleton providing shared, thread-safe database handles, connection pools, and health checks across ingestion and query paths.
   • jaegerquery Extension: Packages the Jaeger UI, HTTP query endpoints, and gRPC Query APIs as a native Collector extension (service.Extension).
4. Standardized Configuration (jaeger.yaml): Replaces dozens of fragmented CLI flags with structured, validated YAML schemas conforming to standard OpenTelemetry Collector syntax.

4. Production Jaeger v2 Configuration Walkthrough#

Below is a production-ready Jaeger v2 configuration file illustrating how receivers, processors, exporters, extensions, and pipelines fit together:

service:
                  extensions: [jaegerstorage, jaegerquery, health_check]
                  pipelines:
                    traces:
                      receivers: [otlp, jaeger]
                      processors: [memory_limiter, batch]
                      exporters: [jaeger_storage_exporter]
                  telemetry:
                    metrics:
                      address: 0.0.0.0:8888
                
                extensions:
                  health_check:
                    endpoint: 0.0.0.0:13133
                  jaegerstorage:
                    backends:
                      clickhouse:
                        endpoint: tcp://clickhouse:9000
                        database: jaeger
                  jaegerquery:
                    storage:
                      traces: clickhouse
                    ui:
                      config_file: /etc/jaeger/ui.json
                
                receivers:
                  otlp:
                    protocols:
                      grpc:
                        endpoint: 0.0.0.0:4317
                      http:
                        endpoint: 0.0.0.0:4318
                  jaeger:
                    protocols:
                      thrift_compact:
                        endpoint: 0.0.0.0:6831
                
                processors:
                  memory_limiter:
                    check_interval: 1s
                    limit_percentage: 80
                    spike_limit_percentage: 20
                  batch:
                    send_batch_size: 8192
                    timeout: 200ms
                    send_batch_max_size: 16384
                
                exporters:
                  jaeger_storage_exporter:
                    trace_storage: clickhouse

5. The Three-Tier Engineering Analysis#

Level 1: Beginner / Conceptual Mental Model#

• The International Shipping Container Analogy:
  • In early freight transport, every shipping company operated custom-built carts, crates, and railway wagons. Moving cargo from a ship to a train required unloading, repacking, and re-sorting every package by hand.
  • In modern global logistics, everyone uses the standardized ISO shipping container (OpenTelemetry).
  • Jaeger v1 was a custom truck fleet that only accepted proprietary crates.
  • Jaeger v2 is the high-throughput logistics hub: it accepts standardized containers from any carrier, verifies their seals, routes them through inspection scanners (processors), and stores them in high-density automated warehouses (ClickHouse/Elasticsearch).

Level 2: Intermediate Go Systems Engineer#

Let us examine how Jaeger v2 initializes itself as an OpenTelemetry Collector distribution inside cmd/jaeger/main.go:

package main
                
                import (
                	"log"
                
                	"go.opentelemetry.io/collector/component"
                	"go.opentelemetry.io/collector/otelcol"
                
                	"github.com/jaegertracing/jaeger/cmd/jaeger/internal/extension/jaegerstorage"
                	"github.com/jaegertracing/jaeger/internal/extension/jaegerquery"
                )
                
                func main() {
                	info := component.BuildInfo{
                		Command:     "jaeger",
                		Description: "Jaeger distributed tracing platform",
                		Version:     "2.x",
                	}
                
                	params := otelcol.CollectorSettings{
                		BuildInfo: info,
                		Factories: components, // Bundles all receivers, processors, exporters, extensions
                	}
                
                	if err := run(params); err != nil {
                		log.Fatalf("collector server run finished with error: %v", err)
                	}
                }

Key Implementation Insights:#

• Rather than invoking http.ListenAndServe or configuring raw gRPC servers manually, Jaeger v2 registers its components into otelcol.CollectorSettings.
• Lifecycle events (Start, Shutdown, health probing, signal handling, and configuration hot-reloading) are coordinated directly by the OpenTelemetry Collector runtime.
• Custom Jaeger capabilities (e.g. storage drivers and UI query endpoints) are implemented cleanly as Collector components adhering to component.Component and service.Extension contracts.

Level 3: Senior / Staff Systems Architect#

At enterprise scale (100,000+ spans/second per node), the architectural shift to v2 eliminates systemic failure modes:

1. Backpressure & Memory Bounds (OOM Prevention):
   • In Jaeger v1, downstream database latency spikes caused in-memory Go channel queues to fill rapidly, resulting in silent span loss or container OOMKilled crashes.
   • In v2, the memory_limiter processor constantly measures process memory. When thresholds are breached, it exerts backpressure by returning Unavailable / ResourceExhausted over gRPC, forcing callers to back off or buffer locally.
2. Shared Database Connection Pools:
   • In v1, running the collector and query service in the same container meant each process established independent connection pools, multiplying connection overhead against Elasticsearch or Cassandra.
   • In v2, the jaegerstorage Extension acts as a singleton database provider. The ingestion pipeline and the query service share the same underlying driver instances, connection pooling, prepared statements, and health checks.
3. Zero-Allocation Columnar Ingestion:
   • The OpenTelemetry Collector uses pdata/ptrace internal structures designed for zero-allocation mutation and sequential memory layouts.
   • Converting OTLP batches directly into ClickHouse block formats minimizes garbage collector (GC) pause times and heap churn.

6. Summary & Key Takeaways#

• Distributed Tracing solves the causal visibility gap in microservices using 128-bit TraceIDs and 64-bit SpanIDs propagated via standardized W3C traceparent headers.
• Spans encapsulate execution timing, attributes, structured events, semantic kinds, and causal links into a Directed Acyclic Graph (DAG).
• Jaeger v2 is not merely an incremental release; it is an OpenTelemetry Collector distribution that unifies ingestion, processing, and querying into a single, scalable, production-grade runtime.
• The transition replaces fragile Thrift/UDP collectors with OTLP standard receivers, batch pipelines, memory limiter guards, and extension-based database singletons.

Official References and Documentation#

• Official Repository: GitHub - jaegertracing/jaeger
• Official Documentation: Jaeger Tracing Documentation
• Architecture and OpenTelemetry Migration: Jaeger v2 Architecture Source (cmd/jaeger)
• OpenTelemetry Collector Framework: OpenTelemetry Collector Documentation
• W3C Trace Context Specification: W3C Recommendation for Distributed Tracing

Author#

• Author: Divyansh Rawat
• GitHub: @DsThakurRawat
• Series: The Jaeger Engineering Reference Series