Architecture

How the Collector engine actually works, verified against live docs and source code on 2026-08-27. Confidence labels reflect adversarial vote outcomes; medium = 2-1 split that survived verification.

Pipeline Engine Internals

flowchart LR
    R1["Receiver A"] --> P1["Processor 1"]
    R2["Receiver B"] --> P1
    P1 -->|"may drop data<br/>sampling / filtering"| P2["Processor N"]
    P2 --> FO["fanoutconsumer"]
    FO -->|"copy"| E1["Exporter 1"]
    FO -->|"copy"| E2["Exporter 2"]

Data flow is push-based and sequential within a pipeline:

1. All receivers feed the first processor.
2. Each processor pushes onward and may drop data (this is how sampling and filtering work).
3. The last processor feeds the built-in fanoutconsumer, so "each exporter gets a copy of each data element." Confirmed in code via service/internal/fanoutconsumer; the fanout node exists even when only one exporter is configured.
4. Whether a copy happens depends on component MutatesData capabilities — mutation-sensitive consumers receive defensive copies.

pdata: The Internal Data Model

pdata ("pipeline data") is the canonical in-memory model for everything the Collector touches:

• All received data is converted into pdata; it traverses the entire pipeline in that format; exporters convert out only when sending. Consumer interfaces (ConsumeTraces / ConsumeMetrics / ConsumeLogs) are typed on pdata signals — there is no raw-bytes passthrough component type in signal pipelines.
• Implementation detail (medium confidence, code-confirmed): pdata wraps OTLP protobuf structs as underlying storage through a private orig pointer, which makes translation to/from the OTLP wire protocol efficient. The representation is deliberately unexported "so that we are free to make changes to it in the future." Since mid-2025 those structs are pdatagen-generated mirrors under pdata/internal, not literal protoc output.

Design Consequence

Because pdata's storage is private, components manipulate telemetry through the accessor API rather than raw protobuf access — upstream can reshape internals without breaking the component ecosystem.

Signal Enforcement And Config Activation

Pipelines operate on exactly three telemetry data types — traces, metrics, logs (medium confidence; profiles exists as an experimental fourth behind feature gates). If any referenced receiver/processor/exporter lacks support for its pipeline's type, the Collector fails at configuration load time with pipeline.ErrSignalNotSupported (pipeline/signal.go). Fail-fast at startup beats silent drop mid-flight — treat config-load errors as the contract tests of your pipeline wiring. Nuance: contrib's receivercreator handles this error gracefully for dynamic receivers — outside static-config scope.

Activation semantics matter operationally:

receivers:
  otlp: {}          # configured...
exporters:
  debug: {}
service:
  pipelines:
    traces:
      receivers: [otlp]      # ...but NOT enabled until referenced here
      exporters: [debug]

• Configuring any component does nothing until it appears in the service section — extensions directly; receivers/processors/exporters/connectors via pipelines.
• Connectors must appear on both ends of their joined pipelines; authenticator extensions must additionally be referenced from auth configuration.
• Pipelines require at least one receiver and one exporter; processors are optional (though recommended ones exist) — enforced by startup validation (errMissingServicePipelineReceivers/exporter rules).

Self-observability lives in the nested service.telemetry section with two documented subsections, logs and metrics, plus an experimental traces option. Note the post-v0.123.0 shift to metrics readers/views replacing the old metrics address binding:

service:
  telemetry:
    metrics:
      readers:
        - pull:
            exporter:
              prometheus:
                host: localhost
                port: 8888   # shape reflects documented post-v0.123.0 readers config

Startup-validation edge cases worth knowing (all code-confirmed):

• Connector architectures still satisfy the required exporter slot of a pipeline — the rule counts connectors where they apply.
• AllowNoPipelines concerns running with zero pipelines entirely; it does not waive the receiver/exporter requirements of pipelines you do declare.
• Components that load but are never wired remain inert — silent misconfiguration surfaces as "no data", which is why this got its own callout above.

Deployment Modes

The docs prescribe two modes, jointly defining the canonical topology:

Mode	Shape	Role
Agent	daemon, sidecar, or DaemonSet; VM binary or container	Deployed independently of SDKs; can aggregate raw measurements for languages lacking in-process stats
Gateway	centrally-run instance(s), per-cluster/per-region	Receives from agents/libraries over supported protocols, processes centrally, forwards to configured exporters

Default Kubernetes install deploys both simultaneously: agent tier as a DaemonSet plus one gateway Deployment. Agents collect application + host telemetry and ship to gateways over OTLP/gRPC port 4317 across the internal cluster network; gateways do centralized processing (filtering, sampling) and secure TLS egress to backends.

flowchart TB
    subgraph Nodes["Every node"]
        PODS["App pods"] --- AGENT["Agent collector<br/>DaemonSet"]
    end
    AGENT -->|"OTLP/gRPC 4317<br/>internal cluster network"| GW["Gateway Deployment<br/>filtering - sampling - TLS egress"]
    GW --> B1["Backend"]
    GW --> B2["SaaS"]

Aspirational Sentence In The Docs

The architecture page contains legacy design language about agents eventually pushing configuration "(such as sampling probability)" down to libraries. Verifiers flagged this as aspirational, not implemented. Today's real control-plane analog is opAMP/the Supervisor managing collector config — not injecting settings into SDKs. Do not build plans around that sentence. (Confidence in this correction itself rests on two failed refutation attempts.)

Tail Sampling Placement

Hard rule from the docs caution block: "The tail-sampling processor can make accurate decisions only if all spans for a trace arrive at the same Collector instance." Hence:

• Tail sampling runs gateway-side, never distributed across agents.
• Agents feed it through the loadbalancingexporter with routing_key: traceID, which makes trace-ID affinity sticky per trace rather than per request batch.
• Whether a copy happens depends on component MutatesData capabilities (see fanout semantics) — sampling decisions downstream stay consistent only when all spans of one trace share an instance.
• Official guidance: prefer a single well-resourced tail-sampling gateway unless you have a robust sticky-routing strategy — multi-instance setups hit routing re-splitting and decision-cache consistency caveats. Processor README is blunter: all spans of a trace MUST land on one instance.
• Scope nuance: "gateways only" applies within multi-instance topologies; a lone single-agent deployment technically hosts it fine.

Multi-instance routing mechanics (verified second pass). The supported pattern hashes consistently to pick a downstream replica — trace ID by default, service.name when feeding span-to-metrics pipelines. It is only eventually consistent: loadbalancingexporter resolvers (static list, DNS A records/headless services, plus k8s and aws_cloud_map variants; DNS default refresh interval 5s) refresh independently, so during scale-up/down replicas briefly disagree about the backend set — maintainers advise lowering the resolver interval in highly elastic environments, since roughly R/N routes get rerouted on every backend-list change. Practitioner reports document harsher-than-documented behavior including data loss on DNS record change (contrib issue #35378).

Resiliency Internals

What actually happens between exporter and backend, per docs cross-checked against code:

Mechanism	Default behavior	Knob
Sending queue capacity	Drop-on-full at queue_size=1000 in units of sizer (requests = batches, most performant; bytes least performant)	block_on_overflow: true opts into blocking until space/timeout instead of dropping
Queue drain	10 consumers (num_consumers)	per-exporter config
Retry policy	Enabled; 5s initial ×1.5 jittered backoff capped at 30s; give up per batch after 300s, dropping oldest queued data (official data-loss circumstance #1)	max_elapsed_time: 0 retries indefinitely through outages

Rejected-before-enqueue data never reaches retry logic — observability lives in otelcol_exporter_enqueue_failed_{spans,metric_points,log_records}, otelcol_exporter_queue_size vs _capacity, throttling logs ("Dropping data because sending_queue is full"), and upstream otelcol_receiver_refused_* movement.

Persistent queues. Pointing sending_queue.storage at a storage extension (file_storage being "a popular and safe choice") removes the in-memory queue entirely: the queue becomes a disk write-ahead log written before each export attempt, and after kill/crash exports resume from where they stopped. Behavioral consequences worth knowing before enabling:

• Delivery becomes at-least-once — duplicates can occur if the process dies between backend success and storage delete.
• Auth-extension context is not propagated through the persistent queue.
• The filestorage extension caps each bbolt database file via max_size (bytes, per component instance, 0 = unlimited); writes that would force growth past the cap fail with storage-full errors. This capability landed around June 2026 — recently released Collectors may lack it.
• Opt-in online "rebound" compaction exists precisely for outage-then-drain workloads: enabled via compaction.on_rebound, triggering only once allocated data first exceeded rebound_needed_threshold_mib (default 100 MiB) and later fell below rebound_trigger_threshold_mib (default 10 MiB), checked every 5s; thresholds must be ≤ max_size.

memory_limiter refuses incoming telemetry above the soft limit (limit_mib - spike_limit_mib) by returning a non-permanent error to the preceding component — which is expected to retry, propagating backpressure rather than silently dropping. Surface: otelcol_processor_refused_spans (and signal siblings). Real process RSS typically runs ~50 MiB above limit_mib. Percentage mode (limit_percentage, spike defaulting to 20% of it) is documented for Linux+cgroups container platforms; (medium confidence) when both fixed and percentage settings are present, fixed silently wins.

OpAMP Management Plane

flowchart LR
    SRV["OpAMP server"] <-->|"remote config - status - heartbeat"| SUP["Supervisor"]
    SUP <-->|"last known config fallback"| COL["Collector process"]
    GIT[("effective.yaml<br/>merged local+remote")] -.-> COL

Split verdict, fully verified:

• Spec status: OpAMP is formally Beta as of 2026-08-27 — breaking changes remain possible between releases, with no spec-level production-readiness guarantee (individual capability bits may be Stable subordinate to the Beta umbrella). Core functions: remote configuration delivery, agent status reporting, heartbeats (recommended default interval 30s).
• Remote configuration: production-viable but conservative in the contrib Supervisor — AcceptsRemoteConfig is off by default; applying remote config merges with optional local config and restarts the Collector process (opt-in use_hup_config_reload hot-reload exists since v0.130.0, non-Windows only). Receiving an empty config map stops the Collector until a non-empty one arrives. Outage fallback keeps the last persisted config running across supervisor restarts while reconnecting with exponential backoff.
• Executable/package upgrades: not production-ready upstream. accepts_packages parses in supervisor config but is disabled at runtime — attempting to enable it blocks startup with "capability is not yet fully implemented"; the design section is explicitly "for design review purposes"; tracked in open issue #47272.

Refuted Overclaim

Verifiers killed (0-3) any claim that the OpAMP spec guarantees secure auto-update including downgrades. Treat vendor-blog claims of guaranteed safe fleet self-updates against the Beta spec status and the blocked package-management implementation.

Auth And Security Posture

The core config/auth module defines exactly two directional authenticator categories, exposed via tls: and auth: blocks on confighttp/configgrpc:

Direction	Authenticates	Documented contrib extensions
Server	Incoming requests (typically receivers)	basicauth, bearertokenauth, oidcauth
Client	Outgoing requests (typically exporters)	asapauth, basicauth, bearertokenauth, oauth2client, sigv4auth

• basicauth/bearertokenauth are dual-role; oidcauth is server-only; oauth2client/sigv4auth/asapauth are client-only.
• headerssetter is correctly absent from both lists — it mutates headers rather than implementing either authenticator interface.
• Caveats: the documented lists are manually maintained (verify against current contrib state per release), and sigv4authextension carries a deprecation notice in contrib's state file.

This pairs directly with Skyscanner's and Mastodon's gateway placements in the guidance topic.

Mapping Engine Features To The Reference Implementations

The organizational topologies reduce to these engine-level primitives:

Reference implementation	Engine primitive they lean on
Adobe — immutable sidecar config	Config-load-time validation + sidecar mode; config changes routed to a Deployment collector instead
Adobe — backend choice via header	Routing through connectors keyed on OTLP transport metadata
Mastodon — one CR per namespace	Operator OpenTelemetryCollector CR in deployment mode, single-pipeline simplicity
Skyscanner — bulk-processing gateways	Gateway mode + fanout exporter copies; agents limited to scrape duties
All three — OTLP everywhere	pdata/OTLP affinity: OTLP is zero-translation into pdata's wrapped storage

That last row generalizes into a design heuristic: components that translate formats should sit at pipeline edges (receivers/exporters), keeping the interior in pdata so fanout copies stay cheap and signal support stays declaratively verifiable at config load.

Scaling Doctrine

Tiered, as documented:

• Agents: "typically don't require horizontal scaling because they run on each host" — scale vertically via resource limits. (Verifiers kept the word "typically".)
• Gateways: scale both vertically and horizontally.
• Load-balancer selection follows sampling needs: plain round-robin LB / K8s Service without tail sampling; trace-ID-aware routing with it.
• Automatic horizontal scaling = Kubernetes HPA on CPU or memory metrics — delivered natively by the Operator for CR-managed collectors (see the autoscaling recipe); constraints apply per workload kind.

Open Gaps

What survived two verification passes versus what remains genuinely open (2026-08-27):

Closed by primary-source + code verification: queue/retry defaults, drop-on-full semantics, persistent-queue behavior, filestorage caps + compaction, memory-limiter refusal mechanics, OpAMP Beta status and per-capability readiness, authenticator matrix, tail-sampling routing pattern.

Still open — do not fill from memory:

• Published, reproducible numeric benchmarks for tailsamplingprocessor (CPU per span, decision-cache growth policy and its interplay with memory limits, throughput ceilings).
• Batch processor tuning (timeout, send_batch_size, send_batch_max_size) specifically under overload/backpressure conditions.
• Secret-handling beyond the static authenticator matrix: confighttp/configgrpc TLS YAML specifics and any documented secret-provider mechanism (security config best practices is the unexplored starting point).
• Joint sizing guidance for queue_size × filestorage max_size × real disk budgets on persistent-queue deployments, given community complaints that the 1000-batch default under-utilizes disk.