Skip to content

Architecture

Verified production patterns from the official OpenTelemetry guidance corpus: one normative blueprint and three reference implementations, all adversarially fact-checked against live pages on 2026-08-27. Sources are cited inline; full link table lives in index.

Managed Telemetry Platforms for Kubernetes Workloads

The blueprint prescribes a target architecture for platform-engineering organizations running heterogeneous, multi-tenant Kubernetes:

flowchart LR
    subgraph Workloads["Application workloads"]
        P1["Pod A<br/>Operator-injected auto-instrumentation"]
        P2["Pod B<br/>zero-code base image"]
        P3["Pod C<br/>shared language library"]
    end
    subgraph GatewayTier["Central Collector Gateway tier"]
        G1["Gateway 1<br/>horizontally scaled"]
        G2["Gateway 2"]
        S[("Backend endpoints<br/>and API keys<br/>live ONLY here")]
    end
    B1["Observability backend"]
    B2["SaaS backend"]
    P1 -->|"OTLP"| G1
    P2 -->|"OTLP gRPC"| G2
    P3 -->|"OTLP HTTP/protobuf"| G1
    G1 --> B1
    G2 --> B2
    S -.->|config| G1
    S -.->|config| G2

The five verified prescriptions:

  1. Automatic centralized ingestion. "Telemetry in this type of Kubernetes environment is automatically ingested into a centralized layer deployed as an OpenTelemetry Collector Gateway."
  2. OTLP as the sole export protocol. Base configuration ensures workloads export via OTLP — HTTP/protobuf default or gRPC "configured to export to the most optimal Collector".
  3. Secrets never at the application layer. Implementation Action 2 verbatim: "Backend/SaaS endpoints or API keys should not be included in application-level configuration, as we recommend handling these at a Collector Gateway." Direct egress from hundreds of applications to the internet is framed as an anti-pattern and single point of failure.
  4. Operator-first instrumentation. Prioritize the OpenTelemetry Operator: install it, create Instrumentation custom resources, annotate pods or entire namespaces. Fallbacks when the Operator is unavailable/incompatible: zero-code base container images bundling agents (supported languages) or shared language-specific libraries (unsupported ones).
  5. Deploy gateways via Operator or Helm charts (Implementation Action 3), kept centrally managed.

Compute-Efficiency Argument

The blueprint argues centrally managed Gateways use compute more efficiently than per-node DaemonSets or per-pod Sidecars in heterogeneous multi-tenant environments: a DaemonSet must be over-provisioned for variable node sizes ("a node may serve 4 or 40 pods") and fluctuating per-pod volume, while a gateway tier scales independently against total telemetry volume. Verifiers flagged scope explicitly — this is an efficiency argument within the blueprint's context, not a prohibition of agent/sidecar patterns elsewhere. Skyscanner's DaemonSet below is a live counterexample doing deliberately narrow work.

Declared out of scope by the blueprint itself: audit logging and business reporting uses of telemetry.

Adobe

flowchart TB
    subgraph ServiceNS["Service team namespace"]
        subgraph Pod["Application pod"]
            APP["App container"] --- SC["Sidecar Collector<br/>IMMUTABLE config"]
        end
        DC["Deployment Collector<br/>per service"]
    end
    subgraph ManagedNS["Managed namespace - observability team"]
        M_METRICS["Collector Deployment<br/>metrics only"]
        M_LOGS["Collector Deployment<br/>logs only"]
        M_TRACES["Collector Deployment<br/>traces only"]
        RC["routing connector<br/>keyed on OTLP HTTP header"]
    end
    BE1["Backend chosen by team A"]
    BE2["Backend chosen by team B"]

    SC -->|"OTLP"| DC
    DC --> M_METRICS & M_LOGS & M_TRACES
    M_METRICS --> RC
    M_LOGS --> RC
    M_TRACES --> RC
    RC --> BE1
    RC --> BE2

Three tiers, designed around one constraint: configuration changes must never restart application pods.

  • Tier 1 — a user-facing Helm chart creates two collectors per service: a sidecar whose config is "intentionally locked down… immutable to prevent application restarts caused by configuration changes", plus a standalone Deployment collector receiving sidecar telemetry over OTLP. "When configuration changes, only the deployment collector restarts. The application pod and its sidecar remain untouched."
  • Tier 2 — the observability-team namespace runs a separate collector Deployment per signal: metrics, logs, traces. The isolation rationale is verbatim: "If a backend becomes rate-limited or starts rejecting data for one signal type, the others continue flowing uninterrupted."
  • Tier 3 — the backends themselves (destinations, not collectors). Multiple are supported.

Backend selection is delegated to teams: Helm values set an HTTP header on OTLP exports; managed-namespace collectors feed that header to the routing connector, which steers each stream to the right exporter. Adobe originally used the routing processor and migrated to the routing connector after deprecation — see Operations.

Onboarding is two annotations. The Operator runs in every cluster; a Java service opts in with exactly:

annotations:
  instrumentation.opentelemetry.io/inject-java: "true"
  sidecar.opentelemetry.io/inject: "true"

Both names cross-checked against official Operator docs. Adoption is voluntary — "People add two lines in their deployment. And it just works." The write-up candidly notes Operator upgrades have broken instrumentation compatibility, so it is not marketing copy.

Skyscanner

flowchart TB
    SVC["Services across 24 clusters<br/>1000+ microservices"] -->|"OTEL_EXPORTER_OTLP_ENDPOINT=otel.skyscanner.net"| DNS["Single DNS entry point"]
    DNS --> ISTIO["Istio nearest-available routing"]
    ISTIO --> GW["Gateway Collectors<br/>ReplicaSet - bulk processing<br/>traces + metrics"]
    ISTIO --> AG["Agent Collectors<br/>DaemonSet - minimal processing<br/>Prometheus scrape of non-OTLP services"]
    ISTIO_2["Istio proxies"] -->|"Zipkin spans"| GW
    GW -->|"OTLP"| VENDOR["Commercial observability vendor"]
    AG -->|"scrapes then exports"| VENDOR
  • Two pillars, chosen after weeks of PoCs/reviews/prototypes replacing a fragmented RUM/tracing/synthetics vendor zoo plus internal OpenTSDB, Prometheus, and multiple ELK stacks: OpenTelemetry as the single instrument-and-transport standard, and one commercial backend (named as New Relic in Skyscanner's own engineering post; anonymized on opentelemetry.io). New Relic SDKs lingered temporarily where "OpenTelemetry was not yet ready at the time, like mobile or browser", with full OTel-nativeness the stated North Star.
  • Vendor-agnostic ingress: globally, services send telemetry to one central DNS endpoint (otel.skyscanner.net); "Istio handles routing requests to the nearest available collector." Backend swaps change simple config, not topology.
  • Division of labor: Gateway ReplicaSets handle bulk OTLP traffic with most processing; Agent DaemonSets do minimal processing and exist chiefly to scrape Prometheus endpoints from open-source/platform services not yet OTLP-native — so legacy scrape-path traffic bypasses direct OTLP push while everything else flows through the single endpoint.
  • Cardinality lesson: Istio native metrics suffered "cardinality explosion issues that would overwhelm their Prometheus deployment"; span metrics are now derived via the span-metrics connector at lower cardinality.
  • Operational leverage: six Hubble platform engineers manage most collectors for 1,400 employees running 1,000+ microservices across 24 clusters. Java services (the dominant estate) inherit a base Docker image containing the pre-configured OTel Java agent with org-wide defaults; Python/Node.js use wrapper libraries.
  • Migration proof: 300+ microservices moved from OpenTracing to OTel "in a matter of weeks" with zero instrumentation-code changes — owners bumped one opinionated core-library version exporting to the central gateway. (Self-reported figure.) Cluster math reconciles with their staged rollout: 3 Alpha + 8 Beta + 13 remaining = 24.

Mastodon

flowchart TB
    subgraph NS["One Kubernetes namespace"]
        COL["Single Collector Deployment<br/>traces + metrics + logs<br/>tail sampling inside"]
        APPS["Applications"] -->|"OTLP"| COL
    end
    OP["OpenTelemetry Operator<br/>reconcile - restart - lifecycle"]
    GIT["Git repo"] --> ARGO["Argo CD"] -->|"deploys/promotes"| CR["OpenTelemetryCollector CR v1beta1"]
    CR -.-> OP
    OP -.-> COL
    COL --> BACKEND["Observability backends"]

The deliberately minimal counterpoint: one all-signals Collector per Kubernetes namespace, run as a plain Deployment.

  • Declared as opentelemetry.io/v1beta1 OpenTelemetryCollector custom resources; the Operator handles reconciliation, automatic restart, lifecycle. "There are no separate gateway and agent tiers, no complex routing layers, and no custom deployment tooling." Argo CD provides GitOps promotion with Git-history auditability. (Verifier note: the word "sidecar" never appears in the source — tier exclusion is implicit in the single-Deployment topology.)
  • Scale headline: mastodon.social serves up to 300,000 daily actives and ~10M requests/min on 9–15 autoscaling nodes (16 cores / 64 GB each) running ~70–80 pods — and this design "has proven more than sufficient". Mastodon.online shares the setup; the whole org is ~20 people.
  • Tail sampling keeps ~0.1% of successful traces ("a few dozen per minute") but all error traces; metrics/logs flow unsampled through the same Collector, which runs without strict CPU/memory limits. Tim Campbell: "if it ever does have any issue, it just restarts automatically."

Cross-Case Comparison

Dimension Adobe Mastodon Skyscanner
Tiering Three tiers One collector per namespace Two tiers (Gateway RS + Agent DS)
Deploy vehicle User-facing Helm chart + managed namespace Operator CR + Argo CD GitOps Platform-managed behind single DNS/Istio
Restart blast radius Only Deployment collector restarts; pods untouched Operator auto-restarts collector only Infrastructure decoupled from app deploys
Secrets location Managed namespace Per-namespace CR config Gateway tier
Onboarding abstraction Two annotations Apps export OTLP to namespace-local collector Base Docker image / wrapper library bump
Signal isolation Explicit — one Deployment per signal None needed at its scale Partial — scrape path separate
Team model Voluntary self-service platform ~20-person org runs it globally 6 engineers manage most of 24 clusters
Scale reported Enterprise-wide 300k DAU, 10M req/min 1,000+ microservices

Unifying Themes

Across all four pages, verified commonalities form an emergent doctrine:

  1. Small platform teams operate centralized collection — leverage comes from abstraction (annotations, base images, CRs), not headcount.
  2. OTLP is the sole transport everywhere.
  3. Signal isolation buys resilience where rate-limits/backpressure are a real risk (Adobe explicit; Skyscanner partial).
  4. Developer-facing API stays tiny — two annotation lines, a library bump, or nothing at all.
  5. Restart isolation and immutability are first-class design goals, whether achieved via locked sidecars, Operator reconciliation, or GitOps-only mutation.

Verification Notes

Epistemic Status

  • 24 of 25 candidate claims survived unanimous 3-vote adversarial verification (16 synthesized findings, zero refuted, no split votes). One claim — the exact contents/count of the blueprints index page — could not complete verification due to voter-agent infrastructure errors, so index avoids asserting an exact blueprint count.
  • All operational figures are company-authored case-study numbers (self-reported, not audited): scale stats, Skyscanner's "weeks-long" migration, Mastodon's 0.1% sampling outcome.
  • Reference implementations are officially "snapshots in time" and "not actively maintained"; figures reflect 2026-08-27 snapshots of pages last modified April–July 2026.
  • Vendor naming asymmetry: New Relic is named only in Skyscanner's own Medium post; opentelemetry.io anonymizes it as a "commercial observability vendor."