Operations

Deployment recipes and configuration fragments distilled from the verified reference implementations. Fragments marked illustrative show documented component shapes rather than byte-exact copies of any org's internal config. Provenance lives in index.

Onboarding Recipes

Adobe-Style: Two Annotations

Prerequisites:

1. OpenTelemetry Operator installed in every cluster (Adobe runs it cluster-wide).
2. An Instrumentation custom resource created for the target language/runtime.
3. The service team's user-facing Helm chart deployed (Adobe's Tier 1 — chart name is theirs; the pattern generalizes to any packaging that carries the annotations below).

Opt-in on a Java workload by adding exactly two annotations to the deployment manifest:

# deployment.yaml - Java service opting into auto-instrumentation + sidecar
annotations:
  instrumentation.opentelemetry.io/inject-java: "true"
  sidecar.opentelemetry.io/inject: "true"

Both annotation names are verified against the official Operator automatic-instrumentation docs. Adapt the language segment for other runtimes (inject-nodejs, inject-python, …) per those docs. Expected behavior at Adobe: telemetry flows through the locked-down sidecar to a Deployment collector, whose config can change freely without touching the pod.

Mastodon-Style: One CR Per Namespace, GitOps Only

Declare one all-signals collector as an OpenTelemetryCollector custom resource and let the Operator own its lifecycle; commit the manifest so Argo CD deploys and promotes it.

apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
  name: mastodon-social   # matches the published production example's namespace object
spec:
  # mode defaults to deployment - Mastodon runs it as a plain single Deployment
  config: |
    processors:
      tail_sampling:
        decision_wait: 10s
        policies:
          - name: errors-policy
            type: status_code
            status_code:
              status_codes: [ERROR]
          - name: probabilistic-policy
            type: probabilistic
            probabilistic:
              sampling_percentage: 0.1
    exporters:
      otlp: {}
    service:
      pipelines:
        traces:
          receivers: [otlp]
          processors: [tail_sampling]
          exporters: [otlp]

Verified shape from their case study: apiVersion: opentelemetry.io/v1beta1, kind: OpenTelemetryCollector, with the tail-sampling processor holding a probabilistic sampling_percentage: 0.1 plus an ERROR status-code policy. Error traces are always kept (~"a few dozen" successful samples per minute survive); metrics/logs pass through unsampled. Reported operational posture: no strict CPU/memory limits on the collector — "if it ever does have any issue, it just restarts automatically." Remaining fragment values are illustrative until copied from their page's embedded manifest.

Skyscanner-Style: Endpoint + Opinionated Library

Two moving parts only:

# From Skyscanner's sample base image - the entire vendor contract is this env var
ENV OTEL_EXPORTER_OTLP_ENDPOINT="http://otel.skyscanner.net"

• Service teams inherit the base Docker image containing the pre-configured OTel Java agent carrying org-wide defaults (Python/Node.js services use wrapper libraries instead of the base image).
• Migrating legacy OpenTracing services = bumping the core-library version — SDK swap is transparent behind the shared API design; instrumentation code does not change.
• Legacy open-source/platform services that cannot emit OTLP natively get scraped by the Agent DaemonSet tier rather than blocked.

Configuration Recipes

Signal Isolation Per Backend Risk (Adobe Tier 2)

Run one collector Deployment per signal in the platform-managed namespace so that one backend rate-limiting or rejecting data for one signal never blocks the others:

metrics-deployment ----\
logs-deployment --------+--> routing connector --> per-team exporter
traces-deployment -----/

Team-Controlled Backend Routing Via Header (Adobe)

Service teams choose backends through Helm values without platform intervention:

# values.yaml of the team-facing chart (illustrative names follow the described mechanism)
telemetry:
  backend: newrelic          # sets an HTTP header on OTLP exports

The managed-namespace collectors feed that header into the routing connector, which selects the matching exporter (multiple backends supported). Adobe originally used the routing processor for header-based routing and migrated to the connector upon deprecation.

Span Metrics Instead Of Native Istio Metrics (Skyscanner)

Skyscanner dropped reliance on Istio native metrics after cardinality explosions overwhelmed Prometheus; derive lower-cardinality span metrics at the collector using the span-metrics connector before export. Generalizes to: prefer collector-side derivation over raw mesh-metric ingestion when series counts explode.

Upgrade Gotchas

• Routing processor → routing connector: if you adopted header-based backend routing early, plan the migration to the connector — the processor was deprecated (the exact path Adobe took).
• Operator upgrades can break instrumentation compatibility: called out candidly in Adobe's write-up despite the otherwise smooth two-annotation story; pin and test Operator versions against your Instrumentation CRs.
• DaemonSet sizing on heterogeneous nodes: the blueprint's efficiency argument (nodes serving anywhere from 4 to 40 pods force over-provisioned DaemonSets) is a reason to re-evaluate agent tiers as clusters grow diverse in node size — not a rule to rip them out (see Skyscanner's narrow-scrape DaemonSet).
• Secrets placement audit: search application configs for backend endpoints/API keys; the blueprint's Action 2 expects them exclusively at the gateway tier. Direct app→internet egress for telemetry is the anti-pattern being eliminated.

Related

• Architecture — the topologies these recipes deploy
• Topic index — program taxonomy, verification status, full source list