Operations

Install, autoscaling, build, and release-management recipes from verified docs. Configuration fragments below carry only claim-verified fields. Anything unverified stays in Open Gaps instead of being invented.

Installation Paths

Three documented routes on Kubernetes, in the order the docs present them:

1. Pinned manifest — starting point. A single command installs the Collector as a DaemonSet (agent, kind DaemonSet) plus one gateway instance (Service + Deployment). The docs describe this as a starting point.
2. Helm charts — the named path for production installs.
3. OpenTelemetry Operator — "provision and maintain an OpenTelemetry Collector instance", with automatic upgrade handling, Service objects generated from the OTel configuration, and automatic sidecar injection into deployments.

Whatever the vehicle, remember the config-activation rule from Architecture: components count only when listed under service.

Operator Autoscaling Recipe

Collectors managed by the Operator get built-in HPA — you configure it on the CRD rather than creating your own HorizontalPodAutoscaler:

apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
  name: gateway
spec:
  mode: deployment            # REQUIRED for autoscaling
  autoscaler:
    minReplicas: 2
    maxReplicas: 8
    targetCPUUtilization: 90  # documented default
    targetMemoryUtilization: 80 # illustrative value

Verified constraints:

• Only works with mode: deployment or statefulset — "HPA only applies to StatefulSets and Deployments in Kubernetes". DaemonSets have no scalable replicas field, so agent collectors cannot be HPA-scaled this way (Operator issue #2605 tracks exactly this incompatibility). Sidecar-mode collectors are likewise excluded.
• KEDA's own scaling model matches the same Deployment/StatefulSet-only restriction.
• Default targetCPUUtilization is 90 per the Operator's webhook defaults.

Scaling Playbook

Distilled doctrine (all verified):

Tier	Scale how	Load balancing
Agent DaemonSet	Vertically — adjust resource limits	n/a (host-local)
Gateway without tail sampling	Vertical + horizontal	Any round-robin LB / K8s Service
Gateway with tail sampling	Vertical + horizontal, cautiously	loadbalancingexporter, routing_key: traceID. Prefer a single well-resourced instance

Cross-reference: the three reference implementations in the guidance topic instantiate these tiers differently at real scale.

Custom Builds With ocb

The OpenTelemetry Collector Builder (ocb) generates a complete custom Go binary mixing three component classes:

1. your own custom components,
2. upstream core/contrib components,
3. any other publicly available Go components.

Documented motivations: smaller binary footprint, or capabilities like authenticator extensions, receivers, processors, exporters, connectors that upstream does not ship. Builder version tracks the release train (docs currently reference v0.159.0).

Forking Caveat

Forked components that import the collector repos' internal/ packages can fail to compile against different versions — keep forks shallow or re-implement against public APIs.

Release Hygiene

• Cadence: strict ~14-day cycles ("all OpenTelemetry Collector repositories have very short 2 week release cycles"), measured 13–15 days across ten releases through v1.65.0/v0.159.0 (2026-08-17).
• Dual tagging: stable v1.x + module v0.x on the same release.
• No LTS/EOL policy — plan recurring upgrades into platform ownership duties. Use the Operator's automatic upgrade handling where installed. Security fixes aim for ≤30 days.
• Time-sensitive surfaces to re-check each upgrade: internal-telemetry metrics configuration (readers replacing address post-v0.123.0), experimental traces self-telemetry, feature-gated profiles signal, and the distribution list (count changed to five mid-2025).

Quick version check against the release train:

curl -s https://api.github.com/repos/open-telemetry/opentelemetry-collector/releases/latest \
  | jq -r '.name'   # e.g. "v1.65.0/v0.159.0"

Resiliency Recipes

All defaults below are documented and code-cross-checked (see Architecture for mechanics).

Outage-tolerant exporter with persistent queueing and indefinite retry:

extensions:
  file_storage:
    directory: /var/lib/otelcol/storage
    timeout: 1s
    compaction:
      on_rebound: true                    # default false; opt-in drain compaction
      # defaults: rebound_needed_threshold_mib 100, trigger 10 MiB, check_interval 5s

exporters:
  otlp/backend:
    endpoint: backend:4317                # credentials belong in the exporter's auth block -
    retry_on_failure:
      enabled: true                       # 5s x1.5 jittered, cap 30s per docs+code
      max_elapsed_time: 0                 # never give up while backend is down
    sending_queue:
      enabled: true
      storage: file_storage               # removes the in-memory queue; at-least-once delivery
      queue_size: 5000                    # 1000 batches is the default; community finds it disk-limiting
      block_on_overflow: false            # false = drop-on-full; true = block until space

service:
  extensions: [file_storage]

Memory-ceiling guard upstream of exporters:

processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 400        # process RSS typically runs ~50MiB above this
    spike_limit_mib: 80   # soft limit = limit_mib - spike_limit_mib; refusals above it

Operational gotchas, all verified:

• Watch otelcol_exporter_enqueue_failed_* and otelcol_processor_refused_spans — refused data relies on upstream components retrying correctly. A misbehaving receiver turns backpressure into loss.
• Auth-extension context does not survive the persistent queue — configure exporter-side auth explicitly.
• Crash windows make persistent queues at-least-once: assume downstream deduplication for exactly-once semantics.
• When configuring both fixed (limit_mib) and percentage memory limits, fixed silently wins.
• Tail-sampler sizing: budget RAM as arrival rate × decision_wait (Little's Law — 100k spans/s at a 2-minute wait buffers ~12M spans). Alerts belong on ..._sampling_trace_dropped_too_early, because default overflow silently evicts oldest traces rather than refusing input. Prefer explicit block_on_overflow: true (v0.134.0+) when loss is unacceptable.
• queue_size counts batches, not bytes: the operative ceiling is real disk — monitor actual file growth rather than trusting the item budget (an unresolved stall anomaly on a 10Gi PVC is documented as #30770).

Remaining Gaps

Unverified areas intentionally left open (full list with starting links in Architecture): numeric tail-sampling benchmarks, batch tuning under overload, secret-provider mechanisms beyond the authenticator matrix, joint queue/disk sizing guidance.

Provenance

Extracted from three deep-research verification passes on 2026-08-27 against live primary sources. Pass one: 25 confirmed claims, zero refuted. Pass two (gaps): 23 confirmed, 2 refuted overclaims (notably OpAMP spec-level secure auto-update guarantees). Pass three (numbers): official benchmark corpus parsed for verified absence (100 runs / 177 labels / no tail-sampling, batch-tuning, or queue-sizing scenarios), plus external measured deltas from Elastic Labs and contrib #42326 with confounds documented. Split votes are labeled medium confidence throughout.

Related

• Architecture — engine internals behind every recipe here
• Topic index — distributions table, release facts, source list