Skip to content

How-to Guides

Task recipes for adopting wide events: instrument a service so it emits one enriched event per request, send it through OpenTelemetry, tail-sample it in the Collector, query it with SQL, stand up a local event store, and migrate from a three-pillar stack without a big-bang cutover. For why each step matters see Explanation; for look-up tables see Reference.

Version assumptions

Collector snippets target recent otelcol-contrib releases (v0.160.0 / v0.161.0, September 2026). Component type names moved to snake_case during 2026: the loadbalancing exporter is now load_balancing and the core otlp exporter is now otlp_grpc, with the old names kept as deprecated aliases (contrib CHANGELOG, core CHANGELOG). On older Collectors use the old names.

Instrument a Service With Wide Events

Goal: every request produces exactly one record that carries request, infrastructure, business and error context. Build the event during the request and emit it once at the end.

Middleware Approach (TypeScript / Hono)

This middleware for Hono creates the event, exposes it to handlers, and emits it in finally. Hono does not rethrow handler errors to middleware; it places them in c.error and produces an error response, so the middleware reads c.error after await next() (Hono Context docs).

// middleware/wideEvent.ts
import type { MiddlewareHandler } from 'hono';
import { logger } from '../logger'; // any JSON logger, e.g. pino

export function wideEventMiddleware(): MiddlewareHandler {
  return async (c, next) => {
    const startTime = Date.now();

    // Initialize the wide event with request and deployment context
    const event: Record<string, unknown> = {
      request_id: c.get('requestId'), // set by hono/request-id
      timestamp: new Date().toISOString(),
      method: c.req.method,
      path: c.req.path,
      service: process.env.SERVICE_NAME,
      version: process.env.SERVICE_VERSION,
      deployment_id: process.env.DEPLOYMENT_ID,
      region: process.env.REGION,
    };

    // Make the event accessible to handlers
    c.set('wideEvent', event);

    try {
      await next();
    } finally {
      event.status_code = c.res.status;
      if (c.error) {
        const err = c.error as Error & { code?: string; retriable?: boolean };
        event.outcome = 'error';
        event.error = {
          type: err.name,
          message: err.message,
          code: err.code,
          retriable: err.retriable ?? false,
        };
      } else {
        event.outcome = c.res.status >= 500 ? 'error' : 'success';
      }
      event.duration_ms = Date.now() - startTime;

      // Emit the wide event: ONE record per request
      logger.info(event);
    }
  };
}

Register it after the request-ID middleware:

import { Hono } from 'hono';
import { requestId } from 'hono/request-id';
import { wideEventMiddleware } from './middleware/wideEvent';

const app = new Hono();
app.use('*', requestId());
app.use('*', wideEventMiddleware());

Handler Enrichment

Handlers add business context as they learn it. Measure sub-operations inside the handler and attach their latency to the same event.

app.post('/api/checkout', async (c) => {
  const event = c.get('wideEvent');
  const user = c.get('user');

  // Who is affected
  event.user = {
    id: user.id,
    subscription: user.subscription,
    account_age_days: daysSince(user.createdAt),
    lifetime_value_cents: user.ltv,
  };

  // What they were doing
  const cart = await getCart(user.id);
  event.cart = {
    id: cart.id,
    item_count: cart.items.length,
    total_cents: cart.total,
    coupon_applied: cart.coupon?.code,
  };

  // Sub-operation latency
  const paymentStart = Date.now();
  const payment = await processPayment(cart, user);
  event.payment = {
    method: payment.method,
    provider: payment.provider,
    latency_ms: Date.now() - paymentStart,
    attempt: payment.attemptNumber,
  };

  if (payment.error) {
    event.error = {
      type: 'PaymentError',
      code: payment.error.code,
      stripe_decline_code: payment.error.declineCode,
    };
    return c.json({ error: payment.error.code }, 402);
  }

  return c.json({ orderId: payment.orderId });
});

daysSince, getCart and processPayment stand for your own application code.

Emit the Wide Event as an OpenTelemetry Span

If the service already runs the OpenTelemetry SDK, make the active server span the wide event instead of (or in addition to) a log line. The trace then links hops across services, and the same attributes are available to Collector sampling policies.

import { trace } from '@opentelemetry/api';

app.post('/api/checkout', async (c) => {
  const span = trace.getActiveSpan(); // server span from auto-instrumentation
  const user = c.get('user');

  span?.setAttributes({
    'app.user.id': user.id,
    'app.user.subscription': user.subscription,
    'app.feature_flags.new_checkout_flow': flags.newCheckoutFlow,
  });

  const cart = await getCart(user.id);
  span?.setAttributes({
    'app.cart.item_count': cart.items.length,
    'app.cart.total_cents': cart.total,
  });

  // ... payment, error attributes the same way
  return c.json({ ok: true });
});

Attribute conventions

  • Use flat, dotted keys under a project prefix such as app. so they never collide with OTel semantic-convention names (http.*, db.*, service.*).
  • Prefer primitive values. OTLP 1.9.0 allows maps and heterogeneous arrays on every signal, but OTel warns that many backends cannot index or aggregate them (OTel blog, 2025-11-05).
  • Record business context as span attributes, not span events: OTel is deprecating the Span Event API in favor of log-based events (OTel blog, 2026-03-17).

Tail-Sample Wide Events

Goal: keep every event that matters (errors, slow requests, important customers, rollouts) and a small, weighted baseline of healthy traffic.

Tail Sampling Implementation

For services that emit wide events as log lines (no trace pipeline), apply the rule in the emitter after the event is finalized. Record the sample rate on the event so the backend can reweight counts.

// Tail sampling decision: returns the sample rate (1 = keep all), or 0 to drop
function sampleRate(event: WideEvent): number {
  if (event.status_code >= 500 || event.error) return 1;          // always keep errors
  if (event.duration_ms > 2000) return 1;                          // always keep slow requests
  if (event.user?.subscription === 'enterprise') return 1;         // always keep key customers
  if (event.feature_flags?.new_checkout_flow) return 1;            // keep rollout traffic
  return Math.random() < 0.05 ? 20 : 0;                            // keep 1 in 20 of the rest
}

const rate = sampleRate(event);
if (rate > 0) {
  logger.info({ ...event, sample_rate: rate });
}

Tune the 2000 ms threshold to your own p99. The keep-rate rules are summarized in Reference.

OTel Collector Tail Sampling

For traced services, run the tail_sampling processor in a gateway tier. It is beta for traces, ships in the contrib and k8s distributions, and holds spans in memory for decision_wait (default 30s) before deciding (README). A trace is kept when any policy samples it (unless a drop policy matches).

# gateway collector (otelcol-contrib)
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  tail_sampling:
    decision_wait: 10s
    num_traces: 100000              # traces held in memory; default 50000
    expected_new_traces_per_sec: 1000
    policies:
      - name: errors
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: slow-traces
        type: latency
        latency: { threshold_ms: 2000 }
      - name: enterprise-customers
        type: string_attribute
        string_attribute: { key: app.user.subscription, values: [enterprise] }
      - name: checkout-rollout
        type: boolean_attribute
        boolean_attribute: { key: app.feature_flags.new_checkout_flow, value: true }
      - name: baseline
        type: probabilistic
        probabilistic: { sampling_percentage: 5 }

exporters:
  otlp_grpc:                        # `otlp` on Collectors before the 2026 rename
    endpoint: event-store.example.internal:4317

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [tail_sampling]
      exporters: [otlp_grpc]

With more than one gateway replica, every span of a trace must reach the same replica. Put an agent tier in front that routes by trace ID with the load-balancing exporter (README):

# agent collector (DaemonSet or sidecar)
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

exporters:
  load_balancing:                   # `loadbalancing` on older Collectors
    routing_key: traceID
    protocol:
      otlp:
        tls:
          insecure: true            # in-cluster only; use TLS across networks
    resolver:
      dns:
        hostname: otel-gateway-headless.observability.svc.cluster.local
        port: 4317

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [load_balancing]

Tail sampling pitfalls

  • A trace evicted from the num_traces buffer before decision_wait is dropped unsampled. Raise num_traces or lower decision_wait, and watch the processor's own metrics.
  • Late spans after a decision inherit it only while the decision is cached; configure decision_cache much larger than num_traces.
  • The processor reassembles spans into new batches, so place it after processors that need request context such as k8sattributes.
  • The probabilistic policy does not by itself record a sample rate that every backend understands. If counts must be reweighted, use Honeycomb Refinery or check how your backend reads OTel TraceState sampling thresholds.

Scaling and memory sizing for this tier are covered in OTel Collector tail sampling placement.

Adaptive Sampling Options

  • Honeycomb Refinery (Apache-2.0, v3.4.0 on 2026-09-03) is a tail-sampling proxy with dynamic, rules-based, throughput-based and deterministic samplers, and it records sample rates for Honeycomb to reweight (Refinery).
  • adaptive_tail_sampling processor (contrib, development stability, not yet in any distribution): first-match rules route traces to adaptive samplers whose rate is encoded as ot=th in W3C TraceState "for correct downstream metric weighting". It was added as dynamic_sampling in v0.156.0 and renamed without an alias in v0.160.0 (README). Treat it as experimental.

Query Wide Events

Queries Enabled by Wide Events

With wide events you run analytics on production traffic, not string searches. These examples use PostgreSQL-style SQL against a flat events table; adapt the dialect to your store.

-- Premium users hitting payment errors in the last hour with the new checkout flow
SELECT user_id, error_code, payment_attempt, duration_ms
FROM events
WHERE status_code >= 500
  AND user_subscription = 'premium'
  AND feature_flag_new_checkout_flow = true
  AND timestamp > NOW() - INTERVAL '1 hour'
ORDER BY user_lifetime_value_cents DESC;

-- Error rate by deployment and region
SELECT deployment_id, region,
       COUNT(*) FILTER (WHERE status_code >= 500) AS errors,
       COUNT(*) AS total,
       ROUND(100.0 * COUNT(*) FILTER (WHERE status_code >= 500) / COUNT(*), 2) AS error_pct
FROM events
WHERE timestamp > NOW() - INTERVAL '15 minutes'
GROUP BY deployment_id, region
ORDER BY error_pct DESC;

-- p99 latency by service version (canary vs stable)
SELECT version,
       PERCENTILE_CONT(0.99) WITHIN GROUP (ORDER BY duration_ms) AS p99_ms,
       COUNT(*) AS request_count
FROM events
WHERE service = 'checkout-service'
  AND timestamp > NOW() - INTERVAL '1 hour'
GROUP BY version;

Sampled data

If events were tail-sampled, multiply by the recorded sample rate: SUM(sample_rate) instead of COUNT(*), and weight percentiles accordingly. Otherwise error rates look far higher than they are.

Query Enriched Spans in ClickHouse

The OpenTelemetry ClickHouse exporter (beta for traces and logs) and ClickStack write spans to an otel_traces table with Duration in nanoseconds, StatusCode as a string ('Error', 'Ok', 'Unset') and attributes in the SpanAttributes map (exporter README). The same questions in ClickHouse SQL:

-- Error rate and p99 by version and plan tier over the last hour
SELECT
    ResourceAttributes['service.version']      AS version,
    SpanAttributes['app.user.subscription']    AS plan,
    count()                                    AS requests,
    countIf(StatusCode = 'Error')              AS errors,
    round(100 * errors / requests, 2)          AS error_pct,
    quantile(0.99)(Duration) / 1e6             AS p99_ms
FROM otel_traces
WHERE ServiceName = 'checkout-service'
  AND ParentSpanId = ''                    -- root spans: one per request
  AND Timestamp >= now() - INTERVAL 1 HOUR
GROUP BY version, plan
ORDER BY error_pct DESC;

-- Every failed checkout for one customer, newest first
SELECT Timestamp, TraceId, Duration / 1e6 AS ms, StatusMessage,
       SpanAttributes['app.cart.total_cents'] AS cart_total
FROM otel_traces
WHERE SpanAttributes['app.user.id'] = 'user_456'
  AND StatusCode = 'Error'
  AND Timestamp >= now() - INTERVAL 1 DAY
ORDER BY Timestamp DESC
LIMIT 50;

Filtering on ParentSpanId = '' keeps one root span per request; in a multi-service trace, filter on the service's entry span instead. Run DESCRIBE TABLE otel_traces to confirm the column set on your exporter version.

Run a Local Event Store

ClickStack (ClickHouse + HyperDX + Collector)

The all-in-one image bundles ClickHouse, the HyperDX UI and a preconfigured OpenTelemetry Collector for local testing (ClickStack README):

docker run -p 8080:8080 -p 4317:4317 -p 4318:4318 \
  docker.hyperdx.io/hyperdx/hyperdx-all-in-one

Open http://localhost:8080, then point an OTel SDK at localhost:4317 (gRPC) or localhost:4318 (HTTP). For production use the Docker Compose or Helm deployment from the ClickStack docs.

GreptimeDB Standalone

The GreptimeDB README's standalone command exposes HTTP (4000, including the dashboard), gRPC (4001), MySQL (4002) and PostgreSQL (4003) protocols (GreptimeDB README):

docker run -p 127.0.0.1:4000-4003:4000-4003 \
  -v "$(pwd)/greptimedb_data:/greptimedb_data" \
  --name greptime --rm \
  greptime/greptimedb:latest standalone start \
  --http-addr 0.0.0.0:4000 \
  --grpc-bind-addr 0.0.0.0:4001 \
  --mysql-addr 0.0.0.0:4002 \
  --postgres-addr 0.0.0.0:4003

Pin a version tag (for example v1.2.1) instead of latest outside experiments. Follow the GreptimeDB OpenTelemetry ingestion guide for the exact OTLP/HTTP paths and headers; OTLP traces land in opentelemetry_traces and logs in opentelemetry_logs.

Migrate From Three Pillars

Migration: 1.0 to 2.0

The transition is incremental, not big-bang:

  1. Start emitting wide events alongside existing logs and metrics (dual-write). Begin with one high-traffic, high-pain service.
  2. Enrich events with business context in handlers (user, cart, payment, feature flags), using an attribute allow-list agreed with privacy/security.
  3. Deploy a 2.0-capable backend (Honeycomb, ClickHouse/ClickStack, GreptimeDB or another columnar store) and route the new events there.
  4. Recreate key dashboards and SLOs from events. Where a backend supports it, replace Prometheus recording rules with materialized views or continuous aggregations over events.
  5. Keep existing Grafana dashboards working through PromQL compatibility or by keeping the old metrics pipeline for infrastructure metrics.
  6. Enable tail sampling once volume makes storing everything unaffordable, and verify that derived counts use sample rates.
  7. Retire redundant pipelines gradually: first per-request application logs that the wide event now covers, then request-level metrics. Keep host, runtime and queue metrics as metrics.

Backward compatibility is non-negotiable

Existing dashboards, alert rules and on-call runbooks must keep working during the move. A migration that breaks alerting to gain query flexibility is a net loss.

Troubleshooting

Symptom Likely cause Fix
Many log lines per request still appear Handlers log directly instead of enriching the event Replace logger.info calls inside handlers with fields on the wide event. Keep separate logs only for background work
Events lack user or business fields Auto-instrumentation only Add setAttributes calls (or event fields) in handlers where the context is known
Error rate looks 10-20x too high Tail-sampled data counted without weights Store sample_rate on each event and use SUM(sample_rate)
Traces arrive incomplete after tail sampling Spans of one trace split across gateway replicas, or buffer eviction Route by traceID with the load-balancing exporter; raise num_traces or lower decision_wait
Storage bill grows with every release Unbounded attribute names (for example IDs used as keys) Put IDs in attribute values, never in attribute names; review new keys in code review
Personal data appears in telemetry Business context copied wholesale from domain objects Use an allow-list of attributes; redact or hash in the Collector (for example with the transform or redaction processors)

Sources