Skip to content

Architecture

1. Default Topology / Flow

flowchart TB
    subgraph K8s["Kubernetes Cluster"]
        subgraph Agents["Data Plane (DaemonSet + Deployment)"]
            NA["coroot-node-agent\n(eBPF DaemonSet)\nPer node"]
            CA["coroot-cluster-agent\n(Deployment)\nDatabase discovery"]
        end

        subgraph Control["Control Plane"]
            OP["Coroot Operator\n(Lifecycle management)"]
            CR["Coroot CR\n(Custom Resource)"]
        end

        subgraph Server["Coroot Server (StatefulSet)"]
            direction TB
            InspEng["Inspection Engine\n18+ auto-inspections"]
            AIRCA["AI RCA Engine\n(pattern detection)"]
            SvcMap["Service Map Builder\n(eBPF topology)"]
            SLOEng["SLO Engine\n(error budget tracking)"]
            APIGW["API / Web UI\n(port 8080)"]
        end

        subgraph Storage["Storage (StatefulSet / External)"]
            Prom["Prometheus / VM / Mimir\n(metrics)"]
            CH["ClickHouse\n(logs, traces, profiles)"]
        end
    end

    subgraph External["Optional"]
        OTEL["OTel SDK\n(app-level traces)"]
        LLM["LLM API\n(Enterprise AI RCA)"]
    end

    NA -->|"metrics, traces,\nlogs, profiles"| Server
    CA -->|"DB metrics\n(pg_stat, INFO)"| Server
    OTEL -->|OTLP| Server
    Server -->|"remote_write /\nPromQL"| Prom
    Server -->|"clickhouse-native"| CH
    LLM -.->|"API"| AIRCA
    OP -->|"reconcile"| CR
    CR -->|"manages"| Agents
    CR -->|"manages"| Server
    CR -->|"manages"| Storage

    style Server fill:#1565c0,color:#fff
    style Agents fill:#2e7d32,color:#fff
    style Storage fill:#e65100,color:#fff

Detailed component breakdown, deployment topologies, and data flow diagrams for Coroot.

Component Architecture

18 Built-In Inspections

Coroot runs 18 automated inspection categories continuously on every discovered service:

Category Inspections
SLOs Availability SLO, Latency SLO
Instances Pod restarts, unavailable replicas
CPU CPU throttling, CPU usage near limits
GPU GPU utilization, memory usage
Memory OOM kills, memory near limits
Storage Disk usage, I/O latency
Network Connection errors, DNS failures, TCP retransmits
Logs Error log rate spikes, warning patterns
Runtime JVM heap/GC, .NET GC, Python GIL contention
Databases Postgres, MySQL, MongoDB, Redis, Memcached health
Deployments Rollout tracking, canary detection

Deployment Topologies

Single-Cluster (Standard)

flowchart LR
    subgraph Cluster["K8s Cluster"]
        NA1["node-agent<br/>(node 1)"]
        NA2["node-agent<br/>(node 2)"]
        NAN["node-agent<br/>(node N)"]
        CA["cluster-agent"]
        CS["Coroot Server"]
        CH["ClickHouse<br/>(2 shards × 2 replicas)"]
        Prom["Prometheus / VM"]
    end

    NA1 --> CS
    NA2 --> CS
    NAN --> CS
    CA --> CS
    CS --> CH
    CS --> Prom

Multi-Cluster (Hub and Spoke)

flowchart TB
    subgraph Central["Central Cluster"]
        CS["Coroot Server\n(full install)"]
        CH["ClickHouse"]
        Prom["Prometheus / VM"]
    end

    subgraph Remote1["Remote Cluster 1"]
        NA_R1["node-agents"]
        CA_R1["cluster-agent"]
    end

    subgraph Remote2["Remote Cluster 2"]
        NA_R2["node-agents"]
        CA_R2["cluster-agent"]
    end

    NA_R1 -->|"agentsOnly=true"| CS
    CA_R1 --> CS
    NA_R2 -->|"agentsOnly=true"| CS
    CA_R2 --> CS
    CS --> CH
    CS --> Prom

    style Central fill:#1565c0,color:#fff

Sequence: Incident Detection → RCA

sequenceDiagram
    participant App as Application
    participant Kernel as Linux Kernel
    participant Agent as node-agent (eBPF)
    participant Server as Coroot Server
    participant Insp as Inspection Engine
    participant RCA as AI RCA
    participant Alert as Alert Channel

    Kernel->>Agent: eBPF events (TCP, DNS, disk)
    Agent->>Server: Metrics + traces + logs
    Server->>Insp: Run 18 inspection categories
    Insp->>Insp: SLO breach detected
    Insp->>RCA: Trigger root cause analysis
    RCA->>RCA: Walk dependency graph
    RCA->>RCA: Correlate metrics ↔ traces ↔ logs
    RCA->>RCA: Rank root causes
    RCA->>Alert: Send alert with RCA summary
    Note over Alert: Slack / PagerDuty / Webhook

Sources

Data Model

1. Default Topology / Flow

erDiagram
    Coroot_CORE ||--o{ CONFIG : requires
    Coroot_CORE ||--o{ STATE : writes
    CONFIG {
        string runtime_params
        string limits
    }
    STATE {
        string metric_id
        json payload
    }

How It Works

How Coroot uses eBPF for zero-instrumentation data collection, automated service discovery, and AI-powered root cause analysis.

Data Collection Pipeline

eBPF-Based Auto-Instrumentation

Coroot's core differentiator is kernel-level telemetry collection via eBPF (extended Berkeley Packet Filter). The coroot-node-agent runs as a DaemonSet on every Kubernetes node and attaches eBPF programs to kernel tracepoints and kprobes:

flowchart LR
    subgraph Kernel["Linux Kernel (4.16+)"]
        TP["Tracepoints"]
        KP["kprobes/kretprobes"]
        TC["Traffic Control (tc)"]
    end

    subgraph Agent["coroot-node-agent"]
        eBPF["eBPF Programs"]
        Perf["Perf Buffer"]
        Agg["Userspace Aggregation"]
    end

    TP --> eBPF
    KP --> eBPF
    TC --> eBPF
    eBPF --> Perf --> Agg
    Agg -->|OTLP / Prom RW| Server["Coroot Server"]

What eBPF Captures (Without Code Changes)

Signal Kernel Attachment Point Data Collected
Network metrics tcp_sendmsg, tcp_recvmsg, tcp_connect Latency, throughput, error rates per connection
HTTP/gRPC traces Socket read/write Request method, path, status code, duration
DNS UDP socket Resolution time, failures
Disk I/O blk_mq_start_request IOPS, latency, bandwidth per container
CPU profiling perf_event_open On-CPU flame graphs per process
Memory profiling Allocation tracepoints Heap allocation patterns
Container lifecycle cgroup events Start/stop times, resource limits
Log collection Container stdout/stderr Application log lines

Cluster Agent Discovery

The coroot-cluster-agent complements eBPF data by connecting directly to databases:

Database Discovery Method Metrics Collected
PostgreSQL SQL queries via pg_stat_* Active connections, query latency, replication lag
MySQL SHOW STATUS / information_schema Thread count, slow queries, buffer pool hit rate
Redis INFO command Memory usage, connected clients, hit rate
MongoDB serverStatus command Operations/sec, document counts, lock percentages

Service Map Generation

Coroot automatically builds a real-time service dependency graph by correlating eBPF network traces:

  1. Connection tracking: eBPF programs track every TCP connection (source IP:port ↔ dest IP:port)
  2. Container resolution: IP addresses are mapped to Kubernetes pods via the container runtime
  3. Service grouping: Pods are grouped by Deployment/StatefulSet/DaemonSet
  4. Protocol detection: L7 protocol (HTTP, gRPC, MySQL, PostgreSQL, Redis, Kafka, and more) is identified from payload patterns
  5. Dependency graph: Directed edges between services are weighted by request rate, latency, and error rate

AI-Powered Root Cause Analysis

When an SLO violation or anomaly is detected, Coroot's AI RCA engine automatically:

  1. Identifies the impacted service from SLO breach alerts
  2. Walks the dependency graph upstream and downstream
  3. Correlates signals across metrics, traces, logs, and profiles for each service in the path
  4. Ranks root causes using statistical anomaly detection (for example, sudden CPU spike, disk saturation, memory leak, new deployment)
  5. Generates remediation suggestions (for example, "Service X shows 95th percentile latency spike correlated with disk I/O saturation on node Y — consider increasing PVC size or migrating to SSD-backed storage class")

AI RCA Integration (Enterprise)

The Enterprise edition integrates with LLM APIs to provide natural-language explanations of incidents, parse log patterns for error classification, and suggest specific remediations based on historical incident patterns.

SLO Monitoring

Coroot provides built-in SLO tracking based on RED metrics (Rate, Error, Duration):

  • Automatically calculates availability and latency SLOs per service
  • Tracks error budgets in real-time
  • Fires alerts when burn rate exceeds thresholds
  • No manual SLO configuration required — automatically derived from eBPF data

Data Flow Summary

sequenceDiagram
    participant App as Application
    participant Kernel as Linux Kernel
    participant NA as coroot-node-agent
    participant CA as coroot-cluster-agent
    participant Server as Coroot Server
    participant Prom as Prometheus / VM
    participant CH as ClickHouse

    Kernel->>NA: eBPF events (TCP, DNS, disk)
    App->>Kernel: syscalls (normal operation)
    NA->>Server: Metrics (Prometheus format)
    NA->>Server: Traces (OTLP)
    NA->>Server: Logs (container stdout)
    NA->>Server: Profiles (pprof)
    CA->>Server: DB metrics (SQL/INFO)
    Server->>Prom: Store metrics
    Server->>CH: Store logs, traces, profiles
    Server->>Server: Build service map
    Server->>Server: Run inspections & AI RCA

Sources

Benchmarks

Performance overhead, resource consumption, and scale limits for Coroot's eBPF-based observability.

eBPF Agent Overhead

Test Conditions

Parameter Value
Workload Go HTTP server (baseline)
Load 10,000 requests per second (RPS)
Agent coroot-node-agent (eBPF)
Methodology Latency comparison: baseline (no agent) vs agent enabled

Results

Metric Without Agent With Agent Impact
Request latency Baseline Within margin of error Negligible
CPU consumption — ~200 millicores ~20% of 1 CPU core
Latency impact — — Within measurement error

Key finding: At 10,000 RPS, the latency difference with the coroot-node-agent enabled vs baseline falls within the margin of measurement error. The eBPF programs are verified by the kernel for finite complexity. This guarantees that they cannot disrupt kernel operations.

CPU Profiler Overhead

Component Overhead Notes
eBPF CPU profiler 1–3% Based on Grafana Pyroscope implementation
JVM with -XX:+PreserveFramePointer 1–3% Required for accurate JVM stack traces

Resource Consumption at Scale

Coroot Server

Cluster Size Recommended CPU Recommended RAM Notes
< 50 services 1 vCPU 2 GB Single-node sufficient
50–200 services 2 vCPU 4 GB Inspection engine overhead
200+ services 4+ vCPU 8+ GB Service map complexity

Node Agent

Per Node CPU RAM
Base overhead ~50 millicores ~50 MB
Under load (10K RPS) ~200 millicores ~100 MB
Heavy profiling ~300 millicores ~150 MB

Scale Limits

Dimension Practical Limit Bottleneck
Services per cluster 500+ Server CPU for service map
Nodes per cluster 200+ Node agent DaemonSet scaling
Multi-cluster 10+ clusters Network bandwidth to central server
Metrics cardinality Backend dependent Prometheus/VM limits apply
Trace throughput Backend dependent ClickHouse write capacity

Storage Backend Requirements

Backend Scenario Resources
Prometheus < 1M series 2 CPU, 8 GB RAM, 100 GB SSD
VictoriaMetrics 1–10M series 2 CPU, 4 GB RAM, 200 GB SSD
ClickHouse 100 GB/day logs + traces 4 CPU, 16 GB RAM, 500 GB SSD

Caveats

  • Benchmarks are from vendor-provided testing. Users with specialized workloads should conduct their own validation.
  • eBPF overhead can vary based on kernel version, workload characteristics, and enabled collection features.
  • If you disable span capture but keep eBPF metrics, you can further reduce resource consumption in extremely high-load environments.

Sources

Security

Security model for Coroot covering node agent eBPF permissions, ClickHouse access control, API authentication, and data privacy. See also: observability/coroot/index, observability/coroot/explanation, observability/coroot/how-to-guides.

Security Architecture Overview

Coroot consists of three components with distinct security requirements:

  1. Coroot Server — Central API server and UI, stores configuration in PostgreSQL/SQLite and telemetry in ClickHouse.
  2. Coroot Node Agent — DaemonSet deployed to every Kubernetes node. Uses eBPF to collect traces, profiles, and network metrics.
  3. ClickHouse — External time-series store for telemetry data (traces, logs, profiles).
flowchart TD
    subgraph "Kubernetes Cluster"
        subgraph "Each Node"
            Agent[Coroot Node Agent<br/>DaemonSet]
            Workloads[Application Pods]
        end
        Coroot[Coroot Server<br/>Deployment]
        Operator[Coroot Operator<br/>Optional]
    end

    subgraph "Data Stores"
        PG[(PostgreSQL / SQLite<br/>Configuration)]
        CH[(ClickHouse<br/>Traces / Logs / Profiles)]
        Prom[(Prometheus<br/>Metrics)]
    end

    subgraph "Users"
        Browser[Browser / API Client]
    end

    Agent -->|eBPF traces / profiles| CH
    Agent -->|Prometheus remote_write| Prom
    Coroot -->|Read queries| CH
    Coroot -->|Read queries| Prom
    Coroot -->|Config state| PG
    Operator -->|Manages| Coroot
    Operator -->|Manages| Agent
    Browser -->|HTTP / API key| Coroot

Node Agent Security

eBPF Kernel Capabilities

The Coroot node agent uses eBPF for zero-instrumentation distributed tracing and continuous profiling. eBPF requires elevated kernel privileges.

Required capabilities and permissions:

Requirement Details
Privileged mode Agent runs with --privileged flag or equivalent SecurityContext
Host PID namespace --pid host required to observe all process namespaces
Filesystem mounts /sys/kernel/debug (rw) for eBPF program attachment. /sys/fs/cgroup (ro) for cgroup discovery
Linux kernel 4.16+ minimum for eBPF system call tracing. 5.2+ recommended for full feature support

Privileged Container Requirement

The node agent currently requires a privileged container because eBPF system calls (bpf(), perf_event_open()) need CAP_SYS_ADMIN on kernels older than 5.8. On kernels 5.8+, CAP_BPF and CAP_PERFMON can replace CAP_SYS_ADMIN for BPF operations, but Coroot's agent has not fully transitioned to fine-grained capabilities.

Agent Resource Isolation

The agent supports resource limits to prevent it from consuming excessive node resources:

# Kubernetes Operator configuration
nodeAgent:
  resources:
    requests:
      cpu: 100m
      memory: 200Mi
    limits:
      cpu: 500m
      memory: 1Gi

Agent Configuration Options

# Coroot Operator: node agent configuration
nodeAgent:
  ebpfTracer:
    enabled: true
    sampling: "1.0"        # 1.0 = 100% sampling
  ebpfProfiler:
    enabled: true           # CPU profiling via eBPF
  logCollector:
    collectLogBasedMetrics: true
    collectLogEntries: true  # Store raw logs in ClickHouse
  trackPublicNetworks: ["0.0.0.0/0"]  # Network tracking scope

Disabling eBPF Profiling Per Process

Individual processes can opt out of eBPF profiling by setting an environment variable:

COROOT_EBPF_PROFILING=disabled

This is useful for security-sensitive workloads where even observability instrumentation is undesirable.

ClickHouse Access Control

Connection Security

Coroot connects to ClickHouse using configured credentials. We recommend TLS for the connection:

# Coroot configuration
global_clickhouse:
  address: "clickhouse.coroot.svc:9000"
  user: "coroot"
  password: "${CLICKHOUSE_PASSWORD}"
  database: "coroot"
  tls_enable: true
  tls_skip_verify: false

ClickHouse Hardening

Create a dedicated ClickHouse user for Coroot with the minimum required permissions: - SELECT and INSERT on the Coroot database. - CREATE TABLE for automatic schema provisioning. - No access to other databases.

Data Retention

TTL policies limit data retention at the ClickHouse table level. These are applied during table creation:

# Coroot configuration
traces:
  ttl: 7d
logs:
  ttl: 7d
profiles:
  ttl: 7d

TTL Immutability

TTL values are applied during table creation and do not currently affect existing tables. To change retention, you must manually alter the ClickHouse table TTL or recreate the tables.

ClickHouse Schema

Coroot uses dedicated ClickHouse tables for each telemetry signal: - Trace spans stored with DoubleDelta and ZSTD compression for efficient time-series storage. - Log entries stored with OTel-compatible schema (timestamp, body, severity_text, trace_id). - Profiling data stored with stack trace aggregation.

Coroot Server Authentication

Authentication Modes

Coroot supports three authentication modes:

# Coroot configuration
auth:
  anonymous_role: ""                    # Empty = authentication required
  bootstrap_admin_password: "changeme"  # Initial admin password
Mode Configuration Behavior
Anonymous anonymous_role: Admin (or Editor/Viewer) No login required. All users get the specified role
Password-based anonymous_role: "" + bootstrap_admin_password Users must log in. The admin can create additional users
SSO (Enterprise) OIDC/SAML integration Delegated to external identity provider

Project-Level API Keys

Projects are the primary isolation boundary in Coroot. Each project has dedicated API keys used by node agents:

projects:
  - name: production
    api_keys:
      - key: "uuid-or-random-string"
        description: "Production cluster agent"
  - name: staging
    api_keys:
      - key: "another-uuid"
        description: "Staging cluster agent"

API keys are scoped to a project. An agent authenticating with a production key cannot write data to the staging project.

Role-Based Access

Coroot defines three roles:

Role Capabilities
Admin Full access: manage projects, users, API keys, configuration
Editor View and modify dashboards, investigate incidents
Viewer Read-only access to dashboards and investigation results

Prometheus Integration Security

Coroot reads metrics from a Prometheus server. The connection supports authentication and TLS:

global_prometheus:
  url: "https://prometheus.monitoring.svc:9090"
  refresh_interval: 15s
  tls_skip_verify: false
  user: "coroot-reader"
  password: "${PROM_PASSWORD}"
  custom_headers:
    X-Custom-Header: value

For write access (remote_write), configure the remote_write_url:

global_prometheus:
  remote_write_url: "https://prometheus.monitoring.svc:9090/api/v1/write"

Data Privacy Considerations

Sensitive Data Exposure

eBPF-based tracing captures request payloads and response data at the network level. This can inadvertently capture sensitive information:

  • PII in log entries: Application logs collected by the log collector can contain user data.
  • Database queries: Traced SQL queries can include parameter values with sensitive data.
  • HTTP headers: Network traces capture request/response headers including cookies and auth tokens.

Mitigation Strategies

Risk Mitigation
PII in logs Set collectLogEntries: false. Keep only log-based metrics
Sensitive traces Reduce sampling rate. Use COROOT_EBPF_PROFILING=disabled on sensitive pods
Data retention Set short TTL values (default 7d) to limit data accumulation
Cross-project leakage Use separate projects with distinct API keys per environment

Kubernetes Security Context

The node agent requires privileged access, but the Coroot server must run with restricted security:

# Coroot server (restricted)
securityContext:
  runAsNonRoot: true
  runAsUser: 65532
  readOnlyRootFilesystem: true
  capabilities:
    drop: ["ALL"]

# Node agent (privileged - required for eBPF)
securityContext:
  privileged: true
  hostPID: true

Network Policies

Restrict network flows to least-privilege:

Direction Source Destination Port Purpose
Ingress Coroot server Node agent 8080 Agent API
Egress Node agent ClickHouse 9000 Trace/log/profile delivery
Egress Node agent Prometheus 9090 Remote write metrics
Ingress Browser Coroot server 8080 UI access

Hardening Checklist

Area Recommendation
ClickHouse TLS Enable tls_enable: true with valid certificates
ClickHouse auth Dedicated user with minimum permissions
Agent resources Set CPU/memory limits to prevent node starvation
Data retention Set TTLs per signal type (traces, logs, profiles)
Authentication Disable anonymous access in production
API key isolation Separate keys per project/environment
Network policies Restrict egress from agents to ClickHouse and Prometheus only
Privacy Disable log collection for workloads processing PII
Privileged pods Accept as required for eBPF. Audit the agent DaemonSet
Secrets Store ClickHouse and Prometheus passwords in Kubernetes secrets or Vault

Sources