Skip to content

Architecture

1. Default Topology / Flow

flowchart TB
    subgraph Sources["Data Sources"]
        OTEL_S["OTel Collector\n(OTLP gRPC/HTTP)"]
        PROM_S["Prometheus\n(remote_write)"]
        ES_S["ES Bulk API\nclients"]
        FB_S["FluentBit /\nVector"]
        KF_S["Kinesis Firehose\n/ GCP Pub/Sub"]
        RUM_S["RUM SDK\n(browser)"]
    end

    subgraph O2Cluster["OpenObserve Cluster"]
        direction TB
        subgraph Stateless["Stateless Compute"]
            Router["Router\n(request dispatch)"]
            Ingester["Ingester\n(WAL → Parquet)"]
            Querier["Querier\n(DataFusion engine)"]
            Compactor["Compactor\n(file merging)"]
            AlertMgr["AlertManager\n(alerts + reports)"]
        end

        subgraph Infra["Infrastructure"]
            WAL["WAL\n(local disk\nmemtable)"]
            Cache["Disk Cache\n(querier-side)"]
        end
    end

    subgraph Storage["Storage Layer"]
        S3["Object Storage\n(S3/GCS/Azure/MinIO)"]
        PQ["Apache Parquet\n(Zstd compressed)"]
        Meta["Metadata Store\n(PostgreSQL / SQLite)"]
    end

    Sources --> Router
    Router --> Ingester
    Ingester --> WAL
    WAL -->|"flush every\n5min or size"| PQ
    PQ --> S3
    Querier -->|"scan"| S3
    Querier --> Cache
    Compactor -->|"merge"| S3
    AlertMgr --> Querier

    style Stateless fill:#e65100,color:#fff
    style Storage fill:#1565c0,color:#fff

Component breakdown, deployment topologies, and storage architecture for OpenObserve.

System Architecture

Node Role Architecture

flowchart LR
    subgraph Roles["ZO_NODE_ROLE"]
        ALL["all\n(single node)"]
        R["router"]
        I["ingester"]
        Q["querier"]
        C["compactor"]
        A["alertmanager"]
    end

    subgraph Groups["ZO_NODE_ROLE_GROUP"]
        Default["default\n(user queries)"]
        Background["background\n(alerts, reports)"]
    end

    R --> I
    R --> Q
    R --> A
    Q -.- Default
    A -.- Background

Role Responsibility

Role State Scales CPU Profile Memory Profile
Router Stateless Horizontal Low Low
Ingester WAL on disk Horizontal Medium Medium (memtable)
Querier Cache on disk Horizontal High (DataFusion) High (scan buffers)
Compactor Stateless 1–2 nodes Medium Low
AlertManager Stateless 1–2 nodes Low Low

Storage Architecture

Data Path

sequenceDiagram
    participant Client as Client
    participant Ingester as Ingester
    participant WAL as Local WAL
    participant S3 as Object Storage
    participant Compactor as Compactor

    Client->>Ingester: JSON / OTLP / ES Bulk
    Ingester->>Ingester: Schema inference
    Ingester->>WAL: Write to memtable (Arrow batches)
    Note over WAL: Flush triggers:<br/>5 min elapsed OR<br/>file size threshold

    WAL->>S3: Write small Parquet file
    Note over S3: Small files (1-10 MB)

    loop Background compaction
        Compactor->>S3: Read small files
        Compactor->>Compactor: Sort, merge, re-partition
        Compactor->>S3: Write large Parquet file
        Compactor->>S3: Delete old small files
    end

    Note over S3: Large files (100+ MB)<br/>Sorted by time, partitioned by stream

Parquet File Structure

Layer Detail
Partitioning By organization → stream → date → time window
Compression Zstd (default), high compression ratio
Bloom filters Per-column, configurable for high-cardinality fields
Row groups Optimized for DataFusion predicate pushdown
Metadata Column statistics for partition pruning

Query Engine: DataFusion

flowchart LR
    SQL["SQL Query"] --> Parser["SQL Parser"]
    Parser --> LP["Logical Plan"]
    LP --> Opt["Optimizer\n(predicate pushdown,\nprojection pruning,\npartition pruning)"]
    Opt --> PP["Physical Plan"]
    PP --> Scan["Parquet Scanner\n(parallel, columnar)"]
    Scan --> S3_Q["Read from S3\n(only needed cols)"]
    S3_Q --> Exec["Vectorized Execution\n(Arrow batches)"]
    Exec --> Result["Query Result"]

    style Opt fill:#2e7d32,color:#fff

HA Deployment Topology

flowchart TB
    LB["Load Balancer"]

    subgraph Routers["Router Pool"]
        R1["Router 1"]
        R2["Router 2"]
    end

    subgraph Ingesters["Ingester Pool"]
        I1["Ingester 1\n(WAL /data1)"]
        I2["Ingester 2\n(WAL /data2)"]
        I3["Ingester 3\n(WAL /data3)"]
    end

    subgraph Queriers["Querier Pool"]
        Q1["Querier 1\n(cache /cache1)"]
        Q2["Querier 2\n(cache /cache2)"]
    end

    C1["Compactor"]
    A1["AlertManager"]

    S3_HA["S3 / MinIO\n(shared storage)"]
    PG["PostgreSQL\n(metadata)"]

    LB --> Routers
    R1 --> Ingesters
    R2 --> Ingesters
    R1 --> Queriers
    R2 --> Queriers
    Ingesters --> S3_HA
    Queriers --> S3_HA
    C1 --> S3_HA
    A1 --> Queriers

    Routers --> PG
    Ingesters --> PG
    Queriers --> PG

    style S3_HA fill:#1565c0,color:#fff

Sources

Data Model

1. Default Topology / Flow

erDiagram
    Openobserve_CORE ||--o{ CONFIG : requires
    Openobserve_CORE ||--o{ STATE : writes
    CONFIG {
        string runtime_params
        string limits
    }
    STATE {
        string metric_id
        json payload
    }

How It Works

How OpenObserve uses Rust, Apache Parquet, and object storage to deliver cost-efficient observability.

Architecture

Component Roles

flowchart TB
    subgraph Sources["Data Sources"]
        OTEL["OTel Collector"]
        PROM["Prometheus"]
        FB["FluentBit / Vector"]
        ES_API["ES Bulk API\nclients"]
        RUM_SDK["RUM SDK"]
        KF["Kinesis Firehose"]
    end

    subgraph O2["OpenObserve Cluster"]
        direction TB
        Router["Router\n(request dispatch)"]
        Ingester["Ingester\n(WAL → Parquet)"]
        Querier["Querier\n(DataFusion)"]
        Compactor["Compactor\n(file merging)"]
        Alert["AlertManager\n(alerts + reports)"]
    end

    subgraph Storage["Storage Layer"]
        WAL["WAL\n(local disk)"]
        S3["Object Storage\n(S3/GCS/Azure/MinIO)"]
        Meta["Metadata Store\n(PostgreSQL / SQLite)"]
    end

    Sources --> Router
    Router --> Ingester
    Ingester -->|batch| WAL
    WAL -->|flush| S3
    Querier --> S3
    Compactor --> S3
    Alert --> Querier

    style O2 fill:#e65100,color:#fff

Data Flow: Ingestion to Query

sequenceDiagram
    participant Client as Data Source
    participant Router as Router
    participant Ingester as Ingester
    participant WAL as Local WAL
    participant S3 as Object Storage
    participant Compactor as Compactor
    participant Querier as Querier

    Client->>Router: OTLP / ES Bulk / Prom RW
    Router->>Ingester: Route by stream
    Ingester->>Ingester: Schema inference
    Ingester->>WAL: Write to memtable
    WAL->>S3: Flush as Parquet (5min or size threshold)
    Note over S3: Small Parquet files
    Compactor->>S3: Merge small files
    Compactor->>S3: Write optimized Parquet
    Note over S3: Large, sorted Parquet files
    Querier->>S3: Read Parquet partitions
    Querier->>Querier: DataFusion vectorized execution
    Querier->>Client: Return results

Storage Architecture

Parquet on Object Storage

OpenObserve stores all data as Apache Parquet files on object storage:

Aspect Detail
Format Apache Parquet (columnar)
Compression Zstd (default)
Partitioning By stream, date, and time window
Object storage S3, GCS, Azure Blob, MinIO
Local mode Disk-backed (for dev/single-node)
Bloom filters Per-column for high-cardinality field acceleration

Why 140x Cheaper Than Elasticsearch

Factor OpenObserve Elasticsearch
Compression Parquet columnar + Zstd (~10:1) Lucene segments (~1.5:1)
Storage tier S3 ($0.023/GB/mo) SSD ($0.10+/GB/mo)
Compute Stateless, scale to zero Always-on data nodes
Replicas S3 provides 11-nines Manually replicated shards
Net effect ~$0.002/GB/mo storage ~$0.28/GB/mo storage

Query Engine: Apache Arrow DataFusion

OpenObserve uses DataFusion (Apache Arrow's query engine) for SQL execution:

  1. SQL parsing: the parser turns the user query into a logical plan
  2. Optimization: Predicate pushdown, projection pruning, partition pruning
  3. Parquet scanning: the scanner reads only the needed columns and row groups from S3
  4. Vectorized execution: the engine processes Arrow columnar batches in CPU cache-friendly patterns
  5. Aggregation: Final merge across partitions

Query Language Support

Signal Language Example
Logs SQL SELECT * FROM logs WHERE body LIKE '%error%' ORDER BY _timestamp DESC LIMIT 100
Traces SQL SELECT * FROM traces WHERE service_name='api' AND duration > 1000
Metrics PromQL rate(http_requests_total[5m])

Node Roles

In HA mode, OpenObserve separates concerns via ZO_NODE_ROLE:

Role Responsibility
router Dispatches requests to correct backend node
ingester Receives data, writes WAL, flushes Parquet to S3
querier Reads Parquet from S3, executes queries
compactor Merges small Parquet files for efficiency
alertmanager Evaluates alert rules, generates notifications

Workload Separation

ZO_NODE_ROLE_GROUP prevents resource-intensive queries (alerts, reports) from impacting real-time user searches by routing them to dedicated node groups.

Pipelines

OpenObserve supports ingestion-time pipelines for data transformation:

  • Field extraction: Parse structured data from log lines
  • Enrichment: Add fields from lookup tables
  • Filtering: Drop unwanted logs before storage
  • Routing: Send data to different streams based on content

Pipelines use VRL (Vector Remap Language) for transformation logic.

Sources

Benchmarks

Storage efficiency, query performance, and cost analysis for OpenObserve.

Storage Efficiency vs Elasticsearch

Architectural Comparison

Factor OpenObserve Elasticsearch
Storage format Parquet (columnar) + Zstd Lucene segments (row-oriented) + inverted index
Compression ratio ~10:1 ~1.5:1
Storage tier Object storage (S3: $0.023/GB/mo) SSD ($0.10+/GB/mo)
Indexing Optional per-column bloom filters Full inverted index on every field
Net storage cost ~$0.002/GB/mo ~$0.28/GB/mo
Claimed advantage ~140x cheaper storage —

Why 140x Cheaper

The 140x claim combines two factors:

  1. Compression: Parquet columnar + Zstd achieves ~10:1 vs Lucene's ~1.5:1 → ~7x less raw bytes
  2. Storage tier: S3 ($0.023/GB/mo) vs SSD ($0.10+/GB/mo) → ~4–5x cheaper per GB
  3. No replica overhead: S3 provides 11-nines durability natively vs manually replicated Elasticsearch shards → ~2–3x savings

Combined: ~7x × ~4x × ~2.5x ≈ ~70–140x depending on configuration.

Caveat: This is a vendor-provided comparison. Actual ratios depend on data patterns, compression ratios, and S3 pricing tiers.

Query Performance

Analytical Queries (Aggregations)

Aspect OpenObserve Elasticsearch
Column pruning Yes (read only needed columns) No (reads full documents)
Predicate pushdown Yes (DataFusion → Parquet row group stats) Partial (inverted index)
Vectorized execution Yes (Apache Arrow batches) No
Aggregation speed Often faster for analytical patterns Faster for full-text search

Full-Text Search

Aspect OpenObserve Elasticsearch
Approach Parquet scan + bloom filters Inverted index
Wildcard search Full scan (slower) Fast (inverted index)
Best for Known-field searches, aggregations Complex full-text search

Resource Footprint

Single-Node (Dev/POC)

Metric Value
Binary size ~50 MB
Startup time < 5 seconds
Idle RAM ~50–100 MB
Minimum resources 1 CPU, 512 MB RAM

Production HA

Component CPU RAM Storage
Ingester (×3) 2 vCPU 4 GB 100 GB WAL disk
Querier (×2) 4 vCPU 8 GB 50 GB cache disk
Compactor (×1) 2 vCPU 4 GB 50 GB temp
Router (×2) 1 vCPU 1 GB —
AlertManager (×1) 1 vCPU 1 GB —
PostgreSQL 1 vCPU 2 GB 20 GB
S3 — — Unlimited

Cost Comparison (100 GB/day logs, 30-day retention)

Cost Item OpenObserve (self-hosted) Elasticsearch (self-hosted)
Storage ~$7/mo (S3, 300 GB after compression) ~$300/mo (3 TB SSD, 3× replicated)
Compute ~$500/mo (small stateless nodes) ~$1,500/mo (3× data nodes, 64 GB each)
Total ~$507/mo ~$1,800/mo

Scale Limits

Dimension Practical Limit Notes
Daily ingestion PB-scale S3 write throughput bottleneck
Query concurrency 50–100 Add querier replicas
Retention Unlimited S3 lifecycle policies
Streams (indices) 10,000+ Metadata store can need PostgreSQL
Single query scan TB-range DataFusion parallelizes across partitions

Caveats

  • 140x cost claim is from vendor benchmarks and combines compression + storage tier + replication savings.
  • Full-text search performance lags behind Elasticsearch's inverted index for wildcard/fuzzy queries.
  • DataFusion is less battle-tested than ClickHouse or Elasticsearch at extreme scale.
  • Performance varies significantly with data patterns and query types.

Sources

Security

Security model for OpenObserve covering multi-tenancy, RBAC, API token management, S3 bucket encryption, and authentication. See also: observability/openobserve/index, observability/openobserve/explanation, observability/openobserve/how-to-guides.

Security Architecture Overview

OpenObserve provides three tiers of security capability depending on edition:

Feature Open Source Enterprise Cloud
Basic auth (username/password) Yes Yes Yes
Multiple users Yes Yes Yes
Multi-tenancy (organizations) Yes Yes Yes
RBAC (predefined roles) No Yes (with OpenFGA) Yes (preconfigured)
Custom roles No Yes Yes
User groups No Yes Yes
Service accounts No Yes Yes
SSO (SAML/OIDC) No Yes Yes 
flowchart TD
    subgraph "External"
        Clients[API Clients<br/>Collectors / Agents]
        Users[Browser Users]
        IdP[Identity Provider<br/>SAML / OIDC / Dex]
    end

    subgraph "OpenObserve"
        API[API Gateway<br/>Basic Auth Validation]
        RBAC[RBAC Engine<br/>OpenFGA Enterprise]
        Ingester[Ingester<br/>Data Processing]
        Querier[Querier<br/>SQL / PromQL]
    end

    subgraph "Storage"
        Meta[(Meta Store<br/>PostgreSQL / SQLite)]
        S3[S3 / GCS / Azure Blob<br/>SSE-KMS Encrypted]
        WAL[Local WAL<br/>Disk Buffer]
    end

    Clients -->|Authorization: Basic base64 user:pass| API
    Users -->|Login / SSO| API
    API -->|Validate credentials| Meta
    API -->|Check permissions| RBAC
    RBAC --> IdP
    API --> Ingester
    API --> Querier
    Ingester --> WAL --> S3
    Querier --> S3

Authentication

Basic Authentication

All OpenObserve API endpoints require HTTP Basic authentication:

GET /api/default/_search
Authorization: Basic base64("username:password")

The root user is configured at startup:

auth:
  ZO_ROOT_USER_EMAIL: "[email protected]"
  ZO_ROOT_USER_PASSWORD: "secure-password"
  ZO_ROOT_USER_TOKEN: ""

Root User

The root user has unrestricted access to all organizations and settings. The root token (ZO_ROOT_USER_TOKEN) can be used for API automation.

SSO Integration (Enterprise)

OpenObserve Enterprise supports SAML 2.0 and OIDC for single sign-on. The Dex identity service is commonly deployed alongside OpenObserve for OIDC:

# values.yaml for Enterprise Edition
openfga:
  enabled: true    # Required for RBAC
dex:
  enabled: true    # OIDC identity provider

SSO configuration steps: 1. Deploy OpenFGA for authorization decisions. 2. Deploy Dex (or configure an external IdP) for authentication. 3. Map IdP groups to OpenObserve roles via the UI or API.

Multi-Tenancy

OpenObserve uses organizations as the primary multi-tenancy boundary. Each organization has isolated:

  • Streams (logs, metrics, traces)
  • Dashboards
  • Alerts
  • Pipelines
  • User permissions

Organization Isolation

# Query data in a specific organization
GET /api/{organization}/_search
Authorization: Basic base64("user:pass")
  • Data from Organization A is not accessible from Organization B unless the user has explicit permissions in both.
  • Each organization can have separate retention policies, stream settings, and pipeline configurations.
  • Quotas can be set per organization (Enterprise) to limit ingestion volume, storage, and query frequency.

Stream-Level Quotas (Enterprise)

Organizations can enforce quotas to prevent noisy-neighbor scenarios:

  • Maximum ingestion rate per stream.
  • Maximum storage per organization.
  • Maximum query frequency per user.

Role-Based Access Control

Predefined Roles (Enterprise/Cloud)

OpenObserve provides five built-in roles:

Role Dashboards Streams Alerts Users Settings
Root All All All All All
Admin All (org) All (org) All (org) Manage (org) All (org)
Editor Create/Edit Read/Write Create/Edit None None
Viewer View only Read only View only None None
User Limited Limited None None None

Custom Roles (Enterprise)

Administrators can create custom roles with fine-grained permissions using OpenFGA:

POST /api/roles
{
  "name": "logs-analyst",
  "permissions": [
    "streams:read",
    "streams:query",
    "dashboards:read",
    "dashboards:create"
  ]
}

User Groups (Enterprise)

Users can be organized into groups, and permissions assigned at the group level:

  • Group membership managed via the UI or API.
  • Role assignments inherited by all group members.
  • Simplifies permission management for large teams.

Service Accounts (Enterprise)

Service accounts provide machine-to-machine authentication with role-based token access:

POST /api/service_accounts
{
  "name": "fluentbit-ingester",
  "role": "Editor"
}
  • Tokens are scoped to the service account's role and organization.
  • Separate from user accounts. Not affected by user lifecycle events.
  • Suitable for collectors, CI/CD pipelines, and automation tools.

API Token Management

Root Token

The root token is configured at deployment time:

auth:
  ZO_ROOT_USER_TOKEN: "secure-random-token"

This token grants unrestricted API access. Protect it as a secret:

# Kubernetes secret
kubectl create secret generic openobserve-root-token \
    --from-literal=token="$(openssl rand -hex 32)"

User Tokens

Each user authenticates via Basic Auth. The API validates credentials against the metadata store (PostgreSQL or SQLite). There is no separate token issuance for OSS users. Enterprise users can create service account tokens.

Object Storage Encryption

OpenObserve stores all telemetry data in S3-compatible object storage as compressed Parquet files. Encryption at rest is enforced at the bucket level.

S3 Configuration

auth:
  ZO_S3_ACCESS_KEY: "AKIAIOSFODNN7EXAMPLE"
  ZO_S3_SECRET_KEY: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
config:
  ZO_S3_BUCKET_NAME: "openobserve-prod"
  ZO_S3_REGION_NAME: "us-west-1"

EKS IAM Role Integration

On Amazon EKS, use IAM Roles for Service Accounts (IRSA) instead of static credentials:

serviceAccount:
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789:role/zo-s3-eks

This eliminates static S3 credentials from the deployment and relies on temporary, automatically-rotated IAM credentials.

Encryption Recommendations

Method Use Case
SSE-S3 Default. AWS-managed keys
SSE-KMS Recommended. Audit trail via CloudTrail. Customer-managed key rotation
SSE-C Maximum control. The client provides the key per request

Bucket policies must restrict access to the OpenObserve service account only:

{
  "Effect": "Deny",
  "Principal": "*",
  "Action": "s3:*",
  "Resource": "arn:aws:s3:::openobserve-prod/*",
  "Condition": {
    "StringNotEquals": {"aws:PrincipalArn": "arn:aws:iam::123456789:role/zo-s3-eks"}
  }
}

Metadata Store Security

OpenObserve uses PostgreSQL (recommended for production) or SQLite (development) for metadata:

auth:
  ZO_META_POSTGRES_DSN: "postgres://openobserve:password@postgres:5432/openobserve?sslmode=require"
  • Always use sslmode=require for PostgreSQL connections in production.
  • Create a dedicated database user with schema-level permissions only.
  • Store the DSN as a Kubernetes secret, not in plaintext values.

Network Security

Port Protocol Purpose Exposure
5080 HTTP Web UI and API Behind TLS proxy
5081 HTTP gRPC ingestion (OTLP) Internal only
5082 HTTP Internal cluster communication Internal only

TLS Termination

OpenObserve does not include built-in TLS termination. Deploy behind a reverse proxy (Nginx, Envoy, AWS ALB) for HTTPS:

server {
    listen 443 ssl;
    ssl_certificate /etc/nginx/certs/server.crt;
    ssl_certificate_key /etc/nginx/certs/server.key;

    location / {
        proxy_pass http://openobserve:5080;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
    }
}

Hardening Checklist

Area Recommendation
Root credentials Change default password immediately. Use a strong password or token
TLS Terminate TLS at reverse proxy. Enforce HTTPS
S3 encryption Use SSE-KMS with customer-managed keys
S3 credentials Use IAM roles (IRSA) instead of static access keys
PostgreSQL Enable sslmode=require. Use a dedicated user
RBAC Enable OpenFGA for granular permissions (Enterprise)
Service accounts Use dedicated tokens for collectors and automation
Network Restrict gRPC ingestion port to internal traffic only
Secrets Store all credentials in Kubernetes secrets or Vault
Quotas Set ingestion and storage quotas per organization

Sources