Skip to content

OpenObserve — How It Works

How OpenObserve uses Rust, Apache Parquet, and object storage to deliver cost-efficient observability.

Architecture

Component Roles

flowchart TB
    subgraph Sources["Data Sources"]
        OTEL["OTel Collector"]
        PROM["Prometheus"]
        FB["FluentBit / Vector"]
        ES_API["ES Bulk API\nclients"]
        RUM_SDK["RUM SDK"]
        KF["Kinesis Firehose"]
    end

    subgraph O2["OpenObserve Cluster"]
        direction TB
        Router["Router\n(request dispatch)"]
        Ingester["Ingester\n(WAL → Parquet)"]
        Querier["Querier\n(DataFusion)"]
        Compactor["Compactor\n(file merging)"]
        Alert["AlertManager\n(alerts + reports)"]
    end

    subgraph Storage["Storage Layer"]
        WAL["WAL\n(local disk)"]
        S3["Object Storage\n(S3/GCS/Azure/MinIO)"]
        Meta["Metadata Store\n(PostgreSQL / SQLite)"]
    end

    Sources --> Router
    Router --> Ingester
    Ingester -->|batch| WAL
    WAL -->|flush| S3
    Querier --> S3
    Compactor --> S3
    Alert --> Querier

    style O2 fill:#e65100,color:#fff

Data Flow: Ingestion to Query

sequenceDiagram
    participant Client as Data Source
    participant Router as Router
    participant Ingester as Ingester
    participant WAL as Local WAL
    participant S3 as Object Storage
    participant Compactor as Compactor
    participant Querier as Querier

    Client->>Router: OTLP / ES Bulk / Prom RW
    Router->>Ingester: Route by stream
    Ingester->>Ingester: Schema inference
    Ingester->>WAL: Write to memtable
    WAL->>S3: Flush as Parquet (5min or size threshold)
    Note over S3: Small Parquet files
    Compactor->>S3: Merge small files
    Compactor->>S3: Write optimized Parquet
    Note over S3: Large, sorted Parquet files
    Querier->>S3: Read Parquet partitions
    Querier->>Querier: DataFusion vectorized execution
    Querier->>Client: Return results

Storage Architecture

Parquet on Object Storage

OpenObserve stores all data as Apache Parquet files on object storage:

Aspect Detail
Format Apache Parquet (columnar)
Compression Zstd (default)
Partitioning By stream, date, and time window
Object storage S3, GCS, Azure Blob, MinIO
Local mode Disk-backed (for dev/single-node)
Bloom filters Per-column for high-cardinality field acceleration

Why 140x Cheaper Than Elasticsearch

Factor OpenObserve Elasticsearch
Compression Parquet columnar + Zstd (~10:1) Lucene segments (~1.5:1)
Storage tier S3 ($0.023/GB/mo) SSD ($0.10+/GB/mo)
Compute Stateless, scale to zero Always-on data nodes
Replicas S3 provides 11-nines Manually replicated shards
Net effect ~$0.002/GB/mo storage ~$0.28/GB/mo storage

Query Engine: Apache Arrow DataFusion

OpenObserve uses DataFusion (Apache Arrow's query engine) for SQL execution:

  1. SQL parsing: User query parsed into logical plan
  2. Optimization: Predicate pushdown, projection pruning, partition pruning
  3. Parquet scanning: Only reads needed columns and row groups from S3
  4. Vectorized execution: Arrow columnar batches processed in CPU cache-friendly patterns
  5. Aggregation: Final merge across partitions

Query Language Support

Signal Language Example
Logs SQL SELECT * FROM logs WHERE body LIKE '%error%' ORDER BY _timestamp DESC LIMIT 100
Traces SQL SELECT * FROM traces WHERE service_name='api' AND duration > 1000
Metrics PromQL rate(http_requests_total[5m])

Node Roles

In HA mode, OpenObserve separates concerns via ZO_NODE_ROLE:

Role Responsibility
router Dispatches requests to correct backend node
ingester Receives data, writes WAL, flushes Parquet to S3
querier Reads Parquet from S3, executes queries
compactor Merges small Parquet files for efficiency
alertmanager Evaluates alert rules, generates notifications

Workload Separation

ZO_NODE_ROLE_GROUP prevents resource-intensive queries (alerts, reports) from impacting real-time user searches by routing them to dedicated node groups.

Pipelines

OpenObserve supports ingestion-time pipelines for data transformation:

  • Field extraction: Parse structured data from log lines
  • Enrichment: Add fields from lookup tables
  • Filtering: Drop unwanted logs before storage
  • Routing: Send data to different streams based on content

Pipelines use VRL (Vector Remap Language) for transformation logic.

Sources