Skip to content

Explanation

How the Victoria Stack works: topology, storage engines, data flow, and the security model. See also: hub, Reference, How-to guides.

Default Topology

flowchart TB
    subgraph Sources["Data Sources"]
        K8s["Kubernetes<br/>Pods & Services"]
        Apps["Applications<br/>(OTel SDK)"]
        Infra["Infrastructure<br/>(node_exporter, etc.)"]
        Logs["Log Sources<br/>(Fluentbit, Logstash)"]
    end

    subgraph Collection["Collection Layer"]
        Agent["vmagent<br/>(DaemonSet)<br/>Scrape + Push"]
        OTel["OTel Collector<br/>(optional)"]
    end

    subgraph Proxy["Routing Layer"]
        Auth["vmauth<br/>Auth · Route · LB"]
    end

    subgraph MetricsCluster["VictoriaMetrics (Metrics)"]
        MI["vminsert ×2"]
        MS["vmstorage ×3<br/>(StatefulSet, SSD)"]
        MSel["vmselect ×2"]
        MI --> MS
        MSel --> MS
    end

    subgraph LogsCluster["VictoriaLogs (Logs)"]
        LI["vlinsert ×2"]
        LS["vlstorage ×3<br/>(StatefulSet, SSD)"]
        LSel["vlselect ×2"]
        LI --> LS
        LSel --> LS
    end

    subgraph TracesCluster["VictoriaTraces (Traces)"]
        TI["vtinsert ×2"]
        TS["vtstorage ×3<br/>(StatefulSet, SSD)"]
        TSel["vtselect ×2"]
        TI --> TS
        TSel --> TS
    end

    subgraph Alerting["Alerting"]
        Alert["vmalert"]
        AM["Alertmanager"]
    end

    subgraph Viz["Visualization"]
        Grafana["Grafana"]
        VMUI["VMUI<br/>(built-in)"]
    end

    Sources --> Collection
    Collection --> Auth
    Logs --> Auth

    Auth -->|"Metrics: /api/v1/write"| MI
    Auth -->|"Logs: /insert/jsonline"| LI
    Auth -->|"Traces: /insert/opentelemetry"| TI

    Auth -->|"PromQL query"| MSel
    Auth -->|"LogsQL query"| LSel
    Auth -->|"Jaeger query"| TSel

    Grafana --> Auth
    VMUI --> Auth
    Alert --> Auth
    Alert -->|"Fire alerts"| AM

    style Sources fill:#0d7377,color:#fff
    style Collection fill:#ff6600,color:#fff
    style Proxy fill:#7b42bc,color:#fff
    style MetricsCluster fill:#2a2d3e,color:#fff
    style LogsCluster fill:#2a7de1,color:#fff
    style TracesCluster fill:#e65100,color:#fff
    style Alerting fill:#c62828,color:#fff
    style Viz fill:#ff6600,color:#fff

Deployment Modes

Single-Node vs Cluster

Feature Single-Node Cluster
Scalability Vertical only Horizontal & Vertical
Operational Complexity Very Low (1 binary) Moderate (3 component types)
Multi-tenancy No Yes (via account IDs)
Replication No (relies on durable disk) Yes (-replicationFactor=N)
Target Workload Up to ~1M samples/sec Billions of series, 100M+ samples/sec
External Dependencies None None

Recommendation: Start with single-node. Only move to cluster when you need multi-tenancy, horizontal scaling beyond a single machine, or application-level replication.

Component Roles

Each signal type follows the same tri-component pattern for cluster mode:

Component Role Metrics Logs Traces
Ingestion vminsert vlinsert vtinsert
Querying vmselect vlselect vtselect
Storage vmstorage vlstorage vtstorage

All three types are stateless (insert/select) or stateful (storage), and can be scaled independently.

vmalert Evaluation Flow

sequenceDiagram
    participant A as vmalert
    participant P as vmauth (Proxy)
    participant VM as VictoriaMetrics / Logs
    participant AM as Alertmanager

    Note over A: Evaluate Rules (periodic)
    A->>P: POST /api/v1/query (Query Request)
    P->>VM: Inspect path & Forward to backend
    VM-->>P: Return Query Results
    P-->>A: Return Query Results

    alt Alert Triggered
        A->>AM: Send Alert Notification
    else Recording Rule
        A->>VM: Remote Write Results
    end

Multi-Source Log Ingestion

VictoriaLogs accepts logs from virtually any source without translation:

flowchart LR
    A["Promtail"] -->|"Loki Push API"| B{"vmauth"}
    C["Fluent Bit"] -->|"JSON Lines"| B
    D["Logstash"] -->|"ES Bulk API"| B
    E["OTel Collector"] -->|"OTLP"| B
    F["rsyslog"] -->|"Syslog"| B
    B -->|"Route & Auth"| G["vlinsert"]
    G --> H[("vlstorage")]

    style B fill:#7b42bc,color:#fff
    style H fill:#2a7de1,color:#fff

Storage Layout

VictoriaMetrics (Metrics)

/path/to/vmstorage/data/
├── big/                    # Large, compacted data blocks
│   ├── YYYY_MM/           # Monthly partitions
│   │   ├── parts/         # Compressed TSDB blocks
│   │   └── tmp/           # Temporary merge workspace
├── small/                  # Recently ingested, small blocks
│   └── YYYY_MM/
├── indexdb/               # Inverted index (label → series ID)
└── snapshots/             # Point-in-time snapshots (for vmbackup)

VictoriaLogs (Logs)

/path/to/vlstorage/data/
├── YYYYMMDD/              # Daily partitions
│   ├── bloom_filters/     # Bloom filters for word matching
│   ├── columns/           # Columnar storage (msg, timestamp, labels)
│   └── metadata/

VictoriaTraces (Traces)

VictoriaTraces uses the same storage engine as VictoriaLogs (daily partitions, bloom filters, columnar format) but organizes data by trace ID and span attributes.

Key Design Decisions

Decision Rationale
No external dependencies No PostgreSQL, Redis, ZooKeeper, or object storage required — reduces operational surface
Local disk > Object storage SSDs provide lower latency than S3. Compression compensates for limited capacity
Shared-nothing cluster vmstorage nodes don't communicate — each owns its shard. This simplifies scaling
Consistent hashing vminsert distributes data deterministically without consensus protocol overhead
Bloom filters (VictoriaLogs) Dramatically less RAM than inverted indexes at the cost of slightly higher scan overhead
Apache 2.0 license More permissive than AGPL — no copyleft obligations for SaaS usage

How It Works

Core Mechanisms

Storage Engine

All Victoria databases share foundational design principles:

  1. LSM-Tree Storage: The database uses a custom implementation of the Log-Structured Merge-Tree optimized exclusively for telemetry appending. Incoming data is written to in-memory buffers, then flushed to immutable on-disk files which are periodically merged (compacted) in the background.
  2. ZSTD Compression: All data is compressed using ZSTD with delta-encoding tuned specifically for floats and timestamps. This achieves ~50% less disk usage than Prometheus and 10–20x less than Elasticsearch.
  3. Deterministic Sharding: In cluster mode, vminsert hashes incoming metrics by their labels to decide which vmstorage nodes own them. This negates the need for a complex internal distributed consensus algorithm (like Raft/Paxos).
  4. Data Localization: Blocks of time-series data are grouped tightly by time buckets, compressed, and written to disk asynchronously.
  5. Native Translationless Ingestion: VictoriaTraces opens HTTP/gRPC ports for OTLP data, while VictoriaLogs directly accepts Loki API, Elasticsearch Bulk, and Fluentbit JSON. This bypasses the heavy CPU overhead usually required to translate signals into internal formats.

VictoriaMetrics (Metrics)

flowchart LR
    subgraph Ingestion["Ingestion APIs"]
        PR["Prometheus<br/>remote_write"]
        IL["InfluxDB<br/>line protocol"]
        DD["Datadog<br/>API"]
        OT["OTLP"]
        GR["Graphite"]
    end

    subgraph VM["VictoriaMetrics"]
        WB["Write Buffer<br/>(in-memory)"]
        TSDB["LSM-Tree TSDB<br/>(on-disk, ZSTD)"]
        IDX["Inverted Index<br/>(label → series ID)"]
    end

    subgraph Query["Query"]
        MQL["MetricsQL / PromQL<br/>Engine"]
    end

    Ingestion --> WB --> TSDB
    WB --> IDX
    MQL --> IDX --> TSDB
    MQL --> Grafana["Grafana"]

    style VM fill:#2a2d3e,color:#fff
    style Ingestion fill:#0d7377,color:#fff
    style Query fill:#ff6600,color:#fff

Key insight: VictoriaMetrics stores time-series data using a custom columnar format where timestamps and values are stored in separate columns. This enables efficient batch reads and high compression ratios.

VictoriaLogs (Logs)

flowchart LR
    subgraph Ingestion["Ingestion APIs"]
        LK["Loki Push API"]
        ES["Elasticsearch<br/>Bulk API"]
        SL["Syslog"]
        OT["OTLP"]
        FB["Fluentbit<br/>JSON Lines"]
    end

    subgraph VL["VictoriaLogs"]
        direction TB
        Parse["Parser<br/>(structured + unstructured)"]
        BF["Bloom Filters<br/>(instead of inverted index)"]
        Col["Columnar Storage<br/>(daily partitions)"]
    end

    Ingestion --> Parse --> BF
    Parse --> Col

    LogsQL["LogsQL Engine"] --> BF --> Col
    LogsQL --> Grafana["Grafana"]

    style VL fill:#2a7de1,color:#fff
    style Ingestion fill:#0d7377,color:#fff

Key insight: VictoriaLogs uses Bloom filters instead of traditional inverted indexes. This dramatically reduces RAM and CPU usage — but means full-text search relies on sequential scanning through bloom-filtered partitions rather than instant index lookups.

Storage: Data is organized into daily partitions (for example, 20260410/). This enables efficient retention management by deleting old partition directories.

VictoriaTraces (Traces)

flowchart LR
    subgraph Ingestion["Ingestion"]
        OTLP_H["OTLP HTTP<br/>:10428"]
        OTLP_G["OTLP gRPC<br/>:4317"]
        JG["Jaeger"]
        ZP["Zipkin"]
    end

    subgraph VT["VictoriaTraces"]
        direction TB
        TP["Trace Parser"]
        VLS["VictoriaLogs<br/>Storage Engine"]
    end

    Ingestion --> TP --> VLS

    JQ["Jaeger Query API"] --> VLS
    TQ["Tempo DS API<br/>(experimental v0.8+)<br/>/tags, /search, /v2/traces"] --> VLS
    JQ --> Grafana["Grafana<br/>(Jaeger or Tempo DS)"]
    TQ --> Grafana

    style VT fill:#e65100,color:#fff
    style Ingestion fill:#0d7377,color:#fff

Key insight: VictoriaTraces is built on top of the VictoriaLogs storage engine, inheriting its columnar storage, bloom filters, and compression. It does NOT require external object storage — everything runs on local disk.

Tempo DS compatibility (v0.8+): As of v0.8.0 (March 2026), VictoriaTraces exposes experimental Grafana Tempo datasource APIs (/tags, /search, /v2/traces/*). This enables use with Grafana's native Tempo datasource. TraceQL metrics and pipelines are not yet supported — but basic trace search and lookup work. This makes VT a partial drop-in for Tempo for simple use cases.

Data Flow

sequenceDiagram
    participant App as Applications
    participant Agent as vmagent / OTel Collector
    participant Auth as vmauth (Proxy)
    participant MI as vminsert / vlinsert / vtinsert
    participant MS as vmstorage / vlstorage / vtstorage
    participant MQ as vmselect / vlselect / vtselect
    participant G as Grafana

    App->>Agent: Emit metrics / logs / traces
    Agent->>Auth: Push telemetry (HTTP/gRPC)
    Auth->>Auth: Route by URL path
    Auth->>MI: Forward to correct insert node
    MI->>MS: Hash & distribute to storage
    Note over MS: LSM-Tree write + ZSTD compress
    Note over MS: Background merge & compaction

    G->>Auth: Query (PromQL / LogsQL / Jaeger API)
    Auth->>MQ: Forward to select node
    MQ->>MS: Fetch data chunks
    MS-->>MQ: Return compressed data
    MQ-->>Auth: Aggregate, sort, deduplicate
    Auth-->>G: Return results

Pull Sequence

  1. vmagent scrapes Prometheus targets and pushes data to vmauth
  2. vmauth reads the HTTP path (for example, /api/v1/write vs /insert/jsonline) and routes the payload to the correct backend

Write Sequence

  1. vminsert (or vlinsert/vtinsert) hashes the payload and distributes it to backend storage nodes
  2. Storage nodes write to in-memory buffer + WAL, then async-flush to disk

Merge Sequence

  1. In the background, storage nodes continually merge small data files into larger chunks (LSM compaction) for faster sequential reads

Read Sequence

  1. Grafana sends a query to vmselect via vmauth
  2. vmselect asks all relevant vmstorage nodes for data chunks
  3. vmselect sorts, deduplicates, and runs aggregation functions natively
  4. vmselect returns the results to Grafana

Security Architecture Overview

VictoriaMetrics does not include built-in authentication or authorization in its open-source components. All cluster components (vminsert, vmselect, vmstorage) must operate within a protected private network with no direct internet exposure. Authentication proxies mediate all external access: vmauth (open-source) or vmgateway (Enterprise).

flowchart TD
    subgraph "External"
        Clients[API Clients<br/>Grafana / vmagent]
        Users[Browser Users]
    end

    subgraph "Auth Proxy Layer"
        Vmauth[vmauth<br/>Token / Basic Auth / JWT / mTLS]
        Vmgateway[vmgateway Enterprise<br/>OIDC / JWT Claims]
    end

    subgraph "VictoriaMetrics Cluster"
        Vminsert[vminsert<br/>:8480]
        Vmselect[vmselect<br/>:8481]
        Vmstorage[vmstorage<br/>:8482]
    end

    Clients -->|Bearer Token / Basic Auth| Vmauth
    Users -->|OIDC JWT| Vmgateway
    Vmauth -->|Route by token -> tenant| Vminsert
    Vmauth -->|Route by token -> tenant| Vmselect
    Vmgateway -->|Parse JWT vm_access claims| Vminsert
    Vmgateway -->|Parse JWT vm_access claims| Vmselect
    Vminsert --> Vmstorage
    Vmselect --> Vmstorage

Network Topology

All VictoriaMetrics cluster components must run in a private network. The recommended topology:

Component Network Zone Exposure
vmstorage Private subnet No external access
vminsert Private subnet Behind vmauth only
vmselect Private subnet Behind vmauth only
vmauth DMZ / public subnet TLS termination point

Multi-Tenant Data Isolation

In cluster mode, tenant data is isolated by tenant ID encoded in the URL path:

/insert/<accountID>/prometheus/api/v1/write
/select/<accountID>/prometheus/api/v1/query
  • accountID is a numeric identifier (0 = default tenant).
  • vmstorage stores data per tenant in separate directories.
  • vmauth maps authentication tokens to specific accountID values.
  • Cross-tenant queries require explicit configuration with tenant federation.

VictoriaLogs uses AccountID and ProjectID headers for multi-tenancy. For the vmauth configuration that injects these headers, see VictoriaLogs Tenant Isolation.