Architecture¶

1. Default Topology / Flow¶

flowchart TB
    subgraph Sources["Data Sources"]
        K8s["Kubernetes<br/>Pods & Services"]
        Apps["Applications<br/>(OTel SDK)"]
        Infra["Infrastructure<br/>(node_exporter, etc.)"]
        Logs["Log Sources<br/>(Fluentbit, Logstash)"]
    end

    subgraph Collection["Collection Layer"]
        Agent["vmagent<br/>(DaemonSet)<br/>Scrape + Push"]
        OTel["OTel Collector<br/>(optional)"]
    end

    subgraph Proxy["Routing Layer"]
        Auth["vmauth<br/>Auth · Route · LB"]
    end

    subgraph MetricsCluster["VictoriaMetrics (Metrics)"]
        MI["vminsert ×2"]
        MS["vmstorage ×3<br/>(StatefulSet, SSD)"]
        MSel["vmselect ×2"]
        MI --> MS
        MSel --> MS
    end

    subgraph LogsCluster["VictoriaLogs (Logs)"]
        LI["vlinsert ×2"]
        LS["vlstorage ×3<br/>(StatefulSet, SSD)"]
        LSel["vlselect ×2"]
        LI --> LS
        LSel --> LS
    end

    subgraph TracesCluster["VictoriaTraces (Traces)"]
        TI["vtinsert ×2"]
        TS["vtstorage ×3<br/>(StatefulSet, SSD)"]
        TSel["vtselect ×2"]
        TI --> TS
        TSel --> TS
    end

    subgraph Alerting["Alerting"]
        Alert["vmalert"]
        AM["Alertmanager"]
    end

    subgraph Viz["Visualization"]
        Grafana["Grafana"]
        VMUI["VMUI<br/>(built-in)"]
    end

    Sources --> Collection
    Collection --> Auth
    Logs --> Auth

    Auth -->|"Metrics: /api/v1/write"| MI
    Auth -->|"Logs: /insert/jsonline"| LI
    Auth -->|"Traces: /insert/opentelemetry"| TI

    Auth -->|"PromQL query"| MSel
    Auth -->|"LogsQL query"| LSel
    Auth -->|"Jaeger query"| TSel

    Grafana --> Auth
    VMUI --> Auth
    Alert --> Auth
    Alert -->|"Fire alerts"| AM

    style Sources fill:#0d7377,color:#fff
    style Collection fill:#ff6600,color:#fff
    style Proxy fill:#7b42bc,color:#fff
    style MetricsCluster fill:#2a2d3e,color:#fff
    style LogsCluster fill:#2a7de1,color:#fff
    style TracesCluster fill:#e65100,color:#fff
    style Alerting fill:#c62828,color:#fff
    style Viz fill:#ff6600,color:#fff

Deployment Modes¶

Single-Node vs Cluster¶

Feature	Single-Node	Cluster
Scalability	Vertical only	Horizontal & Vertical
Operational Complexity	Very Low (1 binary)	Moderate (3 component types)
Multi-tenancy	No	Yes (via account IDs)
Replication	No (relies on durable disk)	Yes (`-replicationFactor=N`)
Target Workload	Up to ~1M samples/sec	Billions of series, 100M+ samples/sec
External Dependencies	None	None

Recommendation: Start with single-node. Only move to cluster when you need multi-tenancy, horizontal scaling beyond a single machine, or application-level replication.

Component Roles¶

Each signal type follows the same tri-component pattern for cluster mode:

Component Role	Metrics	Logs	Traces
Ingestion	`vminsert`	`vlinsert`	`vtinsert`
Querying	`vmselect`	`vlselect`	`vtselect`
Storage	`vmstorage`	`vlstorage`	`vtstorage`

All three types are stateless (insert/select) or stateful (storage), and can be scaled independently.

Full Stack Architecture¶

vmalert Evaluation Flow¶

sequenceDiagram
    participant A as vmalert
    participant P as vmauth (Proxy)
    participant VM as VictoriaMetrics / Logs
    participant AM as Alertmanager

    Note over A: Evaluate Rules (periodic)
    A->>P: POST /api/v1/query (Query Request)
    P->>VM: Inspect path & Forward to backend
    VM-->>P: Return Query Results
    P-->>A: Return Query Results

    alt Alert Triggered
        A->>AM: Send Alert Notification
    else Recording Rule
        A->>VM: Remote Write Results
    end

Multi-Source Log Ingestion¶

VictoriaLogs accepts logs from virtually any source without translation:

flowchart LR
    A["Promtail"] -->|"Loki Push API"| B{"vmauth"}
    C["Fluent Bit"] -->|"JSON Lines"| B
    D["Logstash"] -->|"ES Bulk API"| B
    E["OTel Collector"] -->|"OTLP"| B
    F["rsyslog"] -->|"Syslog"| B
    B -->|"Route & Auth"| G["vlinsert"]
    G --> H[("vlstorage")]

    style B fill:#7b42bc,color:#fff
    style H fill:#2a7de1,color:#fff

Storage Layout¶

VictoriaMetrics (Metrics)¶

/path/to/vmstorage/data/
├── big/                    # Large, compacted data blocks
│   ├── YYYY_MM/           # Monthly partitions
│   │   ├── parts/         # Compressed TSDB blocks
│   │   └── tmp/           # Temporary merge workspace
├── small/                  # Recently ingested, small blocks
│   └── YYYY_MM/
├── indexdb/               # Inverted index (label → series ID)
└── snapshots/             # Point-in-time snapshots (for vmbackup)

VictoriaLogs (Logs)¶

/path/to/vlstorage/data/
├── YYYYMMDD/              # Daily partitions
│   ├── bloom_filters/     # Bloom filters for word matching
│   ├── columns/           # Columnar storage (msg, timestamp, labels)
│   └── metadata/

VictoriaTraces (Traces)¶

Uses the same storage engine as VictoriaLogs (daily partitions, bloom filters, columnar format) but organizes data by trace ID and span attributes.

Kubernetes Deployment Matrix¶

Component	Kind	Replicas (Min HA)	Key Resource	Helm Chart
vmagent	DaemonSet or Deployment	1 per node (DS) or 2+	CPU, Memory	`victoria-metrics-agent`
vmauth	Deployment	2+	CPU	`victoria-metrics-auth`
vminsert	Deployment	2+	CPU	`victoria-metrics-cluster`
vmselect	Deployment	2+	CPU, Memory	`victoria-metrics-cluster`
vmstorage	StatefulSet	3+	Disk IOPS, Memory	`victoria-metrics-cluster`
VictoriaLogs	StatefulSet (single-node) or cluster	1–3	Disk, Memory	`victoria-logs-single`
VictoriaTraces	StatefulSet (single-node)	1	Disk, Memory	—
vmalert	Deployment	1–2	CPU	`victoria-metrics-alert`
vmoperator	Deployment	1	CPU	`victoria-metrics-operator`

Key Design Decisions¶

Decision	Rationale
No external dependencies	No PostgreSQL, Redis, ZooKeeper, or object storage required — reduces operational surface
Local disk > Object storage	SSDs provide lower latency than S3; compression compensates for limited capacity
Shared-nothing cluster	vmstorage nodes don't communicate — each owns its shard, simplifying scaling
Consistent hashing	vminsert distributes data deterministically without consensus protocol overhead
Bloom filters (VictoriaLogs)	Dramatically less RAM than inverted indexes at the cost of slightly higher scan overhead
Apache 2.0 license	More permissive than AGPL — no copyleft obligations for SaaS usage

Data Model¶

1. Default Topology / Flow¶

erDiagram
    Victoriametrics_CORE ||--o{ CONFIG : requires
    Victoriametrics_CORE ||--o{ STATE : writes
    CONFIG {
        string runtime_params
        string limits
    }
    STATE {
        string metric_id
        json payload
    }

How It Works¶

Core Mechanisms¶

Storage Engine¶

All Victoria databases share foundational design principles:

LSM-Tree Storage: Uses a custom implementation of the Log-Structured Merge-Tree optimized exclusively for telemetry appending. Incoming data is written to in-memory buffers, then flushed to immutable on-disk files which are periodically merged (compacted) in the background.
ZSTD Compression: All data is compressed using ZSTD with delta-encoding tuned specifically for floats and timestamps, achieving ~50% less disk usage than Prometheus and 10–20x less than Elasticsearch.
Deterministic Sharding: In cluster mode, vminsert hashes incoming metrics by their labels to decide which vmstorage nodes should own them. This negates the need for a complex internal distributed consensus algorithm (like Raft/Paxos).
Data Localization: Blocks of time-series data are grouped tightly by time buckets, compressed, and written to disk asynchronously.
Native Translationless Ingestion: VictoriaTraces opens HTTP/gRPC ports for OTLP data, while VictoriaLogs directly accepts Loki API, Elasticsearch Bulk, and Fluentbit JSON. This bypasses the heavy CPU overhead usually required to translate signals into internal formats.

VictoriaMetrics (Metrics)¶

flowchart LR
    subgraph Ingestion["Ingestion APIs"]
        PR["Prometheus<br/>remote_write"]
        IL["InfluxDB<br/>line protocol"]
        DD["Datadog<br/>API"]
        OT["OTLP"]
        GR["Graphite"]
    end

    subgraph VM["VictoriaMetrics"]
        WB["Write Buffer<br/>(in-memory)"]
        TSDB["LSM-Tree TSDB<br/>(on-disk, ZSTD)"]
        IDX["Inverted Index<br/>(label → series ID)"]
    end

    subgraph Query["Query"]
        MQL["MetricsQL / PromQL<br/>Engine"]
    end

    Ingestion --> WB --> TSDB
    WB --> IDX
    MQL --> IDX --> TSDB
    MQL --> Grafana["Grafana"]

    style VM fill:#2a2d3e,color:#fff
    style Ingestion fill:#0d7377,color:#fff
    style Query fill:#ff6600,color:#fff

Key insight: VictoriaMetrics stores time-series data using a custom columnar format where timestamps and values are stored in separate columns, enabling efficient batch reads and high compression ratios.

VictoriaLogs (Logs)¶

flowchart LR
    subgraph Ingestion["Ingestion APIs"]
        LK["Loki Push API"]
        ES["Elasticsearch<br/>Bulk API"]
        SL["Syslog"]
        OT["OTLP"]
        FB["Fluentbit<br/>JSON Lines"]
    end

    subgraph VL["VictoriaLogs"]
        direction TB
        Parse["Parser<br/>(structured + unstructured)"]
        BF["Bloom Filters<br/>(instead of inverted index)"]
        Col["Columnar Storage<br/>(daily partitions)"]
    end

    Ingestion --> Parse --> BF
    Parse --> Col

    LogsQL["LogsQL Engine"] --> BF --> Col
    LogsQL --> Grafana["Grafana"]

    style VL fill:#2a7de1,color:#fff
    style Ingestion fill:#0d7377,color:#fff

Key insight: VictoriaLogs uses Bloom filters instead of traditional inverted indexes. This dramatically reduces RAM and CPU usage — but means full-text search relies on sequential scanning through bloom-filtered partitions rather than instant index lookups.

Storage: Data is organized into daily partitions (e.g., 20260410/), enabling efficient retention management by simply deleting old partition directories.

VictoriaTraces (Traces)¶

flowchart LR
    subgraph Ingestion["Ingestion"]
        OTLP_H["OTLP HTTP<br/>:10428"]
        OTLP_G["OTLP gRPC<br/>:4317"]
        JG["Jaeger"]
        ZP["Zipkin"]
    end

    subgraph VT["VictoriaTraces"]
        direction TB
        TP["Trace Parser"]
        VLS["VictoriaLogs<br/>Storage Engine"]
    end

    Ingestion --> TP --> VLS

    JQ["Jaeger Query API"] --> VLS
    TQ["Tempo DS API<br/>(experimental v0.8+)<br/>/tags, /search, /v2/traces"] --> VLS
    JQ --> Grafana["Grafana<br/>(Jaeger or Tempo DS)"]
    TQ --> Grafana

    style VT fill:#e65100,color:#fff
    style Ingestion fill:#0d7377,color:#fff

Key insight: VictoriaTraces is built on top of the VictoriaLogs storage engine, inheriting its columnar storage, bloom filters, and compression. It does NOT require external object storage — everything runs on local disk.

Tempo DS compatibility (v0.8+): As of v0.8.0 (March 2026), VictoriaTraces exposes experimental Grafana Tempo datasource APIs (/tags, /search, /v2/traces/*), enabling use with Grafana's native Tempo datasource. TraceQL metrics and pipelines are not yet supported — but basic trace search and lookup work, making VT a partial drop-in for Tempo for simple use cases.

Query Languages¶

MetricsQL (Metrics)¶

MetricsQL is a backward-compatible superset of PromQL with extensions that fix common PromQL pain points:

Feature	PromQL	MetricsQL
Auto lookbehind window	❌ Required	✅ Auto-calculated from step
`rate()` extrapolation	Yes (causes fractional integers)	No (returns accurate integers)
`keep_metric_names`	❌	✅ Preserves metric names
Numeric suffixes (Ki, Mi, Gi)	❌	✅ `8Ki` = `8 * 1024`
NaN handling	Returns NaN	Automatically removes NaN
`aggr_over_time()`	❌	✅ Multiple aggregates in one pass
Graphite filter syntax	❌	✅ `{__graphite__="foo.*.bar"}`

# MetricsQL example: rate without explicit window (auto-calculated)
rate(http_requests_total{job="api"})

# Keep metric names when applying functions
rate({__name__=~"foo|bar"}) keep_metric_names

# Use numeric suffixes
process_memory_bytes > 2Gi

LogsQL (Logs)¶

LogsQL is purpose-built for VictoriaLogs with a pipe-based syntax:

# Filter error logs from last 5 minutes
_time:5m AND level:="error"

# Full-text search with pipe transformations
error connection refused | stats count() by (host)

# Extract fields at query time
_time:1h | extract "status=<status_code>" | stats count() by (status_code)

# JSON log parsing
_time:5m | unpack_json | level:="error" | fields _time, msg, trace_id

Key difference from LogQL (Loki): LogsQL does NOT require a mandatory label selector. You can search across all log streams with free-text queries, while Loki requires {label="value"} first.

Data Flow¶

sequenceDiagram
    participant App as Applications
    participant Agent as vmagent / OTel Collector
    participant Auth as vmauth (Proxy)
    participant MI as vminsert / vlinsert / vtinsert
    participant MS as vmstorage / vlstorage / vtstorage
    participant MQ as vmselect / vlselect / vtselect
    participant G as Grafana

    App->>Agent: Emit metrics / logs / traces
    Agent->>Auth: Push telemetry (HTTP/gRPC)
    Auth->>Auth: Route by URL path
    Auth->>MI: Forward to correct insert node
    MI->>MS: Hash & distribute to storage
    Note over MS: LSM-Tree write + ZSTD compress
    Note over MS: Background merge & compaction

    G->>Auth: Query (PromQL / LogsQL / Jaeger API)
    Auth->>MQ: Forward to select node
    MQ->>MS: Fetch data chunks
    MS-->>MQ: Return compressed data
    MQ-->>Auth: Aggregate, sort, deduplicate
    Auth-->>G: Return results

Pull Sequence¶

vmagent scrapes Prometheus targets and pushes data to vmauth
vmauth reads the HTTP path (e.g., /api/v1/write vs /insert/jsonline) and routes the payload to the correct backend

Write Sequence¶

vminsert (or vlinsert/vtinsert) hashes the payload and distributes it to backend storage nodes
Storage nodes write to in-memory buffer + WAL, then async-flush to disk

Merge Sequence¶

In the background, storage nodes continually merge small data files into larger chunks (LSM compaction) for faster sequential reads

Read Sequence¶

Grafana sends a query to vmselect via vmauth
vmselect asks all relevant vmstorage nodes for data chunks
vmselect sorts, deduplicates, and runs aggregation functions natively
Results returned to Grafana

Benchmarks¶

Test Conditions¶

VictoriaMetrics versions: v1.136.x (stable), v1.122.x (LTS)
Date: April 2026
Note: VictoriaMetrics publishes a community benchmarking tool: prometheus-benchmark. Results below combine official benchmarks and community reports.

VictoriaMetrics vs Competitors (Metrics)¶

Resource Efficiency¶

Metric	VictoriaMetrics	Prometheus	Thanos	Mimir
RAM usage (1M active series)	~2 GB	~10–20 GB	~8–15 GB	~8–12 GB
Disk usage (compression)	0.4–1.0 bytes/sample	1.3 bytes/sample	1.5 bytes/sample	1.2–1.5 bytes/sample
Ingestion (single-node)	~1M samples/sec	~500k samples/sec	N/A (sidecar)	N/A (distributed)
Ingestion (cluster)	100M+ samples/sec	N/A	~50M+	30M+
Query latency (simple PromQL)	< 100ms	< 100ms (local)	Variable (ObjStore)	< 200ms

Key Findings¶

RAM: VictoriaMetrics consistently uses 5–10x less RAM than Prometheus for the same workload due to aggressive memory management and ZSTD compression
Disk: ~50% less disk than Prometheus, due to specialized ZSTD + delta-encoding for floats/timestamps
Cardinality resilience: VM handles high-cardinality spikes more gracefully via -memory.allowedPercent, while Prometheus OOM-kills

Production Scale Records¶

Company	Scale	Notes
Roblox	Billions of active series	100% uptime across quarters
DreamHost	76M active series	80% memory reduction vs previous stack
Grammarly	Large-scale metrics	10x cost reduction
CERN	CMS detector monitoring	Long-term Prometheus storage

VictoriaLogs Benchmarks¶

Metric	VictoriaLogs	Loki	Elasticsearch
Storage efficiency	10–30x less disk than ES	10–20x less than ES	Baseline
RAM usage	Minimal (bloom filters, no inverted index)	Low (label-only index)	High (inverted index)
Ingestion	Very high (columnar, batch writes)	High	Moderate (indexing overhead)
Query (filtered)	Sub-second	Sub-second	Sub-second
Query (full scan)	Depends on bloom filter selectivity	Depends on label cardinality	Fast (full-text index)
Full-text search	✅ (via bloom filters)	❌ (label-first)	✅ (inverted index)

Key advantage: VictoriaLogs allows free-text search WITHOUT mandatory label selectors, unlike Loki which requires {label="value"} first.

VictoriaTraces Benchmarks¶

Metric	VictoriaTraces	Tempo	Jaeger + ES
Storage	Local disk (columnar + bloom)	Object storage (Parquet)	Elasticsearch cluster
External deps	None	S3/GCS required	ES cluster required
RAM usage	Low	Low	High
Trace ID lookup	Fast	< 200ms	Fast
Search	Bloom-filter-based	Parquet column scan	Full-text index
Operational overhead	Very low (single binary)	Low-moderate	High

Cost Comparison: Victoria Stack vs LGTM vs Datadog¶

At 1M Active Series + 100 GB/day Logs + 50M Spans/day¶

Stack	Est. Monthly Cost	RAM Footprint	Disk Footprint	Ops Burden
Victoria Stack (self-hosted)	$500–1,500	Lowest	Lowest	Low
Self-hosted LGTM	$1,000–3,000	Medium	Medium	High
Grafana Cloud Pro	$1,500–4,000	N/A (managed)	N/A	Low
VM Cloud (single-node)	$225–1,300	N/A (managed)	N/A	Very Low
Datadog	$5,000–17,000	N/A (SaaS)	N/A	Very Low

Victoria Stack is typically 2–3x cheaper than LGTM due to lower resource footprint and no object storage costs.

Caveats¶

Benchmarks heavily depend on "data shape" — active series churn rate, label cardinality, and query density
VictoriaMetrics' RAM savings are most dramatic at high cardinality (> 1M active series)
VictoriaLogs bloom-filter approach trades slightly higher CPU for dramatically lower RAM vs inverted indexes
VictoriaTraces is newer and has fewer large-scale public benchmarks than Tempo
Local SSD storage means you trade object storage durability for latency — use vmbackup to S3 for DR
Run prometheus-benchmark on YOUR workload before making migration decisions

Sources¶

URL	Source Kind	Authority	Date
https://docs.victoriametrics.com/	docs	primary	2026-04-10
https://github.com/VictoriaMetrics/prometheus-benchmark	tool	primary	2026-04-10
https://victoriametrics.com/case-studies/	case study	primary	2026-04-10

Architecture¶

1. Default Topology / Flow¶

Deployment Modes¶

Single-Node vs Cluster¶

Component Roles¶

Full Stack Architecture¶

vmalert Evaluation Flow¶

Multi-Source Log Ingestion¶

Storage Layout¶

VictoriaMetrics (Metrics)¶

VictoriaLogs (Logs)¶

VictoriaTraces (Traces)¶

Kubernetes Deployment Matrix¶

Key Design Decisions¶

Related Notes¶

Data Model¶

1. Default Topology / Flow¶

How It Works¶

Core Mechanisms¶

Storage Engine¶

VictoriaMetrics (Metrics)¶

VictoriaLogs (Logs)¶

VictoriaTraces (Traces)¶

Query Languages¶

MetricsQL (Metrics)¶

LogsQL (Logs)¶

Data Flow¶

Pull Sequence¶

Write Sequence¶

Merge Sequence¶

Read Sequence¶

Related Notes¶

Benchmarks¶

Test Conditions¶

VictoriaMetrics vs Competitors (Metrics)¶

Resource Efficiency¶

Key Findings¶

Production Scale Records¶

VictoriaLogs Benchmarks¶

VictoriaTraces Benchmarks¶

Cost Comparison: Victoria Stack vs LGTM vs Datadog¶

At 1M Active Series + 100 GB/day Logs + 50M Spans/day¶

Caveats¶

Sources¶

Related Notes¶