Reference
Facts and figures for the Victoria Stack: deployment matrices, query languages, benchmarks, tuning flags, cost, and hardening checklists. See also: hub, Explanation, How-to guides.
Kubernetes Deployment Matrix
| Component |
Kind |
Replicas (Min HA) |
Key Resource |
Helm Chart |
| vmagent |
DaemonSet or Deployment |
1 per node (DS) or 2+ |
CPU, Memory |
victoria-metrics-agent |
| vmauth |
Deployment |
2+ |
CPU |
victoria-metrics-auth |
| vminsert |
Deployment |
2+ |
CPU |
victoria-metrics-cluster |
| vmselect |
Deployment |
2+ |
CPU, Memory |
victoria-metrics-cluster |
| vmstorage |
StatefulSet |
3+ |
Disk IOPS, Memory |
victoria-metrics-cluster |
| VictoriaLogs |
StatefulSet (single-node) or cluster |
1–3 |
Disk, Memory |
victoria-logs-single |
| VictoriaTraces |
StatefulSet (single-node) |
1 |
Disk, Memory |
— |
| vmalert |
Deployment |
1–2 |
CPU |
victoria-metrics-alert |
| vmoperator |
Deployment |
1 |
CPU |
victoria-metrics-operator |
Data Model
erDiagram
Victoriametrics_CORE ||--o{ CONFIG : requires
Victoriametrics_CORE ||--o{ STATE : writes
CONFIG {
string runtime_params
string limits
}
STATE {
string metric_id
json payload
}
Query Languages
MetricsQL (Metrics)
MetricsQL is a backward-compatible superset of PromQL with extensions that fix common PromQL pain points:
| Feature |
PromQL |
MetricsQL |
| Auto lookbehind window |
❌ Required |
✅ Auto-calculated from step |
rate() extrapolation |
Yes (causes fractional integers) |
No (returns accurate integers) |
keep_metric_names |
❌ |
✅ Preserves metric names |
| Numeric suffixes (Ki, Mi, Gi) |
❌ |
✅ 8Ki = 8 * 1024 |
| NaN handling |
Returns NaN |
Automatically removes NaN |
aggr_over_time() |
❌ |
✅ Multiple aggregates in one pass |
| Graphite filter syntax |
❌ |
✅ {__graphite__="foo.*.bar"} |
# MetricsQL example: rate without explicit window (auto-calculated)
rate(http_requests_total{job="api"})
# Keep metric names when applying functions
rate({__name__=~"foo|bar"}) keep_metric_names
# Use numeric suffixes
process_memory_bytes > 2Gi
LogsQL (Logs)
LogsQL is purpose-built for VictoriaLogs with a pipe-based syntax:
# Filter error logs from last 5 minutes
_time:5m AND level:="error"
# Full-text search with pipe transformations
error connection refused | stats count() by (host)
# Extract fields at query time
_time:1h | extract "status=<status_code>" | stats count() by (status_code)
# JSON log parsing
_time:5m | unpack_json | level:="error" | fields _time, msg, trace_id
Key difference from LogQL (Loki): LogsQL does NOT require a mandatory label selector. You can search across all log streams with free-text queries, while Loki requires {label="value"} first.
Benchmarks
Test Conditions
- VictoriaMetrics versions: v1.136.x (stable), v1.122.x (LTS)
- Date: April 2026
- Note: VictoriaMetrics publishes a community benchmarking tool: prometheus-benchmark. Results below combine official benchmarks and community reports.
Resource Efficiency
| Metric |
VictoriaMetrics |
Prometheus |
Thanos |
Mimir |
| RAM usage (1M active series) |
~2 GB |
~10–20 GB |
~8–15 GB |
~8–12 GB |
| Disk usage (compression) |
0.4–1.0 bytes/sample |
1.3 bytes/sample |
1.5 bytes/sample |
1.2–1.5 bytes/sample |
| Ingestion (single-node) |
~1M samples/sec |
~500k samples/sec |
N/A (sidecar) |
N/A (distributed) |
| Ingestion (cluster) |
100M+ samples/sec |
N/A |
~50M+ |
30M+ |
| Query latency (simple PromQL) |
< 100ms |
< 100ms (local) |
Variable (ObjStore) |
< 200ms |
Key Findings
- RAM: VictoriaMetrics consistently uses 5–10x less RAM than Prometheus for the same workload due to aggressive memory management and ZSTD compression
- Disk: ~50% less disk than Prometheus, due to specialized ZSTD + delta-encoding for floats/timestamps
- Cardinality resilience: VM handles high-cardinality spikes via
-memory.allowedPercent, while Prometheus OOM-kills
Production Scale Records
| Company |
Scale |
Notes |
| Roblox |
Billions of active series |
100% uptime across quarters |
| DreamHost |
76M active series |
80% memory reduction vs previous stack |
| Grammarly |
Large-scale metrics |
10x cost reduction |
| CERN |
CMS detector monitoring |
Long-term Prometheus storage |
VictoriaLogs Benchmarks
| Metric |
VictoriaLogs |
Loki |
Elasticsearch |
| Storage efficiency |
10–30x less disk than ES |
10–20x less than ES |
Baseline |
| RAM usage |
Minimal (bloom filters, no inverted index) |
Low (label-only index) |
High (inverted index) |
| Ingestion |
Very high (columnar, batch writes) |
High |
Moderate (indexing overhead) |
| Query (filtered) |
Sub-second |
Sub-second |
Sub-second |
| Query (full scan) |
Depends on bloom filter selectivity |
Depends on label cardinality |
Fast (full-text index) |
| Full-text search |
✅ (via bloom filters) |
❌ (label-first) |
✅ (inverted index) |
Key advantage: VictoriaLogs allows free-text search WITHOUT mandatory label selectors, unlike Loki which requires {label="value"} first.
VictoriaTraces Benchmarks
| Metric |
VictoriaTraces |
Tempo |
Jaeger + ES |
| Storage |
Local disk (columnar + bloom) |
Object storage (Parquet) |
Elasticsearch cluster |
| External deps |
None |
S3/GCS required |
ES cluster required |
| RAM usage |
Low |
Low |
High |
| Trace ID lookup |
Fast |
< 200ms |
Fast |
| Search |
Bloom-filter-based |
Parquet column scan |
Full-text index |
| Operational overhead |
Very low (single binary) |
Low-moderate |
High |
Cost Comparison: Victoria Stack vs LGTM vs Datadog
At 1M active series, 100 GB/day logs, and 50M spans/day:
| Stack |
Est. Monthly Cost |
RAM Footprint |
Disk Footprint |
Ops Burden |
| Victoria Stack (self-hosted) |
$500–1,500 |
Lowest |
Lowest |
Low |
| Self-hosted LGTM |
$1,000–3,000 |
Medium |
Medium |
High |
| Grafana Cloud Pro |
$1,500–4,000 |
N/A (managed) |
N/A |
Low |
| VM Cloud (single-node) |
$225–1,300 |
N/A (managed) |
N/A |
Very Low |
| Datadog |
$5,000–17,000 |
N/A (SaaS) |
N/A |
Very Low |
Victoria Stack is typically 2–3x cheaper than LGTM due to lower resource footprint and no object storage costs.
Caveats
- Benchmarks heavily depend on "data shape" — active series churn rate, label cardinality, and query density
- VictoriaMetrics' RAM savings are most dramatic at high cardinality (> 1M active series)
- VictoriaLogs bloom-filter approach trades slightly higher CPU for dramatically lower RAM vs inverted indexes
- VictoriaTraces is newer and has fewer large-scale public benchmarks than Tempo
- Local SSD storage means you trade object storage durability for latency — use vmbackup to S3 for DR
- Run
prometheus-benchmark on YOUR workload before making migration decisions
Critical Tuning Flags
| Component |
Flag |
Purpose |
Default |
| All |
-retentionPeriod |
Data retention duration |
1 month |
| vmstorage |
-search.maxUniqueTimeseries |
Prevent OOM on high-cardinality queries |
300,000 |
| vmstorage |
-memory.allowedPercent |
Max RAM usage percent before aggressive GC |
60% |
| vmstorage |
-search.maxQueryDuration |
Max single query execution time |
30s |
| vminsert |
-replicationFactor=N |
Replicate data to N storage nodes |
1 |
| vmselect |
-dedup.minScrapeInterval |
Deduplicate data when RF > 1 |
0s |
| vmagent |
-remoteWrite.label |
Add global labels to all scraped metrics |
— |
| VictoriaLogs |
-retentionPeriod |
Log retention |
7d |
Scaling Decision Matrix
| Symptom |
Component to Scale |
How |
| Slow metric queries |
vmselect |
Add replicas |
| Write backpressure |
vminsert |
Add replicas |
| Disk full on metrics |
vmstorage |
Add nodes or increase disk |
| High RAM on storage |
vmstorage |
Increase -memory.allowedPercent, reduce cardinality |
| Slow log search |
VictoriaLogs |
Add CPU/RAM (single-node) or cluster |
| Log ingestion lag |
VictoriaLogs |
Increase resources or switch to cluster |
High Availability
| Mechanism |
Implementation |
| Metrics replication |
-replicationFactor=2 on vminsert + -dedup.minScrapeInterval on vmselect |
| Metrics availability |
If 1 vmstorage fails with RF=2, vmselect returns partial results transparently |
| Logs/Traces HA |
Deploy cluster mode with vlinsert/vlstorage/vlselect |
| Proxy HA |
Multiple vmauth replicas behind load balancer |
| Backup |
vmbackup creates instant, consistent snapshots without locking the DB |
Cost
Cost Drivers
| Factor |
Driver |
Optimization |
| Compute |
Insert + select pods |
Right-size, use spot nodes for vmselect |
| Storage |
Data volume × retention |
ZSTD compression reduces 2–7x naturally, tune retention |
| Network |
Internal cluster traffic |
Co-locate in same AZ |
| NO object storage |
Local SSD only |
Eliminates S3/GCS egress costs entirely |
Cost at Scale (Self-Hosted)
| Scale |
Active Series |
Logs (GB/day) |
Estimated Monthly |
| Small |
100k |
10 |
$100–300 |
| Medium |
1M |
100 |
$500–1,500 |
| Large |
10M |
1 TB |
$2,000–8,000 |
| Enterprise |
100M+ |
10 TB+ |
$10,000–50,000 |
VictoriaMetrics Cloud Pricing
| Tier |
Starting Cost |
Includes |
| Single-node |
~$225/mo |
Up to 500k active series, 1-month retention |
| Cluster |
~$1,300/mo |
Multi-tenancy, HA, advanced networking |
Key Self-Monitoring Metrics
| Metric |
What It Tells You |
vm_rows_inserted_total |
Ingestion throughput |
vm_active_timeseries |
Current cardinality |
vm_slow_queries_total |
Queries exceeding duration threshold |
vm_cache_entries |
Cache utilization |
vm_data_size_bytes |
On-disk data size |
process_resident_memory_bytes |
Actual RAM usage |
vm_merge_duration_seconds |
Background compaction health |
API Endpoint Access Control
vmauth and vmgateway must restrict which API endpoints are accessible per user. This minimizes the attack surface:
| Endpoint |
Risk |
Recommendation |
/api/v1/write |
Data injection |
Restrict to vmagent/service accounts |
/api/v1/query |
Data exposure |
Restrict to Grafana/humans |
/api/v1/export |
Bulk data export |
Restrict heavily |
/metrics |
Internal diagnostics |
Protect with -metricsAuthKey |
/debug/* |
Pprof, vars |
Block at network level |
/snapshot* |
Storage access |
Block entirely in production |
/force/merge* |
Performance impact |
Block at network level |
Hardening Checklist
| Area |
Recommendation |
| Network isolation |
All backends in private subnet. vmauth is the only ingress |
| TLS |
HTTPS between clients and vmauth. Optional mTLS to backends |
| Auth tokens |
Unique token per tenant. Rotate on compromise |
| Endpoint restriction |
Allow only necessary API paths per user in vmauth config |
| Kubernetes |
Enable useStrictSecurity in the Operator |
| Metrics protection |
Set -metricsAuthKey on all components |
| Debug endpoints |
Block /debug/* and /snapshot* at the network level |
| Secret management |
Store vmauth config passwords in Vault or sealed secrets |
| Audit |
Enable access logging on vmauth for forensic analysis |
Sources