How-to Guides

Task recipes for the Victoria Stack: deployment, configuration, security setup, troubleshooting, and the Commands & Recipes library. See also: hub, Explanation, Reference.

Deployment & Typical Setup

Single-Node (Simplest Production Path)

# VictoriaMetrics — single binary, metrics
./victoria-metrics -storageDataPath=/data/vm -retentionPeriod=12

# VictoriaLogs — single binary, logs
./victoria-logs -storageDataPath=/data/vl -retentionPeriod=30d

# VictoriaTraces — single binary, traces
./victoria-traces -storageDataPath=/data/vt

Each binary starts an HTTP server and is immediately ready to receive data. Basic usage needs no configuration files.

Kubernetes (vmoperator)

The recommended production path uses the vmoperator with CRDs:

# Install the operator
helm repo add vm https://victoriametrics.github.io/helm-charts/
helm repo update
helm install vmoperator vm/victoria-metrics-operator -n monitoring --create-namespace

# Deploy cluster via CRD
kubectl apply -f - <<EOF
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMCluster
metadata:
  name: vm-cluster
spec:
  retentionPeriod: "12"
  replicationFactor: 2
  vminsert:
    replicaCount: 2
    resources:
      requests: { cpu: "500m", memory: "512Mi" }
  vmselect:
    replicaCount: 2
    resources:
      requests: { cpu: "500m", memory: "1Gi" }
  vmstorage:
    replicaCount: 3
    storageDataPath: /vm-data
    resources:
      requests: { cpu: "1", memory: "4Gi" }
    storage:
      volumeClaimTemplate:
        spec:
          resources:
            requests: { storage: 100Gi }
          storageClassName: fast-ssd
EOF

Production Readiness Checklist

• VictoriaMetrics deployed (single-node or cluster)
• VictoriaLogs deployed for log aggregation
• VictoriaTraces deployed for distributed tracing
• vmauth configured as routing proxy (with auth)
• vmagent deployed as DaemonSet for metric scraping
• vmalert configured with recording rules and alerts
• vmbackup scheduled for automated snapshots to S3/GCS
• Grafana configured with Prometheus, Loki, and Jaeger data sources
• Resource requests/limits set on all pods
• SSD-backed storage for all stateful components
• Monitoring the monitoring (self-scrape)

Configuration & Optimal Tuning

vmauth Routing Configuration

The single most important config file — it routes traffic across all three databases:

# vmauth-config.yaml
unauthorized_user:
  url_map:
    # === METRICS ===
    - src_paths:
        - "/api/v1/write"
        - "/api/v1/import.*"
      url_prefix: "http://vminsert:8480/insert/0/prometheus"
    - src_paths:
        - "/api/v1/query.*"
        - "/api/v1/series.*"
        - "/api/v1/labels.*"
      url_prefix: "http://vmselect:8481/select/0/prometheus"

    # === LOGS ===
    - src_paths:
        - "/insert/jsonline.*"
        - "/insert/elasticsearch.*"
        - "/loki/api/v1/push"
      url_prefix: "http://victorialogs:9428"
    - src_paths:
        - "/select/logsql/.*"
      url_prefix: "http://victorialogs:9428"

    # === TRACES ===
    - src_paths:
        - "/insert/opentelemetry/.*"
      url_prefix: "http://victoriatraces:10428"
    - src_paths:
        - "/api/traces.*"
        - "/api/services.*"
      url_prefix: "http://victoriatraces:10428"

For the flag reference, see Critical Tuning Flags.

Security Setup

vmauth — Authentication Proxy

vmauth is the primary authentication and routing component for VictoriaMetrics deployments. It sits in front of vminsert and vmselect, authenticating requests and routing them to the correct tenant.

Token-Based Authentication

The simplest auth method. Each user is assigned a bearer_token or username/password pair. vmauth maps authenticated users to backend URLs and tenant paths.

# vmauth configuration
users:
  - username: "team-alpha"
    password: "secure-token-alpha"
    url_prefix:
      - "http://vminsert:8480/insert/1/prometheus/"
      - "http://vmselect:8481/select/1/prometheus/"

  - username: "team-beta"
    password: "secure-token-beta"
    url_prefix:
      - "http://vminsert:8480/insert/2/prometheus/"
      - "http://vmselect:8481/select/2/prometheus/"

Token Isolation

Use distinct authentication tokens for every tenant. If a single token is compromised, the blast radius is limited to that tenant's data only. Never share tokens across teams or services.

URL Map Routing

For fine-grained control, use url_map to route different API paths to different backends:

users:
  - username: "writer-only"
    password: "write-token"
    url_map:
      - src_paths: ["/api/v1/write"]
        url_prefix: "http://vminsert:8480/insert/1/prometheus/"
      - src_paths: ["/api/v1/query", "/api/v1/query_range"]
        url_prefix: "http://vmselect:8481/select/1/prometheus/"

This pattern allows: - Write-only service accounts (no query access). - Read-only Grafana connections (no write access). - Different tenants for different API endpoints.

JWT Authentication

vmauth validates JWT tokens using configured public keys. Claims from the JWT are available as placeholders in URL routing:

users:
  - jwt:
      public_keys:
        - |
          -----BEGIN PUBLIC KEY-----
          MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA...
          -----END PUBLIC KEY-----
    url_map:
      - src_paths: ["/api/v1/write"]
        url_prefix: "http://vminsert:8480/insert/{{.MetricsTenant}}/prometheus/"
      - src_paths: ["/api/v1/query", "/api/v1/query_range"]
        url_prefix: "http://vmselect:8481/select/{{.MetricsTenant}}/prometheus/"

JWT claims available as template variables: - {{.MetricsTenant}} — tenant ID for cluster mode - {{.MetricsExtraLabels}} — injected label filters - {{.MetricsExtraFilters}} — query-time filter expressions - {{.LogsAccountID}}, {{.LogsProjectID}} — VictoriaLogs tenant fields

mTLS-Based Routing

vmauth routes requests based on client certificate fields:

users:
  - mtls:
      organizational_unit: finance
    url_prefix: "http://victoriametrics-finance:8428"
  - mtls:
      organizational_unit: devops
    url_prefix: "http://victoriametrics-devops:8428"

mTLS fields available for routing: organizational_unit, organization, common_name.

vmgateway — Enterprise Authentication

vmgateway provides OIDC-based multi-tenant access. It validates JWT tokens from an OIDC provider (for example, Keycloak, Okta) and extracts the vm_access claim to determine tenant routing.

./vmgateway \
    -licenseFile=./vm-license.key \
    -enable.auth=true \
    -clusterMode=true \
    -write.url=http://localhost:8480 \
    -read.url=http://localhost:8481

The vm_access JWT payload routes to the correct cluster tenant, appends extra_labels to ingested metrics, applies extra_filters at query time, and uses mode as a bitfield controlling read (1) and write (2) access:

{
  "exp": 1617304574,
  "vm_access": {
    "tenant_id": {
      "account_id": 1,
      "project_id": 5
    },
    "extra_labels": {
      "team": "dev",
      "project": "mobile"
    },
    "extra_filters": ["{env=~\"prod|dev\",team!=\"test\"}"],
    "mode": 1
  }
}

TLS Configuration

vmauth Automatic TLS

vmauth Enterprise supports automatic TLS certificate provisioning via Let's Encrypt:

./vmauth \
    -httpListenAddr=:443 \
    -tls=true \
    -tlsAutocertHosts=metrics.example.com \
    -tlsAutocertEmail=[email protected] \
    -tlsAutocertCacheDir=/var/cache/vmauth/tls

Manual TLS

For non-Enterprise or custom certificate deployments:

./vmauth \
    -httpListenAddr=:443 \
    -tls=true \
    -tlsCertFile=/etc/vmauth/certs/server.crt \
    -tlsKeyFile=/etc/vmauth/certs/server.key

Backend TLS

Connections from vmauth to backend components can also use TLS:

users:
  - username: "secure-client"
    password: "token"
    url_prefix:
      - "https://vminsert.internal:8480/insert/1/prometheus/"
      - "https://vmselect.internal:8481/select/1/prometheus/"
    tls_insecure_skip_verify: false

Kubernetes Operator Security

The VictoriaMetrics Operator supports a useStrictSecurity flag that enforces security hardening:

• Runs all pods as non-root user.
• Drops all Linux capabilities.
• Sets a read-only root filesystem.
• Restricts discretionary access control.

apiVersion: operator.victoriametrics.com/v1beta1
kind: VMSingle
metadata:
  name: example
spec:
  useStrictSecurity: true

Internal Metrics Protection

The -metricsAuthKey flag protects internal metrics endpoints (/metrics, /debug/pprof):

./vminsert -metricsAuthKey="secret-key-for-metrics"

Requests to metrics endpoints must include the key as a BasicAuth password. This prevents unauthorized access to diagnostic information.

VictoriaLogs Tenant Isolation

Configure vmauth to inject the AccountID and ProjectID headers per team:

# vmauth for VictoriaLogs
users:
  - username: "team-a"
    password: "token-a"
    url_map:
      - src_paths: ["/select/.*", "/insert/.*"]
        headers:
          - "AccountID: 1"
          - "ProjectID: 0"
        url_prefix: ["http://vlselect:9428/"]
  - username: "team-b"
    password: "token-b"
    url_map:
      - src_paths: ["/select/.*"]
        headers:
          - "AccountID: 2"
          - "ProjectID: 0"
        url_prefix: ["http://vlselect:9428/"]

Network Security Best Practices

1. Never expose ingestion nodes to the internet — always put vmauth or NGINX in front
2. Use Kubernetes NetworkPolicies to restrict pod-to-pod communication
3. Expose only vmauth externally
4. Use mTLS between components in sensitive environments
5. Cluster multi-tenancy: account IDs in URL paths (/insert/TENANT_ID/) isolate data

For the full security model, see Security Architecture Overview and the Hardening Checklist.

Best Practices

Metrics

1. Global Relabeling: Append datacenter/environment labels at the vmagent layer before data hits storage
2. Drop high-cardinality labels: Use vmagent relabeling to drop labels like pod_ip, request_id before ingestion
3. Recording rules: Precompute expensive MetricsQL expressions via vmalert
4. Deduplication: With replication, always set -dedup.minScrapeInterval on vmselect

Logs

1. Do Not Translate: Use native APIs whenever possible — point Fluent Bit directly to /insert/jsonline rather than going through an intermediary
2. Structured logging: Use JSON logs to enable field extraction at query time
3. Stream fields: Set _stream_fields on ingestion to logically group related log entries
4. Retention per signal: Set different retention periods for logs (30d) vs metrics (12mo) vs traces (14d)

Operations

1. Monitor with itself: Scrape VictoriaMetrics' own /metrics endpoint
2. Use vmbackup regularly: Schedule daily incremental backups to S3
3. Test upgrades on LTS: Use the LTS release line for production stability

Troubleshooting

Common Issues & Playbook

Symptom	Likely Cause	Fix
High CPU on vmstorage during queries	Large time-window queries	Limit -search.maxQueryDuration, scale vmselect
OOM on vmstorage	High cardinality churn	Tune -memory.allowedPercent, drop unused labels at vmagent
"too many unique timeseries"	Query returns too many series	Increase -search.maxUniqueTimeseries or refine query
Slow VictoriaLogs queries	Large time range without filters	Add time restrictions (_time:1h), use specific filters
vmagent not discovering targets	ServiceMonitor/PodScrape CRDs not picked up	Make sure that the vmoperator is running, examine the CRD labels
VictoriaTraces not receiving spans	OTLP gRPC not enabled	Explicitly enable gRPC port in config
Data gap after vmstorage restart	WAL not flushed	Normal — WAL replays on restart, gap is temporary

Commands & Recipes

Installation

Docker (Quick Start — All Components)

# VictoriaMetrics (metrics)
docker run -d --name vm \
  -p 8428:8428 \
  -v vm-data:/storage \
  victoriametrics/victoria-metrics \
  -storageDataPath=/storage -retentionPeriod=12

# VictoriaLogs (logs)
docker run -d --name vl \
  -p 9428:9428 \
  -v vl-data:/vlogs \
  victoriametrics/victoria-logs \
  -storageDataPath=/vlogs -retentionPeriod=30d

# VictoriaTraces (traces)
docker run -d --name vt \
  -p 10428:10428 \
  -p 4317:4317 \
  -v vt-data:/vtraces \
  victoriametrics/victoria-traces \
  -storageDataPath=/vtraces

Docker Compose (Full Stack)

# docker-compose.yaml — Full Victoria stack for development
version: '3.8'
services:
  victoriametrics:
    image: victoriametrics/victoria-metrics:latest
    ports: ["8428:8428"]
    volumes: ["vm-data:/storage"]
    command:
      - "-storageDataPath=/storage"
      - "-retentionPeriod=12"

  victorialogs:
    image: victoriametrics/victoria-logs:latest
    ports: ["9428:9428"]
    volumes: ["vl-data:/vlogs"]
    command:
      - "-storageDataPath=/vlogs"
      - "-retentionPeriod=30d"

  victoriatraces:
    image: victoriametrics/victoria-traces:latest
    ports:
      - "10428:10428"  # HTTP
      - "4317:4317"    # OTLP gRPC
    volumes: ["vt-data:/vtraces"]
    command:
      - "-storageDataPath=/vtraces"

  vmagent:
    image: victoriametrics/vmagent:latest
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    command:
      - "-promscrape.config=/etc/prometheus/prometheus.yml"
      - "-remoteWrite.url=http://victoriametrics:8428/api/v1/write"

  vmauth:
    image: victoriametrics/vmauth:latest
    ports: ["8427:8427"]
    volumes:
      - ./vmauth-config.yml:/etc/vmauth/config.yml
    command:
      - "-auth.config=/etc/vmauth/config.yml"

  vmalert:
    image: victoriametrics/vmalert:latest
    volumes:
      - ./alert-rules.yml:/etc/rules/rules.yml
    command:
      - "-rule=/etc/rules/*.yml"
      - "-datasource.url=http://victoriametrics:8428"
      - "-remoteWrite.url=http://victoriametrics:8428"

  grafana:
    image: grafana/grafana-oss:latest
    ports: ["3000:3000"]
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin

volumes:
  vm-data:
  vl-data:
  vt-data:

Helm (Kubernetes)

helm repo add vm https://victoriametrics.github.io/helm-charts/
helm repo update

# Single-node VictoriaMetrics
helm install vm vm/victoria-metrics-single -n monitoring --create-namespace

# Cluster VictoriaMetrics
helm install vm-cluster vm/victoria-metrics-cluster -n monitoring -f vm-values.yaml

# vmoperator (manages all components via CRDs)
helm install vmoperator vm/victoria-metrics-operator -n monitoring

# vmagent
helm install vmagent vm/victoria-metrics-agent -n monitoring

# vmalert
helm install vmalert vm/victoria-metrics-alert -n monitoring

# VictoriaLogs (single-node)
helm install vl vm/victoria-logs-single -n monitoring

vmagent Recipes

# Start vmagent as drop-in Prometheus replacement
./vmagent \
  -promscrape.config=/path/to/prometheus.yml \
  -remoteWrite.url=http://victoriametrics:8428/api/v1/write

# Add global labels to all scraped metrics
./vmagent \
  -remoteWrite.label=datacenter=us-east-1 \
  -remoteWrite.label=env=production \
  -promscrape.config=prometheus.yml \
  -remoteWrite.url=http://vminsert:8480/insert/0/prometheus/api/v1/write

# Multi-destination remote write (fan-out)
./vmagent \
  -remoteWrite.url=http://vm-primary:8428/api/v1/write \
  -remoteWrite.url=http://vm-secondary:8428/api/v1/write

Data Ingestion Recipes

Fluent Bit → VictoriaLogs

# fluent-bit.conf — Push logs directly to VictoriaLogs
[OUTPUT]
    Name  http
    Match *
    Host  victorialogs
    Port  9428
    URI   /insert/jsonline?_stream_fields=stream&_msg_field=log&_time_field=date
    Format json_lines
    Compress gzip

OpenTelemetry Collector → VictoriaTraces

# otel-collector-config.yaml
exporters:
  otlp/victoriatraces:
    endpoint: "victoriatraces:4317"
    tls:
      insecure: true

  prometheusremotewrite/vm:
    endpoint: "http://victoriametrics:8428/api/v1/write"

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/victoriatraces]
    metrics:
      receivers: [otlp, prometheus]
      processors: [batch]
      exporters: [prometheusremotewrite/vm]

Promtail / Loki Push → VictoriaLogs

# promtail-config.yaml — VictoriaLogs accepts Loki push API
clients:
  - url: http://victorialogs:9428/insert/loki/api/v1/push

Direct OTLP → VictoriaTraces

• HTTP: http://victoriatraces:10428/insert/opentelemetry/v1/traces
• gRPC: grpc://victoriatraces:4317

vmauth Routing Config

For the full url_map with commentary, see vmauth Routing Configuration in the Configuration section.

Backup & Restore

# Create instant snapshot (single-node)
curl http://victoriametrics:8428/snapshot/create
# Returns: {"status":"ok","snapshot":"20260410120000-..."}

# Backup snapshot to S3
./vmbackup \
  -storageDataPath=/data/vm \
  -snapshot.createURL=http://localhost:8428/snapshot/create \
  -dst=s3://my-bucket/vm-backups/

# Incremental backup (only new data since last backup)
./vmbackup \
  -storageDataPath=/data/vm \
  -snapshot.createURL=http://localhost:8428/snapshot/create \
  -dst=s3://my-bucket/vm-backups/ \
  -origin=s3://my-bucket/vm-backups/  # previous backup path

# Restore from backup
./vmrestore \
  -src=s3://my-bucket/vm-backups/latest \
  -storageDataPath=/data/vm-restored

Note: For clustered setup, vmbackup must be executed on EVERY vmstorage node.

API Recipes

# Query VictoriaMetrics (PromQL/MetricsQL)
curl -s "http://vm:8428/api/v1/query?query=up" | jq .

# Range query
curl -s "http://vm:8428/api/v1/query_range?query=rate(http_requests_total[5m])&start=-1h&step=60s" | jq .

# Import data via JSON
curl -d '{"metric":{"__name__":"test","job":"api"},"values":[1,2,3],"timestamps":[1617000000000,1617000001000,1617000002000]}' \
  http://vm:8428/api/v1/import

# Query VictoriaLogs (LogsQL)
curl -s "http://vl:9428/select/logsql/query?query=_time:5m+AND+error" | jq .

# Push a test log
curl -X POST "http://vl:9428/insert/jsonline?_stream_fields=app&_msg_field=msg" \
  -d '{"app":"test","msg":"hello from curl","level":"info"}'

# Look up a trace by ID (Jaeger API)
curl -s "http://vt:10428/api/traces/abc123" | jq .

# Check health
curl -s "http://vm:8428/-/healthy" && echo "OK"

Grafana Data Source Config

# Grafana provisioning for Victoria Stack
apiVersion: 1
datasources:
  - name: VictoriaMetrics
    type: prometheus
    url: http://vmauth:8427
    isDefault: true
    jsonData:
      httpMethod: POST

  - name: VictoriaLogs
    type: victoriametrics-logs-datasource
    url: http://vmauth:8427

  - name: VictoriaTraces
    type: jaeger
    url: http://vmauth:8427

Related Notes

• Victoria Stack Hub
• Explanation
• Reference