Security

Security model for VictoriaMetrics covering vmauth proxy authentication, cluster security, TLS, tenant data isolation, and auth tokens. See also: observability/victoriametrics/index, observability/victoriametrics/architecture, observability/victoriametrics/operations.

Security Architecture Overview

VictoriaMetrics does not include built-in authentication or authorization in its open-source components. All cluster components (vminsert, vmselect, vmstorage) must operate within a protected private network with no direct internet exposure. External access is mediated exclusively through authentication proxies: vmauth (open-source) or vmgateway (Enterprise).

flowchart TD
    subgraph "External"
        Clients[API Clients<br/>Grafana / vmagent]
        Users[Browser Users]
    end

    subgraph "Auth Proxy Layer"
        Vmauth[vmauth<br/>Token / Basic Auth / JWT / mTLS]
        Vmgateway[vmgateway Enterprise<br/>OIDC / JWT Claims]
    end

    subgraph "VictoriaMetrics Cluster"
        Vminsert[vminsert<br/>:8480]
        Vmselect[vmselect<br/>:8481]
        Vmstorage[vmstorage<br/>:8482]
    end

    Clients -->|Bearer Token / Basic Auth| Vmauth
    Users -->|OIDC JWT| Vmgateway
    Vmauth -->|Route by token -> tenant| Vminsert
    Vmauth -->|Route by token -> tenant| Vmselect
    Vmgateway -->|Parse JWT vm_access claims| Vminsert
    Vmgateway -->|Parse JWT vm_access claims| Vmselect
    Vminsert --> Vmstorage
    Vmselect --> Vmstorage

vmauth — Authentication Proxy

vmauth is the primary authentication and routing component for VictoriaMetrics deployments. It sits in front of vminsert and vmselect, authenticating requests and routing them to the correct tenant.

Token-Based Authentication

The simplest auth method. Each user is assigned a bearer_token or username/password pair. vmauth maps authenticated users to backend URLs and tenant paths.

# vmauth configuration
users:
  - username: "team-alpha"
    password: "secure-token-alpha"
    url_prefix:
      - "http://vminsert:8480/insert/1/prometheus/"
      - "http://vmselect:8481/select/1/prometheus/"

  - username: "team-beta"
    password: "secure-token-beta"
    url_prefix:
      - "http://vminsert:8480/insert/2/prometheus/"
      - "http://vmselect:8481/select/2/prometheus/"

Token Isolation

Use distinct authentication tokens for every tenant. If a single token is compromised, the blast radius is limited to that tenant's data only. Never share tokens across teams or services.

URL Map Routing

For fine-grained control, use url_map to route different API paths to different backends:

users:
  - username: "writer-only"
    password: "write-token"
    url_map:
      - src_paths: ["/api/v1/write"]
        url_prefix: "http://vminsert:8480/insert/1/prometheus/"
      - src_paths: ["/api/v1/query", "/api/v1/query_range"]
        url_prefix: "http://vmselect:8481/select/1/prometheus/"

This pattern allows: - Write-only service accounts (no query access). - Read-only Grafana connections (no write access). - Different tenants for different API endpoints.

JWT Authentication

vmauth validates JWT tokens using configured public keys. Claims from the JWT are available as placeholders in URL routing:

users:
  - jwt:
      public_keys:
        - |
          -----BEGIN PUBLIC KEY-----
          MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA...
          -----END PUBLIC KEY-----
    url_map:
      - src_paths: ["/api/v1/write"]
        url_prefix: "http://vminsert:8480/insert/{{.MetricsTenant}}/prometheus/"
      - src_paths: ["/api/v1/query", "/api/v1/query_range"]
        url_prefix: "http://vmselect:8481/select/{{.MetricsTenant}}/prometheus/"

JWT claims available as template variables: - {{.MetricsTenant}} — tenant ID for cluster mode - {{.MetricsExtraLabels}} — injected label filters - {{.MetricsExtraFilters}} — query-time filter expressions - {{.LogsAccountID}}, {{.LogsProjectID}} — VictoriaLogs tenant fields

mTLS-Based Routing

vmauth routes requests based on client certificate fields:

users:
  - mtls:
      organizational_unit: finance
    url_prefix: "http://victoriametrics-finance:8428"
  - mtls:
      organizational_unit: devops
    url_prefix: "http://victoriametrics-devops:8428"

mTLS fields available for routing: organizational_unit, organization, common_name.

vmgateway — Enterprise Authentication

vmgateway provides OIDC-based multi-tenant access. It validates JWT tokens from an OIDC provider (e.g., Keycloak, Okta) and extracts the vm_access claim to determine tenant routing.

./vmgateway \
    -licenseFile=./vm-license.key \
    -enable.auth=true \
    -clusterMode=true \
    -write.url=http://localhost:8480 \
    -read.url=http://localhost:8481

JWT Payload Structure

{
  "exp": 1617304574,
  "vm_access": {
    "tenant_id": {
      "account_id": 1,
      "project_id": 5
    },
    "extra_labels": {
      "team": "dev",
      "project": "mobile"
    },
    "extra_filters": ["{env=~\"prod|dev\",team!=\"test\"}"],
    "mode": 1
  }
}

• tenant_id: Routes to the correct cluster tenant.
• extra_labels: Labels automatically appended to ingested metrics.
• extra_filters: Query-time filters that restrict which series a user can see.
• mode: Bitfield controlling read (1) and write (2) access.

TLS Configuration

vmauth Automatic TLS

vmauth Enterprise supports automatic TLS certificate provisioning via Let's Encrypt:

./vmauth \
    -httpListenAddr=:443 \
    -tls=true \
    -tlsAutocertHosts=metrics.example.com \
    -tlsAutocertEmail=[email protected] \
    -tlsAutocertCacheDir=/var/cache/vmauth/tls

Manual TLS

For non-Enterprise or custom certificate deployments:

./vmauth \
    -httpListenAddr=:443 \
    -tls=true \
    -tlsCertFile=/etc/vmauth/certs/server.crt \
    -tlsKeyFile=/etc/vmauth/certs/server.key

Backend TLS

Connections from vmauth to backend components can also use TLS:

users:
  - username: "secure-client"
    password: "token"
    url_prefix:
      - "https://vminsert.internal:8480/insert/1/prometheus/"
      - "https://vmselect.internal:8481/select/1/prometheus/"
    tls_insecure_skip_verify: false

Cluster Security

Network Topology

All VictoriaMetrics cluster components must run in a private network. The recommended topology:

Component	Network Zone	Exposure
vmstorage	Private subnet	No external access
vminsert	Private subnet	Behind vmauth only
vmselect	Private subnet	Behind vmauth only
vmauth	DMZ / public subnet	TLS termination point

Kubernetes Operator Security

The VictoriaMetrics Operator supports a useStrictSecurity flag that enforces security hardening:

• Runs all pods as non-root user.
• Drops all Linux capabilities.
• Sets a read-only root filesystem.
• Restricts discretionary access control.

apiVersion: operator.victoriametrics.com/v1beta1
kind: VMSingle
metadata:
  name: example
spec:
  useStrictSecurity: true

Internal Metrics Protection

The -metricsAuthKey flag protects internal metrics endpoints (/metrics, /debug/pprof):

./vminsert -metricsAuthKey="secret-key-for-metrics"

Requests to metrics endpoints must include the key as a BasicAuth password. This prevents unauthorized access to diagnostic information.

Multi-Tenant Data Isolation

In cluster mode, tenant data is isolated by tenant ID encoded in the URL path:

/insert/<accountID>/prometheus/api/v1/write
/select/<accountID>/prometheus/api/v1/query

• accountID is a numeric identifier (0 = default tenant).
• vmstorage stores data per tenant in separate directories.
• vmauth maps authentication tokens to specific accountID values.
• Cross-tenant queries require explicit configuration with tenant federation.

VictoriaLogs Tenant Isolation

VictoriaLogs uses AccountID and ProjectID headers for multi-tenancy:

# vmauth for VictoriaLogs
users:
  - username: "team-a"
    password: "token-a"
    url_map:
      - src_paths: ["/select/.*", "/insert/.*"]
        headers:
          - "AccountID: 1"
          - "ProjectID: 0"
        url_prefix: ["http://vlselect:9428/"]
  - username: "team-b"
    password: "token-b"
    url_map:
      - src_paths: ["/select/.*"]
        headers:
          - "AccountID: 2"
          - "ProjectID: 0"
        url_prefix: ["http://vlselect:9428/"]

API Endpoint Access Control

vmauth and vmgateway should restrict which API endpoints are accessible per user. Minimizing the attack surface:

Endpoint	Risk	Recommendation
/api/v1/write	Data injection	Restrict to vmagent/service accounts
/api/v1/query	Data exposure	Restrict to Grafana/humans
/api/v1/export	Bulk data export	Restrict heavily
/metrics	Internal diagnostics	Protect with -metricsAuthKey
/debug/*	Pprof, vars	Block at network level
/snapshot*	Storage access	Block entirely in production
/force/merge*	Performance impact	Block at network level

Hardening Checklist

Area	Recommendation
Network isolation	All backends in private subnet; vmauth is the only ingress
TLS	HTTPS between clients and vmauth; optional mTLS to backends
Auth tokens	Unique token per tenant; rotate on compromise
Endpoint restriction	Allow only necessary API paths per user in vmauth config
Kubernetes	Enable useStrictSecurity in the Operator
Metrics protection	Set -metricsAuthKey on all components
Debug endpoints	Block /debug/* and /snapshot* at the network level
Secret management	Store vmauth config passwords in Vault or sealed secrets
Audit	Enable access logging on vmauth for forensic analysis

Sources

• VictoriaMetrics Cluster Security
• vmauth Documentation
• vmgateway Documentation
• VictoriaMetrics Operator