Skip to content

Architecture

1. Default Topology / Flow

flowchart TB
    subgraph Frontend["Frontend (TypeScript / React)"]
        direction LR
        DashUI["Dashboard UI"]
        ExploreUI["Explore"]
        AlertUI["Alerting UI"]
        PluginUI["Panel & App Plugins"]
    end

    subgraph Backend["Backend (Go)"]
        direction LR
        API["HTTP API Server"]
        Auth["Auth & RBAC"]
        QEngine["Query Engine"]
        AlertEng["Alert Rule Evaluator"]
        Prov["Provisioning Engine"]
        PluginMgr["Plugin Manager<br/>(gRPC subprocess host)"]
    end

    subgraph State["State Layer"]
        DB["Database<br/>(PostgreSQL / MySQL / SQLite)"]
        Cache["Session Cache<br/>(Redis / Memcached)"]
    end

    subgraph External["External Data Sources"]
        Prom["Prometheus / Mimir"]
        LokiDS["Loki"]
        TempoDS["Tempo"]
        SQL["MySQL / PostgreSQL"]
        ES["Elasticsearch"]
        CW["CloudWatch"]
    end

    Frontend --> API
    API --> Auth
    API --> QEngine
    API --> AlertEng
    API --> Prov
    QEngine --> PluginMgr
    PluginMgr -->|gRPC| External
    Auth --> DB
    AlertEng --> DB
    Prov --> DB
    Auth --> Cache

    style Frontend fill:#ff6600,color:#fff
    style Backend fill:#2a2d3e,color:#fff
    style State fill:#1a1d2e,color:#fff
    style External fill:#0d7377,color:#fff

Grafana Server Components

The Grafana server itself is a stateless web application with the following internal layers:

Key Architectural Properties

Property Detail
Stateless frontend All state is in the external DB and cache
Plugin isolation Backend plugins run as gRPC subprocesses
Provisioning Dashboards, data sources, alerts loaded from YAML/JSON at startup
Multi-org Single Grafana instance, multiple isolated organizations
API-first All UI operations have corresponding REST API endpoints

LGTM Stack — Full Production Architecture

flowchart TB
    subgraph Apps["Instrumented Applications"]
        App1["Service A<br/>(OTel SDK)"]
        App2["Service B<br/>(OTel SDK)"]
        App3["Service C<br/>(Prometheus client)"]
    end

    subgraph Infra["Infrastructure"]
        K8s["Kubernetes"]
        Nodes["VM / Bare Metal"]
    end

    subgraph Collection["Grafana Alloy (DaemonSet / Sidecar)"]
        Recv["Receivers<br/>OTLP, Prometheus, Syslog"]
        Proc["Processors<br/>Batch, MemoryLimiter, ResourceDetection"]
        Exp["Exporters"]
    end

    subgraph Mimir["Grafana Mimir"]
        MD["Distributor"]
        MI["Ingester"]
        MQ["Querier"]
        MSg["Store-Gateway"]
        MC["Compactor"]
    end

    subgraph Loki["Grafana Loki"]
        LD["Distributor"]
        LI["Ingester"]
        LQ["Querier"]
        LQF["Query Frontend"]
        LC["Compactor"]
    end

    subgraph Tempo["Grafana Tempo"]
        TD["Distributor"]
        TI["Ingester"]
        TQ["Querier"]
        TQF["Query Frontend"]
        TMG["Metrics Generator"]
    end

    subgraph ObjStore["Object Storage (S3 / GCS / Azure)"]
        Blocks["Metric Blocks"]
        Chunks["Log Chunks + Index"]
        Traces["Trace Blocks (Parquet)"]
    end

    subgraph Grafana["Grafana Server (HA)"]
        GF1["Grafana Pod 1"]
        GF2["Grafana Pod 2"]
        GFn["Grafana Pod N"]
    end

    subgraph Supporting
        PG["PostgreSQL<br/>(Grafana metadata DB)"]
        Redis["Redis<br/>(Session cache)"]
        LB["Load Balancer / Ingress"]
    end

    Apps --> Collection
    Infra --> Collection
    Collection -->|remote_write| MD
    Collection -->|push| LD
    Collection -->|OTLP gRPC| TD

    MD --> MI
    MI --> Blocks
    MQ --> MI
    MQ --> MSg
    MSg --> Blocks
    MC --> Blocks

    LD --> LI
    LI --> Chunks
    LQF --> LQ
    LQ --> LI
    LQ --> Chunks
    LC --> Chunks

    TD --> TI
    TI --> Traces
    TQF --> TQ
    TQ --> TI
    TQ --> Traces
    TMG --> MD

    GF1 --> PG
    GF2 --> PG
    GFn --> PG
    GF1 --> Redis
    LB --> GF1
    LB --> GF2
    LB --> GFn

    Grafana -.->|PromQL| MQ
    Grafana -.->|LogQL| LQF
    Grafana -.->|TraceQL| TQF

    style Apps fill:#0d7377,color:#fff
    style Infra fill:#0d7377,color:#fff
    style Collection fill:#ff6600,color:#fff
    style Mimir fill:#7b42bc,color:#fff
    style Loki fill:#2a7de1,color:#fff
    style Tempo fill:#e65100,color:#fff
    style ObjStore fill:#0d1117,color:#fff
    style Grafana fill:#ff6600,color:#fff
    style Supporting fill:#1a1d2e,color:#fff

Mimir Architecture (Metrics)

flowchart LR
    subgraph Write["Write Path"]
        D["Distributor<br/>(validates, shards, replicates)"]
        I["Ingester<br/>(in-memory TSDB + WAL)"]
    end

    subgraph Read["Read Path"]
        QF["Query Frontend<br/>(splits, caches, queues)"]
        Q["Querier<br/>(executes PromQL)"]
        SG["Store-Gateway<br/>(indexes object storage)"]
    end

    subgraph Background["Background"]
        C["Compactor<br/>(vertical + horizontal compaction)"]
    end

    subgraph Storage["Object Storage"]
        OS["S3 / GCS / Azure<br/>(TSDB Blocks)"]
    end

    Prom["Prometheus / Alloy"] -->|remote_write| D
    D -->|hash ring| I
    I -->|flush every 2h| OS
    QF --> Q
    Q -->|recent data| I
    Q -->|historical data| SG
    SG --> OS
    C --> OS

    style Write fill:#7b42bc,color:#fff
    style Read fill:#2a7de1,color:#fff
    style Background fill:#1a1d2e,color:#fff
    style Storage fill:#0d1117,color:#fff

Deployment Modes

Mode Description Use Case
Monolithic All components in a single process/pod Dev, testing, small scale
Read-Write Separate read and write paths Medium scale
Microservices Each component as independent pods Production, hyperscale

Loki Architecture (Logs)

flowchart LR
    subgraph Write["Write Path"]
        LD["Distributor<br/>(validates, routes by label hash)"]
        LI["Ingester<br/>(compresses into chunks, indexes labels)"]
    end

    subgraph Read["Read Path"]
        LQF["Query Frontend<br/>(splits time ranges, queues)"]
        LQ["Querier<br/>(executes LogQL)"]
        LIG["Index Gateway<br/>(metadata lookups)"]
    end

    subgraph Background["Background"]
        LC["Compactor<br/>(merges index files, retention)"]
    end

    subgraph Storage["Object Storage"]
        LOS["S3 / GCS / Azure<br/>(Chunks + Index)"]
    end

    Alloy["Alloy / Promtail"] -->|push| LD
    LD --> LI
    LI -->|flush| LOS
    LQF --> LQ
    LQ -->|recent| LI
    LQ -->|historical| LIG
    LIG --> LOS
    LC --> LOS

    style Write fill:#2a7de1,color:#fff
    style Read fill:#0d7377,color:#fff
    style Background fill:#1a1d2e,color:#fff
    style Storage fill:#0d1117,color:#fff

Key Design Choice: Loki only indexes labels, not log content. This makes it 10–100x cheaper to operate than full-text-indexing alternatives (for example, Elasticsearch) but requires effective label design.

Tempo Architecture (Traces)

flowchart LR
    subgraph Write["Write Path"]
        TD["Distributor<br/>(OTLP, Jaeger, Zipkin)"]
        TI["Ingester<br/>(Parquet columns + bloom filters)"]
    end

    subgraph Read["Read Path"]
        TQF["Query Frontend<br/>(splits, shards)"]
        TQ["Querier<br/>(TraceQL engine)"]
    end

    subgraph SideEffects["Side Effects"]
        TMG["Metrics Generator<br/>(RED metrics → Mimir)"]
    end

    subgraph Storage["Object Storage"]
        TOS["S3 / GCS / Azure<br/>(Parquet Trace Blocks)"]
    end

    OTel["Apps (OTel SDK)"] -->|OTLP| TD
    TD --> TI
    TD --> TMG
    TI -->|flush blocks| TOS
    TQF --> TQ
    TQ --> TI
    TQ --> TOS
    TMG -->|remote_write| Mimir["Mimir"]

    style Write fill:#e65100,color:#fff
    style Read fill:#ff6600,color:#fff
    style SideEffects fill:#7b42bc,color:#fff
    style Storage fill:#0d1117,color:#fff

Key Design Choice: No traditional index — Tempo uses Parquet columnar storage with bloom filters. TraceQL queries selectively load required columns. This makes large-scale trace search performant.

Kubernetes Deployment Topology

A typical production Grafana + LGTM deployment on Kubernetes uses these Helm charts:

Component Helm Chart Min Replicas Scaling
Grafana grafana/grafana 2+ (HA) HPA on CPU/memory
Mimir grafana/mimir-distributed 3+ ingesters Per-component HPA
Loki grafana/loki 3+ ingesters Per-component HPA
Tempo grafana/tempo-distributed 3+ ingesters Per-component HPA
Alloy grafana/alloy (DaemonSet) 1 per node DaemonSet auto-scales
PostgreSQL External managed (RDS/CloudSQL) HA pair Managed service
Redis External managed (ElastiCache) HA pair Managed service

Data Model

1. Default Topology / Flow

erDiagram
    Grafana_CORE ||--o{ CONFIG : requires
    Grafana_CORE ||--o{ STATE : writes
    CONFIG {
        string runtime_params
        string limits
    }
    STATE {
        string metric_id
        json payload
    }

How It Works

Core Mechanism

Grafana is fundamentally a query, transform, and visualize engine. It does not store time-series data itself (except for configuration and alert state). Instead, it proxies queries to external data sources and renders the results in a browser.

Request Lifecycle

  1. User opens a dashboard → browser loads the dashboard JSON model
  2. Panel queries are dispatched → each panel sends its query to the Grafana backend
  3. Backend proxies to data source → Grafana translates the query and forwards it to the appropriate backend (Prometheus, Loki, SQL, and more) using the configured data-source plugin
  4. Results are returned → data frames are sent back to the frontend
  5. Frontend renders → React-based panel plugins render the visualization
sequenceDiagram
    participant User as Browser
    participant GF as Grafana Server
    participant DS as Data Source<br/>(Prometheus, Loki, etc.)

    User->>GF: Open Dashboard
    GF-->>User: Dashboard JSON + Panel Config
    User->>GF: Execute Panel Queries
    GF->>DS: Proxy Query (PromQL, LogQL, SQL...)
    DS-->>GF: Data Frames / Results
    GF-->>User: Transformed Data
    User->>User: Render Visualization

Data Frames

Grafana uses a unified data abstraction called Data Frames — typed, columnar data structures (similar to Pandas DataFrames) that all data-source plugins must return. This abstraction lets any panel plugin render data from any source without tight coupling.

The LGTM Stack

The Grafana ecosystem addresses all pillars of observability through purpose-built backends:

Signal Backend Query Language Storage
Metrics Grafana Mimir PromQL Object Storage (S3/GCS/Azure)
Logs Grafana Loki LogQL Object Storage
Traces Grafana Tempo TraceQL Object Storage (Parquet columnar)
Profiles Grafana Pyroscope FlameQL Object Storage
Collection Grafana Alloy HCL-based config (River) N/A (pipeline agent)

Cross-Signal Correlation

The true power of the LGTM stack is cross-signal linking:

  • Exemplars: Metric data points carry trace IDs → click a spike in Mimir and jump to the exact trace in Tempo
  • Trace-to-Logs: A trace span carries labels that map to Loki log streams → jump from trace to logs
  • Derived Fields: Loki logs are parsed for trace IDs → jump from logs back to traces
  • Profiles: Pyroscope profiles are linked via labels to traces and metrics
flowchart TB
    subgraph Collection["Grafana Alloy (OTel Collector)"]
        direction LR
        R[Receivers<br/>OTLP, Prometheus, Syslog]
        P[Processors<br/>Batch, Filter, Transform]
        E[Exporters<br/>Remote Write, OTLP]
        R --> P --> E
    end

    subgraph Backends["LGTM Backends"]
        Mimir["Mimir<br/>(Metrics)"]
        Loki["Loki<br/>(Logs)"]
        Tempo["Tempo<br/>(Traces)"]
        Pyroscope["Pyroscope<br/>(Profiles)"]
    end

    subgraph Storage["Object Storage"]
        S3["S3 / GCS / Azure Blob"]
    end

    Collection -->|remote_write| Mimir
    Collection -->|push| Loki
    Collection -->|OTLP| Tempo
    Collection -->|push| Pyroscope

    Mimir --> S3
    Loki --> S3
    Tempo --> S3
    Pyroscope --> S3

    subgraph Grafana["Grafana UI"]
        Dash[Dashboards]
        Explore[Explore]
        Alert[Alerting]
    end

    Mimir -.->|PromQL| Grafana
    Loki -.->|LogQL| Grafana
    Tempo -.->|TraceQL| Grafana
    Pyroscope -.->|FlameQL| Grafana

    style Collection fill:#2a2d3e,color:#fff
    style Backends fill:#1a1d2e,color:#fff
    style Storage fill:#0d1117,color:#fff
    style Grafana fill:#ff6600,color:#fff

Plugin Architecture

Grafana's extensibility is built on a modular plugin system:

Plugin Types

Type Purpose Example
Data Source Connect to external data backends Prometheus, MySQL, Elasticsearch
Panel Custom visualization types Time series, Stat, Geomap, Flame graph
App Bundles of datasources + panels + pages Grafana Incident, Grafana OnCall
Renderer Server-side image/PDF rendering grafana-image-renderer

Plugin Lifecycle

  1. Discovery — Grafana scans the plugin directory on startup
  2. Bootstrap — reads plugin.json metadata (ID, type, dependencies)
  3. Validation — checks plugin signature (signed/unsigned/private)
  4. Initialization — loads frontend (React) and backend (Go via gRPC subprocess)

Frontend ↔ Backend Communication

Backend plugins run as separate processes and communicate with the Grafana server via gRPC. This isolation means:

  • A crashing plugin does not crash Grafana
  • Plugins can implement custom auth, caching, and alerting
  • Sensitive operations (secrets, credentials) stay server-side

Key SDK packages: - @grafana/data — data structures, plugin base classes - @grafana/ui — reusable React UI components (Grafana design system) - @grafana/runtime — runtime services (data fetching, config) - Grafana Plugin SDK for Go — server-side plugin development in Go

Alerting Pipeline (Unified Alerting)

Since Grafana 9+, alerting uses a unified architecture that works across all data sources:

flowchart LR
    subgraph Rules["Alert Rules"]
        R1["Rule 1<br/>PromQL: cpu > 80%"]
        R2["Rule 2<br/>LogQL: error rate"]
    end

    subgraph Eval["Rule Evaluator"]
        E["Periodic Evaluation<br/>(every N seconds)"]
    end

    subgraph State["Alert State Manager"]
        S["Normal → Pending → Alerting"]
    end

    subgraph NP["Notification Policies"]
        Tree["Routing Tree<br/>(label matchers)"]
    end

    subgraph CP["Contact Points"]
        Slack[Slack]
        PD[PagerDuty]
        Email[Email]
        WH[Webhook]
    end

    Rules --> Eval --> State --> NP
    NP -->|severity=critical| PD
    NP -->|team=backend| Slack
    NP -->|default| Email
    NP -->|custom| WH

    style Rules fill:#1a1d2e,color:#fff
    style Eval fill:#2a2d3e,color:#fff
    style State fill:#2a2d3e,color:#fff
    style NP fill:#ff6600,color:#fff
    style CP fill:#0d7377,color:#fff

Key Concepts

  • Alert Rules define what to evaluate and the threshold conditions
  • Labels on alert instances drive routing (for example, severity=critical, team=infra)
  • Notification Policies form a routing tree — each policy matches labels and routes to contact points
  • Contact Points define destinations (Slack, PagerDuty, Email, Webhook, OpsGenie, and more)
  • Mute Timings suppress alerts during maintenance windows
  • Silences temporarily suppress specific alert instances during incidents

Data Flow

Grafana Alloy Pipeline

Grafana Alloy (successor to Grafana Agent) is the recommended telemetry collection agent:

  • Two configuration modes:
  • Default Engine (Alloy/River syntax) — HCL-based, component-oriented, supports clustering and debug UI

  • OpenTelemetry Engine — standard YAML OTel Collector config for portability

  • Pipeline stages: Receivers → Processors → Exporters

  • Debug UI available at http://localhost:12345 for real-time pipeline inspection

Grafana Mimir (Metrics)

  1. Distributor receives remote-write from Prometheus/Alloy → validates, shards by tenant
  2. Ingester writes to in-memory TSDB + WAL → flushes 2-hour blocks to object storage
  3. Querier executes PromQL across ingesters (recent) and store-gateways (historical)
  4. Compactor merges and deduplicates blocks in object storage
  5. Store-Gateway indexes object storage blocks for fast historical queries

Grafana Loki (Logs)

  1. Distributor receives log streams from Alloy → routes by label hash
  2. Ingester compresses logs into chunks, indexes labels only (not full text)
  3. Querier executes LogQL across ingesters and object storage
  4. Compactor merges index files and enforces retention

Key insight: Loki does not index log content — only metadata labels. This dramatically reduces storage costs but requires queries to start with a label selector.

Grafana Tempo (Traces)

  1. Distributor receives traces (OTLP, Jaeger, Zipkin) → routes by trace ID hash
  2. Ingester organizes spans into Apache Parquet columns, creates bloom filters, flushes blocks
  3. Querier searches by trace ID or uses TraceQL for attribute-based search
  4. Metrics-Generator (optional) extracts RED metrics (Rate, Errors, Duration) from spans → pushes to Mimir

Key insight: Tempo uses no index — it relies on object storage + Parquet columnar format + bloom filters. This makes it extremely cheap to operate at scale.

Lifecycle

Grafana Server Lifecycle

  1. Startup — loads grafana.ini config, runs database migrations, discovers plugins
  2. Runtime — serves HTTP/HTTPS, processes API requests, evaluates alert rules, manages sessions
  3. Shutdown — graceful drain of connections, flushes pending alert state

Dashboard Lifecycle

  1. Created (UI or provisioning) → stored as JSON in the Grafana database
  2. Versioned — each save creates a new version (built-in version history)
  3. Provisioned (optional) — dashboards loaded from YAML/JSON files on disk, watched for changes every 10s
  4. Exported — dashboards can be exported as JSON for sharing or IaC

Benchmarks

Test Conditions

  • Grafana Version tested: v12.x (current stable line, 2026)
  • LGTM Stack Components: Mimir, Loki, Tempo (latest stable)
  • Date: April 2026
  • Note: Grafana itself is primarily a visualization layer. Performance is almost always bottlenecked by the underlying data source, not Grafana's rendering engine. The benchmarks below reflect both Grafana UI limits and backend throughput.

Grafana Server Performance

Dashboard Rendering

Metric Observation Source
Panels per dashboard No hard limit. Practical limit ~25–30 before browser sluggishness Grafana Docs
Recommended panel count 8–12 (overview), 15–20 (detailed) Community best practices
Data points per panel Rendering degrades above ~10k points. Use maxDataPoints to cap Grafana Docs
Dashboard load time target < 3 seconds for 95th percentile Industry SRE standard
Concurrent viewers Grafana server itself handles hundreds. The bottleneck is query load on backends Grafana Docs

Query Performance Guidelines

Query Type Permitted p99 Concern Threshold
Simple PromQL (1 series) < 200ms > 500ms
Moderate PromQL (10–50 series) < 1s > 2s
Complex PromQL (100+ series, range) < 5s > 10s
LogQL (label-filtered) < 2s > 5s
LogQL (full scan, large window) < 30s > 60s
TraceQL (by trace ID) < 500ms > 2s
TraceQL (attribute search) < 10s > 30s

Mimir Benchmarks (Metrics)

Mimir is designed for hyperscale Prometheus metrics:

Metric Benchmark Conditions
Active series 1 billion+ Documented by Grafana Labs
Ingestion rate 30M+ samples/sec Large-scale production deployments
Query throughput Thousands of concurrent PromQL queries With query-frontend sharding
Storage efficiency 1.2–1.5 bytes per sample (compressed TSDB blocks on object storage) With compaction
Ingester flush interval 2 hours (default TSDB block size) Configurable
Replication factor 3 (default for durability) Configurable

Mimir Cost Efficiency

Scale Estimated Infra Cost (Self-Hosted) Notes
100k active series ~$50–100/mo Monolithic mode, minimal nodes
1M active series ~$200–500/mo Microservices mode recommended
10M active series ~$1,000–3,000/mo Full microservices, HA
100M+ active series $5,000–20,000+/mo Enterprise-grade infra

Loki Benchmarks (Logs)

Metric Benchmark Conditions
Ingestion rate 1 TB+/day Documented in production at scale
Query performance Label-filtered: sub-second. Full scan: depends heavily on time range and volume Label cardinality is the primary factor
Compression ratio 10–20:1 (Snappy/GZIP on chunks) Varies by log structure
Storage cost Up to 90% cheaper than Elasticsearch for same data volume Due to label-only indexing
Max active streams Configurable per tenant (default: 5,000) Set via max_global_streams_per_user

Loki Cardinality Guidelines

Label Strategy Active Streams Impact
Ideal (namespace, pod, job) < 10k Performance is best
Moderate (+ container, node) 10k–50k Permitted
High cardinality (+ request ID) 50k–500k+ Performance degrades, ingester memory spikes

Critical: Never use user IDs, request IDs, or IP addresses as Loki labels. Use them in log content and filter with LogQL pipe expressions.

Tempo Benchmarks (Traces)

Metric Benchmark Conditions
Ingestion rate 100M+ spans/day Production at Grafana Labs scale
Trace ID lookup < 200ms typical Direct trace ID queries
TraceQL search Seconds to tens of seconds Depends on time range and attribute selectivity
Storage cost Significantly cheaper than Jaeger + Elasticsearch No index required. Object storage only
Parquet block size Configurable, typically 100–500 MB Larger blocks improve search, increase flush latency

Comparison: LGTM Cost vs Alternatives

Stack Metrics (1M series) Logs (100 GB/day) Traces (50M spans/day) Total Estimated
Self-hosted LGTM $200–500/mo $300–800/mo $200–500/mo $700–1,800/mo
Grafana Cloud Pro $500–1,000/mo $500–1,500/mo $300–800/mo $1,300–3,300/mo
Datadog $1,500–5,000/mo $2,000–8,000/mo $1,000–4,000/mo $4,500–17,000/mo
New Relic (Full Platform) $1,000–3,000/mo $1,500–5,000/mo included $2,500–8,000/mo

Costs are rough estimates for mid-2026, vary significantly by configuration, data volume, retention, and provider.

Caveats

  • Grafana UI performance depends heavily on the browser — Chrome works best for large dashboards
  • Backend query performance is 90% determined by the data source, not Grafana
  • Loki and Tempo are optimized for object storage — running them on local disk undermines cost benefits
  • Mimir benchmarks assume proper recording rules for expensive queries
  • All cost estimates assume reasonable retention (15–30 days for logs/traces, 13 months for metrics)

Sources

URL Source Kind Authority Date
https://grafana.com/docs/grafana/latest/best-practices/ docs primary 2026-04-10
https://grafana.com/docs/mimir/latest/references/architecture/ docs primary 2026-04-10
https://grafana.com/docs/loki/latest/get-started/overview/ docs primary 2026-04-10
https://grafana.com/docs/tempo/latest/getting-started/ docs primary 2026-04-10
https://grafana.com/pricing/ docs primary 2026-04-10

Security

Security model covering authentication, authorization, data access, and embedding for Grafana. See also: observability/grafana/index, observability/grafana/explanation, observability/grafana/how-to-guides.

Authentication Providers

Grafana supports multiple authentication backends, configurable in grafana.ini under the [auth.*] sections.

Built-In (Basic Auth)

  • Default user/password stored in Grafana's internal database (SQLite, MySQL, or PostgreSQL).
  • Passwords hashed with bcrypt. Minimum length enforced via min_password_length (default: 4).
  • Admin user (admin) created on first start. It must be changed on first login.

OAuth2

Grafana supports generic OAuth2 plus pre-configured providers: Google, GitHub, GitLab, Azure AD, Okta, and Keycloak.

[auth.google]
enabled = true
client_id = <client-id>
client_secret = <client-secret>
allowed_domains = mycompany.com
allow_sign_up = true
auto_login = false
scopes = openid email profile

Key behaviors: - allow_sign_up controls whether OAuth users are auto-created on first login. - auto_login redirects unauthenticated users directly to the IdP, skipping the login page. - Team sync (team_ids claim) maps IdP groups to Grafana teams (Enterprise feature).

LDAP

Direct LDAP integration maps directory groups to Grafana org roles. Configuration lives in [auth.ldap] with a separate ldap.toml config file.

[auth.ldap]
enabled = true
config_file = /etc/grafana/ldap.toml
allow_sign_up = true

LDAP supports: - Group-to-role mapping (Admin, Editor, Viewer). - Team sync: LDAP groups mapped to Grafana teams. - Multiple LDAP servers with fallback. - Enhanced LDAP (Enterprise): scheduled sync, group search filter improvements.

SAML

SAML 2.0 available in Grafana Enterprise and Grafana Cloud. Configured in grafana.ini:

[auth.saml]
enabled = true
name = Corporate IdP
private_key_path = "/etc/grafana/saml/private_key.pem"
certificate_path = "/etc/grafana/saml/certificate.cert"
idp_metadata_url = "https://idp.YOUR_DOMAIN/saml/metadata"
assertion_attribute_name = DisplayName
assertion_attribute_login = Login
assertion_attribute_email = Email
assertion_attribute_groups = Group

SAML Cookie Requirements

SAML requires cookie_samesite = lax and cookie_secure = true in the [security] section to handle IdP redirects correctly.

Auth Proxy

An external reverse proxy (Nginx, Apache, Envoy, Authelia) authenticates users and passes the identity to Grafana via HTTP headers.

[auth.proxy]
enabled = true
header_name = X-WEBAUTH-USER
header_property = username
auto_sign_up = true
enable_login_token = true

# Apache example
<Proxy *>
    AuthType Basic
    AuthName GrafanaAuthProxy
    AuthBasicProvider file
    AuthUserFile /etc/apache2/grafana_htpasswd
    Require valid-user
    RewriteEngine On
    RewriteRule .* - [E=PROXY_USER:%{LA-U:REMOTE_USER},NS]
    RequestHeader set X-WEBAUTH-USER "%{PROXY_USER}e"
</Proxy>

Auth Proxy Security

Auth proxy bypasses Grafana's built-in auth entirely. Protect the proxy endpoint with TLS and restrict network access. Set enable_login_token = true so Grafana issues its own session cookie after proxy validation.

Authorization and RBAC

Organization Roles

Grafana uses a three-tier org role model. Every user has exactly one org role per organization.

Role Dashboard Access Data Source Access Admin Capabilities
Viewer View only Query (if permitted) None
Editor Create, edit, delete Query (if permitted) Manage own dashboards
Admin Full org management Full management Users, teams, org settings

Folder and Dashboard Permissions

Folders serve as the primary permission boundary. Dashboard permissions inherit from their parent folder and can be overridden at the dashboard level.

Permission levels: - 1 = View: Read-only access to the dashboard. - 2 = Edit: Modify dashboard JSON, panels, and settings. - 4 = Admin: Manage permissions, delete, move the dashboard.

Permissions can be assigned to: - Individual users - Teams - Built-in roles (Viewer, Editor)

POST /api/folders/{uid}/permissions
{
  "items": [
    {"role": "Viewer", "permission": 1},
    {"role": "Editor", "permission": 2},
    {"teamId": 5, "permission": 4}
  ]
}

Admin Override

Users with the org Admin role always have full access to all folders and dashboards. Permissions cannot restrict Admin access.

Role-Based Access Control (Enterprise)

Grafana Enterprise and Cloud extend authorization with fine-grained RBAC:

  • Fixed roles: Pre-defined permission sets (fixed:datasources:reader, fixed:datasources:writer).
  • Custom roles: Administrators create roles with specific actions (for example, datasources:query, dashboards:create).
  • Role assignments: Roles assigned to users, teams, or service accounts.
  • Permission caching: [rbac] permission_cache = true reduces API calls.

RBAC enables least-privilege patterns such as: - Granting a team query-only access to a specific data source. - Allowing a service account to create dashboards without granting org-wide Editor. - Restricting alert management to a specific SRE team.

Data Source Permissions (Enterprise)

Data source permissions restrict which users and teams can query specific data sources.

GET /api/access-control/datasources/{uid}

Returns permission entries with granular actions: - datasources:read - datasources:query - datasources:write - datasources:delete

Service Accounts and API Keys

Service Accounts

Service accounts replace API keys as the primary method for authenticating automated workloads (Terraform, CI/CD pipelines, report generators).

Key properties: - Distinct from Grafana users. Scoped to a single organization. - Support multiple tokens per account. - Tokens remain active even if the creating user is deleted. - Can be assigned org roles (Viewer, Editor, Admin) and RBAC roles (Enterprise). - The None basic role (since 10.2.0) grants no permissions by default. This enables pure RBAC-driven access.

POST /api/serviceaccounts
{
  "name": "terraform-provisioner",
  "role": "Editor"
}

Legacy API Keys

API keys are deprecated in favor of service accounts. Existing keys continue to work but cannot use RBAC. Keys have: - A role (Viewer, Editor, Admin) or a service account scope. - An optional expiration date. - A single token per key (unlike service accounts).

Embedding Security

Iframe Embedding

By default, Grafana blocks iframe embedding. To enable:

[security]
allow_embedding = true

Embedding Risks

Enabling allow_embedding without cookie_samesite = none and cookie_secure = true prevents embedded sessions from working. Combine with Content-Security-Policy headers to restrict which domains can embed Grafana.

Content-Security-Policy

Grafana supports CSP configuration to mitigate XSS and clickjacking:

[security]
content_security_policy = true
content_security_policy_template = """script-src 'self' 'unsafe-eval' 'unsafe-inline' 'nonce-$NONCE' object-src 'none'; font-src 'self'; style-src 'self' 'unsafe-inline' img-src 'self' data:;"""

The $NONCE placeholder is replaced per-request with a cryptographic nonce. This allows only Grafana-generated inline scripts.

Anonymous Access

Anonymous (unauthenticated) access can be enabled for public dashboards:

[auth.anonymous]
enabled = true
org_name = Main Org.
org_role = Viewer

Anonymous Access

Anonymous access grants the configured role to all unauthenticated users. Restrict to a dedicated organization and use data source permissions to control what data anonymous users can query.

[security]
cookie_secure = true          # HTTPS only
cookie_samesite = lax         # or 'none' for cross-site embedding
login_cookie_name = grafana_session
login_maximum_inactive_lifetime_days = 7
login_maximum_lifetime_days = 30

Audit Logging (Enterprise)

Grafana Enterprise records security-relevant events:

[auditing]
enabled = true

Logged events include: - User login/logout and failed authentication attempts. - Role and permission changes. - Dashboard, folder, and data source creation/modification/deletion. - Data source query activity (optional). - API key and service account management.

Audit logs can be forwarded to external systems (Loki, Elasticsearch, file) for long-term retention and compliance.

Grafana Assistant AI Security

The Grafana AI assistant (Grafana Cloud) follows a least-privilege model:

  • The LLM agent can only query dashboards and metrics the invoking user is authorized to see.
  • Permission checks happen before data is sent to the model.
  • Backend LLM responses are proxied through Grafana Cloud infrastructure so external vendors never access tenant data directly.
  • On self-hosted instances, the AI feature is disabled by default and requires explicit opt-in.

Request Security (Enterprise)

Request security restricts outgoing requests from the Grafana server:

  • Block or allowlist domains for data source connections.
  • Prevent SSRF attacks by restricting internal network access.
  • Configure per-data-source URL allowlists.

Vault Integration (Enterprise)

Grafana Enterprise integrates with HashiCorp Vault for secrets management:

  • Store database passwords, API keys, and external credentials in Vault.
  • Reference Vault paths in provisioning configuration.
  • Automatic secret rotation without Grafana restarts.

Security Architecture Diagram

flowchart TD
    subgraph "External"
        Browser[Browser / Client]
        IdP[Identity Provider<br/>OIDC / SAML / LDAP]
    end

    subgraph "Auth Layer"
        Proxy[Reverse Proxy<br/>Nginx / Envoy]
        GrafanaAuth[Grafana Auth Module]
    end

    subgraph "Grafana Server"
        Session[Session Manager<br/>cookie_secure + cookie_samesite]
        RBAC[RBAC Engine<br/>Org Roles + Folder Perms]
        DSPerms[Data Source Permissions]
        Audit[Audit Logger]
        AI[Grafana AI Assistant<br/>Permission-Bound]
    end

    subgraph "Backends"
        DB[(Grafana DB<br/>PostgreSQL / MySQL)]
        Vault[(HashiCorp Vault<br/>Enterprise)]
    end

    Browser --> Proxy --> GrafanaAuth
    GrafanaAuth --> IdP
    GrafanaAuth --> Session
    Session --> RBAC
    RBAC --> DSPerms
    RBAC --> AI
    DSPerms --> Audit
    GrafanaAuth --> DB
    GrafanaAuth -.-> Vault

Hardening Checklist

Area Setting Recommendation
Cookies cookie_secure = true Enforce HTTPS-only cookies
Cookies cookie_samesite = lax Prevent CSRF via cross-site cookies
CSP content_security_policy = true Block XSS and unauthorized scripts
Anonymous auth.anonymous.enabled Disable unless explicitly needed
Embedding allow_embedding Enable only if iframe integration is required
Auth Proxy enable_login_token = true Issue Grafana session tokens after proxy auth
Admin user Default password Change immediately on first start
TLS protocol = https Terminate TLS at proxy or Grafana directly
Audit [auditing] enabled = true Track permission and config changes (Enterprise)
Secrets Vault integration Externalize sensitive credentials (Enterprise)

Sources