Architecture¶

In-depth view of NATS internals: the wire protocol, cluster routes, leaf nodes, gateways, and the JetStream subsystem (including the meta-Raft, asset-Raft, and storage layers).

Component Map¶

flowchart LR
    subgraph Server["nats-server (single binary)"]
        direction TB
        ClientConn["Client connections\n(TCP 4222 / TLS / WS / MQTT)"]
        SubsRouter["Subject router\n(subject tree)"]
        AccountIso["Account isolation\n(per-account interest)"]
        ClusterRoute["Cluster routes\n(:6222)"]
        Gateway["Gateway endpoint\n(:7222)"]
        LeafEndpoint["Leaf node endpoint\n(:7422)"]
        MQTTBridge["MQTT bridge\n(:1883)"]
        WSBridge["WebSocket bridge\n(:80/:443)"]
        subgraph JetStreamSubsystem["JetStream subsystem (optional)"]
            MetaRaft["Meta-group Raft\n($JS.API)"]
            StreamRaft["Per-stream Raft\n(R1/R3/R5)"]
            ConsumerRaft["Per-consumer Raft\n(durable consumers)"]
            FileStore["File store\n(blocks + idx)"]
            MemStore["Memory store"]
            ObjectStore["Object Store API"]
            KVStore["KV bucket API"]
        end
        Monitoring["HTTP monitoring\n(:8222)"]
    end
    ClientConn --> SubsRouter
    SubsRouter --> AccountIso
    AccountIso --> JetStreamSubsystem
    AccountIso --> ClusterRoute
    AccountIso --> Gateway
    AccountIso --> LeafEndpoint
    MQTTBridge --> SubsRouter
    WSBridge --> SubsRouter
    StreamRaft --> FileStore
    StreamRaft --> MemStore
    KVStore --> StreamRaft
    ObjectStore --> StreamRaft

Components¶

Component	Role
nats-server	Single static Go binary that hosts every role: client gateway, route peer, leaf node endpoint, JetStream node, MQTT bridge, WebSocket bridge, monitoring server.
Subject router	Per-account trie that maps subscribers/queue groups to subjects with wildcard support (`*` token, `>` rest).
Account	Hard isolation boundary. Subjects, JetStream streams, KV buckets, and consumers all live inside an account. Cross-account flow only via explicit imports/exports.
System account (`$SYS`)	Special account used for cluster gossip, JetStream cluster state events, and administrative APIs.
Cluster route	Direct full-mesh TCP between servers in one cluster on port 6222 — propagates subscription interest.
Gateway	Cluster-to-cluster link forming a supercluster. Gossip is summarized — gateways carry interest queries, not the full subject graph.
Leaf node	Outbound connection from a remote/edge nats-server (or `nats-server -leaf`) into a hub cluster. Leaf links are account-scoped.
JetStream meta group	A cluster-wide Raft group ($JS.API.*) that holds stream/consumer assignment metadata. Lives only on JetStream-enabled servers.
Stream Raft group	One Raft group per replicated stream (R3, R5). Uses hashicorp/raft semantics.
Consumer Raft group	Independent Raft group per durable consumer for tracking acks and delivery state.
File store	On-disk blocks per stream, indexed for fast subject + time range queries. Memory-mapped.
MQTT bridge	Implements MQTT 3.1.1 over JetStream — sessions and retained messages persist via JetStream KV.

Subject hierarchy & wildcards¶

NATS subjects are dot-delimited tokens (orders.created.us-east.123). Two wildcards apply only to subscribers:

* — single token (orders.*.us-east.123)
> — rest of the subject (orders.>)

The interest graph is account-local, so orders.> in account A never gets to account B.

flowchart LR
    Pub["Publisher\norders.created.us-east.123"]
    SubAll["sub orders.>"]
    SubRegion["sub orders.*.us-east.>"]
    Q1["queue grp 'workers'\norders.created.>"]
    Q2["queue grp 'workers'\norders.created.>"]
    Pub --> SubAll
    Pub --> SubRegion
    Pub -.->|load-balanced| Q1
    Pub -.->|load-balanced| Q2

Request-Reply¶

NATS request-reply uses an inbox subject. The client subscribes to a unique reply subject, sets the reply field on the request, and waits.

sequenceDiagram
    participant C as Client
    participant S as nats-server
    participant Svc as Service
    C->>C: subscribe _INBOX.abc.123
    C->>S: PUB svc.req reply=_INBOX.abc.123
    S->>Svc: MSG svc.req reply=_INBOX.abc.123
    Svc->>S: PUB _INBOX.abc.123 (response)
    S->>C: MSG _INBOX.abc.123

Sub-millisecond round trips are typical inside a single cluster.

Cluster routing¶

Cluster peers form a full mesh of routes. Each server exchanges interest updates so that every peer knows which other peers have subscribers for any given subject. This makes routing decisions O(1) on the publish path.

A subscription update is a RS+ / RS- operation on the route protocol. Optimization: routes carry per-account interest summaries, not per-subject details for high-fanout sets.

Supercluster (gateways)¶

Gateways are not full meshes of subscriptions. They exchange interest queries lazily. When a publish on cluster A targets a subject with no local interest, the cluster asks adjacent gateways "do you have interest in X?" Gateways cache responses with a TTL.

This design keeps inter-cluster traffic proportional to actual cross-cluster demand rather than subscription counts. Trade-off: first-message latency across the gateway includes a query round trip.

Leaf nodes¶

Leaf nodes connect outbound to a hub. They:

Carry their own accounts (or map a local subset into a hub account).
Show the hub only the subjects their account exposes.
Buffer JetStream traffic when the link is unhealthy if domain= is set.

Common patterns:

Edge factory — leaf node on a Raspberry Pi 5, devices speak MQTT into the leaf, leaf publishes via NATS to the hub.
Per-tenant leaf — SaaS customers run a leaf on-prem. Their data never leaves their network until summary subjects are exported.

JetStream¶

JetStream replaces the historical NATS Streaming server with a fully integrated persistence layer.

Storage model¶

Field	Detail
Block file	Append-only, sized via `max_msgs_per_block` or default 256 MB.
Index	Per-block index of subject + sequence + timestamp. Rebuilt on crash from blocks.
Compaction	By age (`max_age`), size (`max_bytes`), count (`max_msgs`), or per-subject (`max_msgs_per_subject`).
Memory store	Same APIs as file store but volatile. Useful for ephemeral fan-out.

Retention policies¶

Policy	Behavior
`Limits` (default)	Discard oldest when limits hit.
`Interest`	Keep messages while at least one consumer does not ack.
`WorkQueue`	Each message must be consumed exactly once across consumers.

Consumer types¶

Consumer Style	Push	Pull	Notes
Durable	✓	✓	Server tracks ack state. Survives client restarts.
Ephemeral	✓	✓	Server forgets state once inactive.
Queue group	✓	–	Push consumers can join a deliver group for load balance.
Pull	–	✓	Recommended for scale: client batches `Fetch(n)`.
Ordered	✓	–	Single-active replay, used when strict order matters.

Ack policies: none, all, explicit. Replay policies: instant, original.

Stream replication¶

sequenceDiagram
    participant P as Publisher
    participant L as Leader (n1)
    participant F1 as Follower (n2)
    participant F2 as Follower (n3)
    P->>L: PUB orders.created
    L->>L: append to log, fsync
    L->>F1: AppendEntries
    L->>F2: AppendEntries
    F1->>L: ack
    F2->>L: ack
    Note right of L: quorum reached (2/3)
    L->>P: PubAck (stream seq)

R3 / R5 mean three- or five-server Raft groups. Quorum is (N/2)+1. Leader election uses standard Raft term + vote semantics.

KV bucket¶

A KV bucket is a JetStream stream with max_msgs_per_subject=1 and subject prefix $KV.<bucket>.. Put appends. Get reads the last sequence. Watch is a JetStream consumer over the subject set. Compare-and-Swap uses Nats-Expected-Last-Subject-Sequence.

Object Store¶

A bucket of two streams: - OBJ_<bucket> for chunks (default 128 KB chunk size). - OBJ_<bucket>_meta for object metadata.

Get streams chunks back to the client. Replication mirrors stream R-factor.

Decentralized auth (Operators / Accounts / Users)¶

flowchart TB
    Op["Operator\n(O-key, signs accounts)"]
    SysAcc["System Account ($SYS)\n(A-key)"]
    AccA["Account A\n(A-key)"]
    AccB["Account B\n(A-key)"]
    UserA["User credentials\n(U-key + JWT)"]
    UserB["User credentials\n(U-key + JWT)"]
    Op --> SysAcc
    Op --> AccA
    Op --> AccB
    AccA --> UserA
    AccB --> UserB
    Resolver["Account JWT resolver\n(memory / NATS / URL)"]
    AccA -. JWT .- Resolver
    AccB -. JWT .- Resolver
    Server["nats-server\n(trusts Operator key)"]
    Resolver --> Server

NKey — Ed25519 keypair encoded in custom base32 with role prefix (O/A/U/X).
JWT — issued by the Operator (for accounts) or Account (for users). Contains permissions, limits, and signing-key references.
Resolver — a server discovers account JWTs at runtime via MEMORY (preloaded), the embedded nats resolver, or an external URL.

Performance characteristics¶

Workload	Latency / Throughput
Core NATS pub/sub same host	sub-microsecond delivery
Core NATS pub/sub LAN	~50–100 µs RTT
Core NATS publish, single client	tens of millions msgs/sec/server (small payloads)
JetStream R1 file store	hundreds of thousands msgs/sec/server (NVMe)
JetStream R3 file store	~tens of thousands msgs/sec quorum-limited
Gateway cross-region first request	RTT + gateway query (~1 RTT extra)

Source for benchmarks

The numbers in the table above reflect Synadia and community benchmarks (see nats bench results). Always re-measure for your own hardware/NVMe class before sizing a deployment.

Comparison hooks¶

vs Kafka — Kafka wins on broad analytic streaming ecosystem. NATS wins on multi-tenant edge fabrics and operational simplicity.
vs RabbitMQ — RabbitMQ wins on routing primitives & dead-letter ergonomics. NATS wins on latency and footprint.
vs Pulsar — Pulsar offers segregated compute/storage and tiered storage. NATS uses local store with mirrors/sources for similar effects but at smaller scale.

Security¶

NATS' security model is decentralized and identity-first. Every connection presents a credential signed by an account key, signed by an operator key, anchored at a trusted operator NKey configured on each server.

Authentication¶

NATS supports several auth modes. Production deployments should use mode (c):

Mode	Use Case
(a) None	Local dev only.
(b) Username/password or token	Small deployments, quick start.
(c) Decentralized JWT (Operator → Account → User)	Production, multi-tenant, supercluster.
(d) Static NKey	Headless services where rotating creds is hard. No JWT, but cryptographic.
(e) TLS client certificate	mTLS-only setups where PKI is the source of truth.
(f) External callout	`auth_callout` block delegates to an external Ed25519-signed service.

Decentralized JWT model¶

flowchart LR
    Op["Operator NKey (O)\nroot of trust"]
    SAcc["System Account ($SYS)"]
    AccA["Account A"]
    AccB["Account B"]
    UA["User A1"]
    UA2["User A2"]
    UB["User B1"]
    Op -->|signs| SAcc
    Op -->|signs| AccA
    Op -->|signs| AccB
    AccA -->|signs| UA
    AccA -->|signs| UA2
    AccB -->|signs| UB
    Resolver["Account JWT resolver"]
    AccA -. JWT .- Resolver
    AccB -. JWT .- Resolver
    SAcc -. JWT .- Resolver
    Server["nats-server\ntrust = Operator pubkey"]
    Resolver --> Server

Operator NKey — Ed25519 keypair. The public key is configured on every server (operator: O...).
Account JWT — issued by the operator (or a delegated signing key). Contains permissions, limits, exports, imports.
User JWT — issued by the account. Contains pub/sub permissions, response permissions, max subscriptions.
Resolver — the server fetches Account JWTs at runtime from MEMORY, the embedded NATS resolver, or a URL.

NKey-only auth¶

When even JWT bookkeeping is too heavy, an NKey alone can authenticate. The server stores authorized public keys. Clients sign a server-issued nonce with their private key.

Authorization¶

Permissions live in user JWTs:

{
  "pub": { "allow": ["app.events.>"], "deny": [] },
  "sub": { "allow": ["app.cmd.>"],   "deny": ["app.cmd.admin.>"] },
  "resp": { "max": 1, "ttl": "5s" }
}

Concept	Effect
`pub.allow` / `deny`	Subjects this user can publish to.
`sub.allow` / `deny`	Subjects this user can subscribe to.
`resp`	Special "allow responses" rule for request-reply patterns — see `--allow-pub-response` on `nsc edit signing-key`.
Account exports/imports	Cross-account flows. Can be public (any account can import) or private (per-import token).

Subject-level isolation¶

Two accounts cannot see each other's subjects, full stop. There is no privileged subject. This is the cornerstone of multi-tenant NATS.

Limits enforced server-side¶

Max subscriptions per user / account.
Max payload, max in-flight requests.
Max JetStream storage (memory + file) per account.
Max gateway + leaf connections per account.

Encryption¶

In transit¶

TLS 1.2 / 1.3 on every listener (clients, routes, gateways, leaf nodes, MQTT, WS, monitoring).
Per-listener configuration: a hub can require mTLS for gateway peers but optional TLS for clients.
Cipher suites configurable. Restrict to AEAD only.

tls {
  cert_file: "/etc/nats/server.pem"
  key_file:  "/etc/nats/server.key"
  ca_file:   "/etc/nats/ca.pem"
  verify:    true
  verify_and_map: true   # map cert SAN to user identity
  cipher_suites: ["TLS_AES_128_GCM_SHA256", "TLS_AES_256_GCM_SHA384", "TLS_CHACHA20_POLY1305_SHA256"]
}

At rest (JetStream)¶

JetStream supports per-account at-rest encryption with two cipher options:

jetstream {
  store_dir: /data/jetstream
  cipher: chachapoly      # or "aes"
  key: "<base64 32 bytes>" # or use key_file
}

All blocks, indexes, and consumer state files are encrypted with a per-server symmetric key.
Re-keying requires a re-encrypt sweep. Document it in your operations runbook.

Server-to-server¶

Cluster routes, gateways, and leaf nodes each have their own TLS configuration block — required for any prod deployment.

Audit & Observability¶

Connection events are published on the system account: $SYS.ACCOUNT.<acc-id>.CONNECT, DISCONNECT, AUTH_ERROR.
Account auth tracing can be enabled with trace_destinations to forward to a separate audit subject.
Server logs capture failed auth and permission denials. Ship them to your SIEM.
JetStream advisories publish on $JS.EVENT.ADVISORY.> — message-loss, consumer-leader-changes, stream-create/delete.

Threat Model¶

Threat	Mitigation
Operator key theft	Keep operator key offline / in an HSM. Use account signing keys for routine operations.
Account JWT forgery	Mitigated by Operator-signed JWTs. Servers validate each chain.
User JWT theft	Issue short-lived JWTs (`expires_at`). Rotate user creds. Revoke via account-level `revocations`.
Subject leakage across accounts	Architectural impossibility — subjects are account-scoped. Verify exports/imports are correctly bounded.
Slow-consumer DoS	`write_deadline`, `max_pending`, and slow-consumer eviction.
JetStream storage tampering	At-rest encryption (`chachapoly`/`aes`). FS-level integrity (dm-verity / fs-verity). Audit JetStream advisory subjects.
MITM on cluster routes	Require route-level mTLS. Servers refuse non-TLS routes when `tls` is set on the route block.
Compromised leaf node	Leaf is account-scoped. Its subjects can only get to what the leaf account can already see.
Replay of published messages	At app level: include nonce / dedupe via JetStream `Nats-Msg-Id` header (`dupe_window`).
Gateway spoofing	Gateway authentication uses TLS + (optionally) `auth.users` block. Rotate certs.
Resolver poisoning	Pin resolver TLS certificate. Resolver responses are themselves Operator-signed JWTs.

CVE History¶

NATS has had a relatively clean security record. Notable items:

CVE	Year	Affected	Summary
CVE-2024-39830	2024	nats-server before 2.10.16	Improper validation of user permissions can allow request-reply auth bypass under specific permission patterns. Fixed in 2.10.16.
CVE-2023-47092	2023	NATS account JWT	Improper handling of certain account JWT claims. Fixed in 2.10.x.
CVE-2022-24450	2022	nats-server 2.7.x	Privileges over `system` account leaked from a non-system context in certain cluster configs.

Always pin to the latest patch in your minor line and subscribe to the nats-io/nats-server security advisories feed.

Hardening Checklist¶

Cross-references¶

messaging/nats/explanation — for the Operator/Account/User trust diagram.
messaging/nats/how-to-guides — for nsc commands and JWT push procedures.
messaging/index — for cross-broker security comparison.