Skip to content

Architecture

Redpanda's internals: the Seastar reactor, thread-per-core model, per-partition Raft replication, the controller, archival uploader (tiered storage), Schema Registry, Pandaproxy, and WASM data transforms.

Component Overview

flowchart TB
    Client["Kafka client / Connect / kcat"]
    subgraph Process["Single Redpanda Process (per node)"]
        direction TB
        Reactor["Seastar Reactor\n(thread-per-core)"]
        subgraph Shards["Per-core Shards"]
            S0["Shard 0\n(reactor + partitions)"]
            S1["Shard 1"]
            S2["Shard 2"]
            S3["Shard 3"]
        end
        Controller["Controller\n(Raft group on Shard 0)"]
        PartitionRaft["Partition Raft groups\n(distributed across shards)"]
        ArchivalUploader["Archival Uploader\n(per-partition uploaders)"]
        SchemaRegistry["Schema Registry"]
        Pandaproxy["Pandaproxy (HTTP REST)"]
        WasmRuntime["WASM Transform Runtime\n(in-broker)"]
        StorageEngine["Storage Engine\n(NTP segments + index)"]
    end
    subgraph ObjectStore["Object Storage Tier"]
        S3["S3 / GCS / Azure ADLS"]
    end
    Client --> Reactor
    Reactor --> Shards
    S0 --> Controller
    Shards --> PartitionRaft
    PartitionRaft --> StorageEngine
    StorageEngine --> ArchivalUploader
    ArchivalUploader --> S3
    PartitionRaft --> WasmRuntime
    Reactor --> SchemaRegistry
    Reactor --> Pandaproxy

Components

Component Role
Seastar reactor Runs one event loop per CPU core. Futures/promises remove the need for syscalls and locks.
Shard (core) Owns a subset of partitions. Messages move between shards via lockless queues.
Controller Cluster metadata: topic config, ACLs, cluster membership, broker registration.
Per-partition Raft Each partition is its own Raft group with its own leader/followers.
Storage engine Append-only segments + indexes per Name-Topic-Partition (NTP).
Archival uploader Uploads closed segments to S3/GCS/Azure for tiered storage.
Cloud storage interface Reads tiered segments back transparently when consumers ask for old offsets.
Schema Registry Stores Avro/Protobuf/JSON schemas, compatible with Confluent SR API.
Pandaproxy REST gateway implementing the Kafka REST Proxy contract.
WASM Transform Runtime Runs user-supplied WASM modules to mutate messages on produce.
rpk CLI bundled with the binary. Talks to the Admin API on :9644.

Thread-per-core (Seastar)

Traditional brokers (Kafka, RabbitMQ) use thread pools shared across CPU cores, which incurs context switches and locking. Redpanda uses Seastar, where:

  • Each CPU core has one reactor thread pinned to it.
  • Tasks on a core never block. I/O is asynchronous via io_uring or aio.
  • Inter-core communication is through lockless message-passing queues.
  • Memory is shard-local. Each core has its own slab allocator.

This eliminates lock contention for hot paths and gives predictable tail latency at the cost of slightly more bookkeeping when partitions need cross-shard work (uncommon).

Per-partition Raft

sequenceDiagram
    participant P as Producer
    participant L as Partition Leader (Node A)
    participant F1 as Follower (Node B)
    participant F2 as Follower (Node C)
    P->>L: Produce(records)
    L->>L: append to log + flush
    L->>F1: AppendEntries
    L->>F2: AppendEntries
    F1->>L: ack
    F2->>L: ack
    Note right of L: quorum reached (2/3)
    L->>P: ProduceResponse

Unlike Kafka (which keeps separate KRaft for metadata and ISR-based replication for data), Redpanda uses Raft for both metadata and partition data. Benefits:

  • Single, well-understood consensus protocol.
  • No "elections vs ISR" duality.
  • Cluster membership changes are just Raft reconfiguration.

Trade-off: per-partition Raft groups have a small fixed cost (heartbeats, vote tracking), so very tiny topics with many partitions are not the optimal regime.

Cluster Metadata (Controller)

The controller is a single Raft group covering the whole cluster. It holds:

  • Topic configurations and ACLs.
  • Partition assignment to brokers.
  • Cluster membership and broker registration.
  • Schema Registry data (when running in single-process mode).
flowchart LR
    AdminAPI["rpk / Admin API"]
    Controller["Controller (Raft)"]
    Broker1["Broker 1"]
    Broker2["Broker 2"]
    Broker3["Broker 3"]
    AdminAPI --> Controller
    Controller --> Broker1
    Controller --> Broker2
    Controller --> Broker3

Tiered Storage

flowchart LR
    LocalDisk["Local NVMe segments"]
    Uploader["Archival Uploader"]
    S3["S3 / GCS / Azure"]
    SegMeta["Segment metadata\n(controller)"]
    LocalDisk -->|on close| Uploader
    Uploader --> S3
    Uploader --> SegMeta
    Reader["Consumer fetch (old offset)"]
    Reader --> CloudIface["Cloud Storage Interface"]
    CloudIface --> S3
    CloudIface --> Reader

When a segment closes (size or time threshold), the uploader pushes it to object storage. Consumers reading old offsets fetch transparently — Redpanda streams from S3 back to the client.

Tiered Storage Read Replicas (Enterprise) take this further: a separate Redpanda cluster reads tiered storage directly. This lets analytics-style consumers bypass the production cluster.

WASM Data Transforms

flowchart LR
    Producer --> Topic1["Topic input"]
    Topic1 --> Transform["WASM transform"]
    Transform --> Topic2["Topic output"]
    Topic2 --> Consumer

A WebAssembly module is registered against an input topic. For each record it can emit zero or more records to one or more output topics. Compiled with rpk transform build from Go or Rust source.

Iceberg Integration

Redpanda topics can be configured to write directly into an Apache Iceberg table:

flowchart LR
    Producer --> Topic["Redpanda topic"]
    Topic --> IcebergWriter["Iceberg Writer"]
    IcebergWriter --> ParquetFiles["Parquet files (S3)"]
    IcebergWriter --> Catalog["Iceberg catalog"]
    Trino["Trino / Spark / dbt"] --> Catalog
    Trino --> ParquetFiles

This pattern collapses the typical Kafka → Connect → Parquet pipeline into a single broker-native flow.

Performance Characteristics

Workload Numbers (Redpanda blog claims. Verify locally)
Single-broker NVMe sustained 1+ GB/s producer throughput
3-broker cluster R3 sustained 4–5 GB/s aggregate
p99 produce latency sub-10 ms typical, sub-1 ms achievable on well-tuned NVMe
Tiered Storage upload throughput proportional to S3 PUT rate (10–50 MB/s/partition)
WASM transform overhead ~50–100 µs/record for simple transforms

Vendor benchmarks

Redpanda's published numbers are vendor-controlled. Confluent has published counter-benchmarks. Always run openmessaging-benchmark with your real workload before sizing decisions.

Comparison Hooks

  • vs Kafka — same wire protocol. Redpanda wins on ops simplicity and tail latency. Kafka wins on ecosystem breadth and JVM tooling familiarity.
  • vs Pulsar — Pulsar's compute/storage split scales differently. Redpanda's per-partition Raft is simpler.
  • vs NATS — different protocol. NATS is lighter for sub-ms request-reply but lacks Kafka-API compatibility.

Security

Redpanda implements the Kafka security model (SASL, ACLs, TLS) and adds OIDC, RBAC, and mTLS for the admin API.

Authentication

Kafka API authentication

Mechanism Use Case
SASL/PLAIN Simple username/password (use only over TLS).
SASL/SCRAM-SHA-256, SCRAM-SHA-512 Salted-hash passwords. Preferred over PLAIN.
OAUTHBEARER OIDC tokens. Supports Keycloak, Auth0, Okta, AWS Cognito.
mTLS (x509) Cert-based. CN/SAN maps to a principal.

Admin API authentication

  • mTLS for admin API on :9644.
  • OIDC for Console UI.
  • HTTP basic for development.
# Enable SASL on Kafka API
redpanda:
  enable_sasl: true
  superusers: ["admin"]
  kafka_api_tls:
    - name: external
      enabled: true
      key_file: /etc/redpanda/certs/server.key
      cert_file: /etc/redpanda/certs/server.crt
      truststore_file: /etc/redpanda/certs/ca.crt
      require_client_auth: true

Console SSO

Redpanda Console supports OIDC, OAuth 2.0, and SAML for user login, with role mapping into the RBAC of the cluster.

Authorization

Kafka ACLs

ACLs apply to resources (topic, group, transactional-id, cluster) with operations (Read, Write, Create, Describe, Alter, Delete, Idempotent-Write, …).

rpk acl create --allow-principal 'User:orders-svc' \
  --operation read,describe \
  --topic 'orders.*'

rpk acl create --allow-principal 'User:orders-svc' \
  --operation read \
  --group 'orders-consumer'

rpk acl list

RBAC (Console / Cloud)

In Redpanda Cloud and via the Console, named roles group ACLs and assign them to users. Roles can be mapped from OIDC group claims for centralized identity.

Cluster super-users

Set superusers: [...] in redpanda.yaml for accounts that bypass ACLs (admin / break-glass). Limit and audit.

Encryption

In transit

  • TLS 1.2 / 1.3 on every listener: Kafka API, Admin API, Schema Registry, Pandaproxy, RPC (inter-broker).
  • mTLS can be required per listener.
  • TLS termination must be at Redpanda — sidecar TLS termination is not supported for the Kafka protocol because of the shared connection pool.
  • Cipher suites are configurable. Restrict to AEAD only for compliance frameworks.

At rest

  • Tiered storage object encryption: SSE-S3 by default. SSE-KMS with a customer-managed KMS key (CMK) for SOC 2 / HIPAA / PCI-DSS deployments. Configure via cloud_storage_kms_key_id.
  • Local NVMe encryption: OS-level (LUKS / dm-crypt or cloud-native EBS/Persistent Disk encryption).
  • BYOK (Bring Your Own Key) is supported in Redpanda Cloud Dedicated.

Audit Logging

Redpanda Enterprise has a dedicated Audit Log topic that records management operations:

  • Cluster config changes.
  • Topic create/alter/delete.
  • ACL changes.
  • User create/delete.
  • Authentication failures.

Audit logs are themselves a Kafka topic with a fixed schema, ready to ship into a SIEM.

Threat Model

Threat Mitigation
ACL bypass via super-user role Limit super-users. Rotate them. Tag with break-glass policy.
Tiered storage misconfigured S3 bucket policy Use SSE-KMS CMKs. Bucket policies that deny anonymous reads. CloudTrail object-level logging.
Console session hijack Short OIDC token TTLs. CSRF protections. HTTPS-only cookies.
Supply chain via 3rd-party Connect connectors Pin connector image versions. Review code. Isolate in network.
Unauthenticated Admin API Always enable mTLS or HTTP basic auth on :9644.
MITM on inter-broker RPC Enable RPC TLS — required for any prod cluster.
WASM transform compromise WASM modules run in-process. Keep them small + reviewed. Restrict who can register them.
Schema Registry spoofing Pin SR endpoints in clients. Require auth on the SR API.
Replay of produce requests Enable transactions / idempotent producer. Scope per-producer-id.
OIDC token theft Short TTLs, audience pinning, refresh-token rotation.
Cross-tenant data exfiltration in Cloud Cloud Dedicated → single-tenant. Cloud Serverless uses tenant-scoped namespaces.

Compliance

Redpanda Cloud Dedicated holds:

  • SOC 2 Type II
  • ISO 27001
  • HIPAA (with BAA)
  • PCI-DSS (Cloud Dedicated)
  • GDPR support (via region selection)

Self-managed Redpanda inherits whatever compliance you build around it.

CVE History (selected)

CVE Year Affected Summary
CVE-2024-39687 2024 Redpanda Console < 2.x patch Path traversal in Console static asset handler.
CVE-2023-2972 2023 Redpanda admin API Insufficient input validation on a debug endpoint.
CVE-2022-39253 2022 Redpanda < 22.2 Improper TLS handshake handling.

The canonical list lives at github.com/redpanda-data/redpanda/security/advisories.

Hardening Checklist

  • SASL enabled (preferably SCRAM-SHA-512 or OIDC).
  • mTLS on Kafka API, Admin API, Schema Registry, Pandaproxy, RPC.
  • TLS 1.3 only. Restrict cipher suites.
  • Tiered storage uses SSE-KMS with a CMK. Bucket has block-public-access set.
  • Super-users restricted to break-glass accounts. Audit reviewed quarterly.
  • OIDC integrated with corporate IdP for Console. SCIM provisioning if available.
  • Audit Logs topic shipped to SIEM (Enterprise).
  • WASM transform registration restricted via ACL.
  • Subscribed to Redpanda Security Advisories.
  • Helm chart pinned to Operator-blessed version.

Cross-references