Architecture¶

Ceph is a unified distributed storage system that delivers object, block, and file storage from a single RADOS (Reliable Autonomic Distributed Object Store) cluster. It uses intelligent daemons and the CRUSH algorithm to eliminate centralized lookup bottlenecks. This enables petabyte-to-exabyte scale with commodity hardware.

Core Component Diagram¶

graph TB
    subgraph Clients
        RBD[Ceph Block Device<br/>librbd / kernel RBD]
        RGW[Ceph Object Storage<br/>radosgw -- S3/Swift]
        CephFS[Ceph File System<br/>FUSE / kernel mount]
        LibR[Custom Apps<br/>librados]
    end

    subgraph RADOS["RADOS Storage Cluster"]
        MON[Ceph Monitor<br/>ceph-mon<br/>Paxos quorum]
        MGR[Ceph Manager<br/>ceph-mgr<br/>Metrics + Dashboard]
        OSD[Ceph OSD<br/>ceph-osd<br/>BlueStore backend]
        MDS[Ceph MDS<br/>ceph-mds<br/>CephFS metadata]
    end

    subgraph Storage
        DISK[(Local disks<br/>per OSD)]
    end

    RBD --> OSD
    RGW --> OSD
    CephFS --> MDS
    CephFS --> OSD
    LibR --> OSD
    MON -->|Cluster maps| OSD
    MON -->|Cluster maps| Clients
    MGR --> MON
    MDS --> OSD
    OSD --> DISK

RADOS Daemons¶

Ceph Monitor (ceph-mon)¶

Monitors maintain the master copy of the cluster map, a collection of five sub-maps that describe the entire cluster topology:

Map	Contents	View Command
Monitor Map	Cluster `fsid`, monitor addresses, epochs	`ceph mon dump`
OSD Map	Pool list, replica sizes, PG numbers, OSD statuses	`ceph osd dump`
PG Map	PG versions, up/acting sets, per-PG state, usage stats	`ceph pg dump`
CRUSH Map	Device list, failure domain hierarchy, placement rules	`ceph osd getcrushmap`
MDS Map	MDS ranks, metadata pool, active/standby state	`ceph fs dump`

Monitors achieve consensus via the Paxos algorithm. A quorum requires a strict majority (for example, 2 of 3, 3 of 5). Losing quorum makes the entire cluster unavailable for writes, which is why production deployments run at least 3 monitors across failure domains.

Ceph OSD Daemon (ceph-osd)¶

Each OSD manages a single storage device (typically one per physical disk). OSDs handle all read, write, replication, recovery, rebalancing, and scrubbing operations locally. Key responsibilities:

Data storage via BlueStore (the default backend since Luminous), which writes objects directly to raw block devices in a database-like structure (RocksDB metadata + WAL on SSD, object data on HDD or SSD).
Heartbeating neighboring OSDs and reporting state changes to monitors via MOSDBeacon messages.
Peer recovery when OSDs join or leave the acting set for a PG.
Scrubbing (light: daily metadata comparison. Deep: weekly bit-for-bit data comparison).

OSDs are statefully described as up/down (daemon running or not) and in/out (participating in data placement or not).

Ceph Manager (ceph-mgr)¶

The manager daemon provides monitoring, orchestration, and a plug-in framework. It hosts the Ceph Dashboard (web UI), Prometheus metrics exporter, and orchestrator modules (cephadm, Rook). At least one active manager is required. A standby takes over automatically if the active fails.

Ceph Metadata Server (ceph-mds)¶

MDS is only required for CephFS. It stores filesystem metadata (directories, ownership, permissions, timestamps) in memory for fast POSIX operations. MDS supports two deployment modes:

Active-standby HA: One active MDS serves a filesystem. Standbys take over on failure.
Active-active scaling: Multiple MDS daemons share the directory tree. They split busy subtrees and even individual directory shards across instances.

RADOS Gateway (radosgw)¶

The RGW daemon provides S3-compatible and Swift-compatible RESTful APIs on top of RADOS. It maintains its own user database, authentication, and access control. The unified namespace means data written via the S3 API is readable via the Swift API and vice versa.

CRUSH Algorithm and Placement Groups¶

CRUSH Overview¶

CRUSH (Controlled Replication Under Scalable Hashing) is the algorithm that determines where each object is stored. Unlike a central lookup table, every client and OSD computes placement independently using the same CRUSH map. This eliminates a single bottleneck.

Object-to-OSD Placement Flow¶

flowchart LR
    OBJ["Object<br/>(pool + object-id)"]
    HASH["Hash object-id<br/>mod PG count"]
    PG["Placement Group<br/>e.g. 4.58"]
    CRUSH["CRUSH algorithm<br/>+ failure domain"]
    OSDS["Acting Set<br/>[Primary, Secondary, ...]"]

    OBJ --> HASH --> PG --> CRUSH --> OSDS

The full computation:

Client provides the pool name and object ID.
Ceph hashes the object ID and takes the result modulo the number of PGs in the pool, yielding a PG ID (for example, 58).
The pool ID is prepended (for example, pool 4 -> PG 4.58).
CRUSH maps the PG to an ordered list of OSDs called the acting set. The first OSD is the primary and handles client writes. It forwards data to secondaries for replication.
Because CRUSH is deterministic, any client with the same cluster map produces the same result.

Placement Groups (PGs)¶

PGs are logical buckets that amortize the cost of per-object placement. Each pool has a configurable number of PGs. The PG autoscaler (default since Pacific) adjusts PG counts automatically based on pool usage and target PGs-per-OSD ratios.

Key concepts:

Acting Set: The ordered list of OSDs responsible for a PG. The first OSD is the primary.
Up Set: The subset of the acting set where OSDs are currently up. When an OSD in the acting set fails, Ceph remaps the PG and the next OSD becomes primary.
Peering: The process by which OSDs in the acting set of a PG agree on the consistent state of all objects. Peering must complete before the PG becomes active + clean.

Failure Domains¶

CRUSH uses a hierarchical topology (bucket types: osd, host, rack, row, room, root) to place replica copies across separate failure domains. A typical rule makes sure that no two replicas of the same PG land on the same host (or rack, if configured).

Rebalancing¶

When OSDs are added or removed, the cluster map changes and CRUSH recomputes placement. Only the PGs whose mapping changes migrate data. CRUSH is stable in that most PGs remain where they are, and new OSDs receive a proportional share without causing load spikes.

Pools and Erasure Coding¶

Replicated Pools¶

By default, pools use replication (size = 3, min_size = 2). The primary OSD writes the object and synchronously replicates to the remaining OSDs in the acting set. This provides the lowest write latency.

Erasure-Coded Pools¶

Erasure coding splits each object into K data chunks and M coding chunks, distributed across K+M OSDs. For example, a K=3, M=2 profile uses 5 OSDs and tolerates 2 failures with only 67% storage overhead (versus 200% for 3x replication).

graph LR
    OBJ["Object NYAN<br/>ABCDEFGHI"]
    subgraph EC["Erasure Encoding K=3, M=2"]
        D1["D1: ABC"]
        D2["D2: DEF"]
        D3["D3: GHI"]
        C1["C1: YXY"]
        C2["C2: QGC"]
    end
    OBJ --> D1
    OBJ --> D2
    OBJ --> D3
    OBJ --> C1
    OBJ --> C2
    D1 --> O1[(OSD 5)]
    D2 --> O2[(OSD 1)]
    D3 --> O3[(OSD 2)]
    C1 --> O4[(OSD 3)]
    C2 --> O5[(OSD 4)]

Reads require K chunks (any combination of data and parity). Writes go through the primary OSD, which encodes the payload and distributes chunks. Erasure-coded pools are best suited for cold data, backups, and large objects where storage efficiency outweighs write performance.

Replication vs Erasure Coding

Use replicated pools for hot data (RBD VM images, CephFS active directories) where write latency matters. Use erasure-coded pools for cold data (RGW archival, backup targets) where storage efficiency matters. The EC write path is heavier because the primary must encode and distribute shards.

Client Interfaces¶

Interface	Daemon/Lib	Access Method	Typical Use Case
RBD (Block)	`librbd`, kernel `rbd`	Block device (QEMU/libvirt, kernel)	VM disks, Kubernetes PVs
RGW (Object)	`radosgw`	S3 / Swift REST API	S3-compatible object storage
CephFS (File)	`ceph-mds`, `libcephfs`	POSIX mount (kernel/FUSE)	Shared filesystem, HPC
Custom	`librados`	Direct object API	Application-specific storage

BlueStore¶

BlueStore is the default OSD storage backend (replacing FileStore). It writes object data directly to a raw block device (no intervening filesystem) and stores metadata in an embedded RocksDB with a separate WAL device. Key advantages:

Direct I/O: Bypasses the Linux page cache for predictable performance.
WAL + DB separation: RocksDB metadata and WAL can be placed on faster SSD/NVMe while bulk data resides on HDD.
Checksums and compression: Per-object checksumming and optional inline compression (zlib, zstd, lz4).
Device encryption: Optional dmcrypt encryption per OSD.

Crimson (Experimental)

Crimson is a next-generation OSD implementation built on the Seastar framework. It targets improved CPU efficiency and lower latency. It is not yet production-ready as of the Reef/Squid release series.

Key Architectural Properties¶

No centralized gateway: Clients compute placement via CRUSH and talk directly to OSDs.
Self-healing: OSDs detect failures, trigger peering, and rebuild data from replicas automatically.
Elastic scaling: Add or remove OSDs. CRUSH rebalances only affected PGs.
Uniform hardware: Designed for commodity servers with local disks. No proprietary hardware is required.
Shared-nothing OSDs: Each OSD owns its disk independently. This prevents shared-disk contention.

Sources¶

Ceph Architecture Documentation
CRUSH Algorithm Paper: Sage Weil et al., "CRUSH: Controlled, Scalable, Decentralized Placement of Replicated Data"
BlueStore Design

How It Works¶

RADOS object store, CRUSH data placement, OSD write path, and recovery mechanics.

RADOS Architecture¶

flowchart TB
    subgraph Clients["Client Access"]
        RBD["RBD Client\n(block)"]
        RGW["RGW\n(S3/Swift)"]
        CephFS_C["CephFS Client\n(POSIX)"]
    end

    subgraph RADOS["RADOS (Core)"]
        PG["Placement Groups\n(PG)"]
        CRUSH_A["CRUSH Map\n(deterministic placement)"]
    end

    subgraph Daemons["Cluster Daemons"]
        MON["MON ×3+\n(quorum, cluster map)"]
        MGR["MGR ×2\n(metrics, dashboard)"]
        OSD1["OSD 1"]
        OSD2["OSD 2"]
        OSD3["OSD 3"]
        OSDN["OSD N"]
        MDS_D["MDS ×2+\n(CephFS metadata)"]
    end

    Clients --> RADOS
    RADOS --> Daemons
    CRUSH_A --> PG
    PG --> OSD1
    PG --> OSD2
    PG --> OSD3

    style RADOS fill:#c62828,color:#fff
    style Daemons fill:#1565c0,color:#fff

CRUSH Algorithm¶

The CRUSH (Controlled Replication Under Scalable Hashing) algorithm is what makes Ceph unique — clients calculate data placement directly, without a central lookup table.

flowchart LR
    Object["Object ID"] --> Hash["Hash\n(CRUSH)"]
    Hash --> PG_C["Placement Group\n(PG = hash mod num_pgs)"]
    PG_C --> CRUSH_C["CRUSH Rules\n(rack/host/OSD failure domains)"]
    CRUSH_C --> OSD_Set["OSD Set\n{OSD.4, OSD.17, OSD.29}"]

    style CRUSH_C fill:#e65100,color:#fff

Write Path¶

sequenceDiagram
    participant Client as Client
    participant Primary as Primary OSD
    participant Replica1 as Replica OSD 1
    participant Replica2 as Replica OSD 2
    participant Journal as WAL/DB (BlueStore)

    Client->>Primary: Write object
    Primary->>Journal: Write to WAL (journal)
    par Replicate
        Primary->>Replica1: Forward write
        Primary->>Replica2: Forward write
    end
    Replica1-->>Primary: ACK
    Replica2-->>Primary: ACK
    Primary-->>Client: Write complete
    Note over Primary: All replicas confirmed<br/>before client ACK (strong consistency)

Read Path¶

sequenceDiagram
    participant Client as Client
    participant MON as Monitor
    participant Primary as Primary OSD
    participant BlueStore as BlueStore (Disk)

    Client->>MON: Get cluster map (cached)
    Client->>Client: CRUSH(object) -> PG -> OSD set
    Client->>Primary: Read object (from primary OSD)
    Primary->>BlueStore: Lookup in RocksDB (omap)
    BlueStore-->>Primary: Object location on block device
    Primary->>BlueStore: Read data from block device
    Primary-->>Client: Return object data
    Note over Client: Client reads only from primary OSD<br/>for strong consistency (min_size=2)

Read path is simpler than writes because the primary OSD serves reads directly. The client uses the CRUSH algorithm to compute the OSD set for a given object, then contacts the primary OSD. The primary looks up the on-disk location of the object via RocksDB (stored in the metadata partition of BlueStore) and reads the data from the raw block device.

For replicated pools (size=3), reads go only to the primary OSD. For erasure-coded pools, reads can involve reconstructing data from multiple OSD shards if some are unavailable.

BlueStore (Default OSD Backend)¶

Component	Role
BlockDevice	Raw block device (HDD/SSD/NVMe) — no filesystem
RocksDB	Object metadata store
WAL	Write-ahead log (best on fast NVMe)
DB	RocksDB data (best on SSD)
Data	Object data (can be HDD)

Data Protection¶

Method	Overhead	Speed	Use Case
Replication (3×)	200%	Fast writes	Hot data, databases
Erasure Coding (4+2)	50%	Slower writes, fast reads	Cold data, archives
FastEC (Tentacle)	50%	Improved small I/O	General purpose

Sources¶

Benchmarks¶

Scope

Performance characteristics, scaling limits, and resource consumption for Ceph.

RADOS Bench Results¶

Operation	3 OSD (HDD)	3 OSD (SSD)	12 OSD (NVMe)
Seq Write	300-500 MB/s	1-2 GB/s	5-10 GB/s
Seq Read	400-600 MB/s	1.5-3 GB/s	8-15 GB/s
Random 4K Write	500-1,000 IOPS	10k-30k IOPS	100k+ IOPS
Random 4K Read	1,000-2,000 IOPS	20k-50k IOPS	200k+ IOPS

RBD (Block) Performance¶

Block Size	Throughput	IOPS	Latency (P99)
4K random	N/A	10k-50k	1-5ms
64K sequential	500MB-2GB/s	N/A	2-10ms
1M sequential	1-5 GB/s	N/A	5-20ms

Scaling Limits¶

Dimension	Limit	Notes
OSDs per cluster	10,000+	Tested by CERN
Total capacity	Exabytes	Linear scaling
Default pg_num per pool	64	Tune per pool based on OSD count
PGs per OSD	100-200 (recommended)	Performance degrades beyond 300
Objects per PG	Millions	No hard limit

Sourcing Status¶

Unsourced Performance Data

The performance numbers in this document are estimated from vendor documentation, community benchmarks, and engineering judgment. They do not represent controlled benchmarks with documented test conditions. Specific hardware configurations, software versions, and test methodologies were not recorded.

Use these figures as rough guidance only. For production capacity planning, run your own benchmarks against your specific workload and infrastructure.

Sources¶

Security¶

Ceph provides multiple layers of security: CephX authentication for daemon and client identity, a capability-based authorization system, encryption at rest, and TLS for network transport. When deployed via Rook on Kubernetes, additional pod security context and RBAC controls apply.

CephX Authentication¶

Protocol Overview¶

CephX is the native authentication protocol of Ceph, enabled by default. It uses shared secret keys to provide mutual authentication between clients and the cluster, similar in design to Kerberos:

A client contacts a Monitor and presents its user ID.
The Monitor generates a session key encrypted with the permanent secret key of the user and returns it to the client.
The client decrypts the session key (proving possession of the secret) and requests a ticket for OSD and MDS access.
The Monitor returns a ticket encrypted with the session key.
The client presents the ticket to OSDs and MDS daemons to authenticate ongoing operations.

Every message between client and daemon is signed with a ticket verifiable by all cluster daemons, because Monitors, OSDs, and MDS share a secret. Tickets expire based on a configurable TTL (auth_service_ticket_ttl, default 1 hour).

CephX Scope

CephX authenticates connections between Ceph clients and Ceph daemons. It does not encrypt data in transit or at rest by itself. For transport encryption, use the msgr2 protocol with TLS. For data at rest, use dmcrypt on OSDs.

Configuration¶

CephX is controlled by three configuration options (all default to cephx):

Setting	Scope	Default
`auth_cluster_required`	Daemon-to-daemon authentication	`cephx`
`auth_service_required`	Client-to-daemon authentication	`cephx`
`auth_client_required`	Daemon-to-client authentication	`cephx`

Disabling any of these (setting to none) is strongly discouraged because it exposes the cluster to man-in-the-middle attacks.

User and Capability System¶

Users¶

Ceph users are identified as client.<name> (for example, client.admin, client.glance, client.rbd). Each user has a secret key stored in a keyring file and a set of capabilities (caps) that define what the user can do.

Capability Syntax¶

Capabilities are expressed per daemon type with specific keywords:

Keyword	Meaning
`allow`	Precedes access settings. Implies `rw` for MDS
`r`	Read access
`w`	Write access
`x`	Execute (call class methods, conduct auth operations)
`class-read`	Subset of `x`. Call class read methods
`class-write`	Subset of `x`. Call class write methods
`*` / `all`	Full read, write, execute, and admin permissions
`profile rbd`	Predefined profile for RBD access
`profile osd`	Predefined profile for OSD-to-Monitor communication
`profile mgr`	Predefined profile for Manager daemon
`profile mds`	Predefined profile for MDS daemon

Capabilities can be scoped to specific pools:

# Grant read/write access to a single pool
ceph auth get-or-create client.app1 mon 'allow r' osd 'allow rw pool=myapp'

# Grant RBD profile access to an OpenStack Glance pool
ceph auth get-or-create client.glance mon 'profile rbd' osd 'profile rbd pool=images' mgr 'profile rbd pool=images'

Daemon Keyrings¶

Each daemon type has a default keyring location and minimal required capabilities:

Daemon	Keyring Location	Required Capabilities
`ceph-mon`	`$mon_data/keyring`	`mon 'allow *'`
`ceph-osd`	`$osd_data/keyring`	`mgr 'allow profile osd' mon 'allow profile osd' osd 'allow *'`
`ceph-mds`	`$mds_data/keyring`	`mds 'allow' mgr 'allow profile mds' mon 'allow profile mds' osd 'allow rwx'`
`ceph-mgr`	`$mgr_data/keyring`	`mon 'allow profile mgr' mds 'allow ' osd 'allow '`
`radosgw`	`$rgw_data/keyring`	`mon 'allow rwx' osd 'allow rwx'`

User Management Commands¶

# List all users
ceph auth ls

# Create a user with specific caps
ceph auth get-or-create client.app1 mon 'allow r' osd 'allow rw pool=data'

# Modify capabilities
ceph auth caps client.app1 mon 'allow rw' osd 'allow rwx pool=data'

# View user details
ceph auth get client.app1

# Delete a user
ceph auth del client.app1

Encryption at Rest¶

OSD Data Encryption (dmcrypt)¶

Ceph supports full-disk encryption on OSDs using Linux dm-crypt (LUKS). When enabled via ceph-volume:

The data device of each OSD is encrypted with a unique dmcrypt_key.
The dmcrypt_key is stored in a "lockbox" -- a special OSD monitored by the Ceph Monitors, encrypted with a cephx_lockbox_secret.
On OSD startup, the Monitor provides the lockbox secret, which decrypts the dmcrypt_key, which in turn unlocks the data device.

The lockbox key payload is structured as:

{
    "cephx_secret": "<CEPHX_SECRET>",
    "dmcrypt_key": "<DMCRYPT_KEY>",
    "cephx_lockbox_secret": "<LOCKBOX_SECRET>"
}

BlueStore Encryption

BlueStore also supports internal encryption of the WAL and DB devices via its own bluefs_buffered_io and key management, but the primary at-rest encryption mechanism remains dm-crypt at the block device level.

Network Transport Security¶

Messenger v2 (msgr2)¶

The Messenger v2 protocol of Ceph supports two modes:

crc mode: Provides CRC32C integrity checking and CephX authentication. Messages are authenticated but not encrypted.
secure mode: Adds full TLS-like encryption using the cephx session keys for key exchange. This encrypts all message payloads.

Configuration:

# Require secure mode for all connections
ms_cluster_mode = secure
ms_service_mode = secure
ms_client_mode = secure

RGW TLS¶

The RADOS Gateway supports TLS termination natively:

rgw_frontends = beast ssl_port=443 ssl_certificate=/etc/ceph/rgw.crt ssl_private_key=/etc/ceph/rgw.key

For deployments behind a load balancer (HAProxy, NGINX), TLS can be terminated at the load balancer with backend connections over plain HTTP or over secure msgr2.

RGW Bucket Policies and ACLs¶

The RADOS Gateway implements S3-compatible access control:

Bucket policies: JSON policy documents attached to buckets, supporting principal-based, IP-based, and VPC-endpoint-based conditions.
ACLs: Legacy S3 ACLs for backward compatibility (canned ACLs like private, public-read, authenticated-read).
User-level permissions: RGW maintains its own user database separate from CephX. RGW users have access keys and secret keys for S3 authentication (AWS Signature V4).

Rook Security Context¶

When Ceph is deployed via Rook on Kubernetes, the operator applies security contexts:

Pod Security Standards: Rook requires privileged or baseline Pod Security Standards for Ceph OSD pods (needed for block device access and dm-crypt).
RBAC: The Rook operator uses ClusterRoles with specific permissions for CRD management. Ceph CSI drivers have separate ServiceAccounts scoped to provisioning and publishing operations.
SecurityContextConstraints (OpenShift): Rook ships with SCC definitions for OSD, MGR, and MON pods.
Network Policies: Rook does not ship default NetworkPolicies. Administrators must create them to restrict inter-namespace traffic to only Ceph daemon communication ports (6789, 3300, 6800-7300).

Threat Model Summary¶

Threat	Mitigation
Unauthorized client access	CephX mutual authentication, capability scoping
Man-in-the-middle on cluster network	msgr2 `secure` mode, CephX message signatures
Data exposure from stolen disks	`dm-crypt` OSD encryption
Unauthorized S3 access	RGW access keys, bucket policies, IAM
Privilege escalation in Kubernetes	Rook RBAC, Pod Security Standards, dedicated ServiceAccounts
Replay attacks	CephX ticket TTL, session key rotation
Silent data corruption	BlueStore checksumming, scrubbing