Skip to content

Explanation

Context

Why Proxmox VE works the way it does: the platform stack, the cluster file system and corosync design that everything else stands on, how HA and the new CRS scheduler actually decide placements, the storage and backup pipelines, the SDN stack, identity and firewall, and the threat model of a hypervisor whose management plane is part of the product.

Platform Architecture

PVE is not a hypervisor kernel — it is a Debian distribution whose packages assemble into a hypervisor appliance. Each node runs the same set:

  • Debian 13 Trixie base with Proxmox's own kernel (7.0 stable default in 9.2; opt-in 6.14/6.8 older series remain installable), QEMU 11.0 for KVM virtual machines, LXC 7.0 for system containers.
  • Management plane: pveproxy (web UI + REST API on TCP 8006, Perl Mojo backend migrating toward Rust), pvedaemon, pvestatd (status broadcasting), pvescheduler (jobs: backups, replication, cron-like), spiceproxy for remote consoles (noVNC/SPICE, RDP-based viewer on roadmap).
  • Guest abstraction: VMs are QEMU processes with config files in /etc/pve/qemu-server/<vmid>.conf; containers are LXC with /etc/pve/lxc/<vmid>.conf. Same tooling (qm vs pct), same storage layer (pvesm), same backup pipeline (vzdump), same API — the unification is PVE's core design decision.
flowchart TB
    subgraph Node["PVE node (repeated N times)"]
        GUI["pveproxy :8006<br/>GUI + REST API"]
        DM["pvedaemon / pvestatd / pvescheduler"]
        QEMU["QEMU/KVM VMs"]
        LXC["LXC containers"]
        STOR["Storage plugins (pvesm)<br/>ZFS / Ceph / LVM / dir / NFS / PBS"]
        CFSCONF["/etc/pve (pmxcfs)"]
        GUI --> DM --> QEMU & LXC
        DM --> STOR
        CFSCONF -.config source.-> DM
    end
    subgraph Cluster["Cluster layer"]
        CORO["corosync 3<br/>UDP 5405-5412, up to 8 links"]
        HA["pve-ha-lrm + pve-ha-crm<br/>watchdog fencing"]
        CRS["Cluster Resource Scheduler<br/>dynamic load balancing (9.2)"]
    end
    CFSCONF <-->|"real-time config replication"| CORO
    CORO --> HA --> CRS

The Cluster File System: pmxcfs

Every design decision in a PVE cluster flows from one choice: cluster configuration lives in a database-backed FUSE filesystem, /etc/pve, replicated in real time by corosync to every node. Consequences:

  • One source of truth per cluster. Edit a VM config on any node; all nodes see it instantly. No config drift, no per-node "vCenter sync".
  • Strong consistency checks — duplicate VM IDs are impossible cluster-wide; a distributed locking mechanism serializes cluster-wide operations.
  • Quorum-gated writes. pmxcfs is read-only when a node loses quorum — a partitioned node cannot start or change guests, trading availability for split-brain safety. This is why two-node clusters need a QDevice (a third corosync vote on separate hardware) and why pvecm expected 1 exists as a conscious override for recovery, not for normal operation.
  • Size bound: the database is capped at 128 MiB (RAM copy) — several thousand VM configs fit; unbounded metadata does not.

Corosync requirements follow from the design: UDP 5405-5412 between all nodes, synchronized clocks, SSH (TCP 22) for node joins, a dedicated NIC recommended (low latency matters more than bandwidth), up to 8 redundant links, and a known corosync-over-bonds failure mode — a single logical link on a bond can mask a partition, so true redundancy means a second link on separate physical networking.

HA, Fencing, and the Cluster Resource Scheduler

HA in PVE is watchdog fencing, not just restart-on-fail. Every HA-managed node holds a hardware watchdog (or software fallback); pve-ha-lrm (per node) must continuously prove liveness. If a node stops feeding the watchdog — kernel hang, power loss, network partition confirmed by corosync — the watchdog resets the box, guaranteeing a dead node cannot keep a VM running while the cluster restarts it elsewhere. pve-ha-crm (one elected master) then restarts the node's HA services per their configured states.

Since 9.2 the scheduler story is explicit: the Cluster Resource Scheduler got a dynamic load-balancing mode that uses real-time node and guest utilization metrics to migrate HA-managed guests and reduce cluster-wide imbalance — tunable via HA panel and datacenter options, with the roadmap extending balancing to non-HA guests and richer affinity rules (soft/hard, topology-aware). Maintenance got safer too: ha-manager disarm-ha / arm-ha cluster-wide commands pause HA reactions (resource state preserved) so planned network work does not trigger mass fencing.

Storage Stack

One abstraction (/etc/pve/storage.cfg) fronts every storage type — local ZFS pools, directory/BTRFS, NFS/CIFS, LVM/LVM-thin, iSCSI/FC, Ceph RBD/CephFS, ZFS-over-iSCSI, and Proxmox Backup Server. The full capability matrix lives in Reference. The design-relevant points:

  • ZFS is the first-class local filesystem: zvols for VM disks, native snapshots, compression, RAID-Z, and PVE's storage replication (pvesr) — scheduled, incremental, snapshot-based replication of guest volumes between nodes for cheap DR without shared storage.
  • Ceph is the hyperconverged answer: pveceph installs and integrates Ceph (Squid 19.2.3 or Tentacle 20.2.1 on 9.2) so RBD pools and CephFS appear as first-class storages; every node is MON/MGR/OSD capable. RAM budget: ~1 GB per TB of used storage for both ZFS and Ceph.
  • Snapshots diverge by storage: native on ZFS/LVM-thin/Ceph/RBD; on file-based storages historically via qcow2 internals (slow — minutes on large disks, blocking the VM). PVE 9 introduced snapshots-as-volume-chains (tech preview; roadmap promotes it to GA on LVM-thick, Directory, NFS, CIFS) where each snapshot is a separate volume layered in a chain — online removal of the top-most snapshot included, and the foundation for extending replication beyond ZFS.

Backup Pipeline

vzdump is the single backup engine behind both legacy file backups and PBS:

  • Modes: stop (downtime, most consistent), suspend (very old fallback), snapshot (live, via storage snapshots — the default for modern setups).
  • Backup fleecing (PBS): changed blocks during backup go to a temporary local fleecing image so a busy guest never stalls behind a slow backup target.
  • Proxmox Backup Server replaces the tar/vzdump-file world with a deduplicated, compressed, optionally encrypted chunk store: incremental forever backups, cross-VM global dedup, Zstd compression, integrity via checksummed chunks, live restore (boot the guest while its disk still restores), single-file restore for VM images, S3-compatible datastores (since PBS 4.0/4.1), and Tape support. PBS 4.2 (Apr 2026) is current.

SDN, Identity, and Firewall

  • SDN (production-graduated since 8.x, still gaining features): zones (Simple, VLAN, QinQ, EVPN) contain VNets with subnet/IPAM management; 9.2 added fabric protocols — WireGuard and BGP — as first-class fabrics, route maps and prefix lists for BGP/EVPN filtering, OSPF route redistribution, and IPv6 EVPN underlay. Roadmap: DHCP IPAM stabilization, NetBox/phpIPAM/PowerDNS plugins, explicit VRFs, L7 HTTP/TLS ingress with ACME certs, and topology visualizations.
  • Identity: authentication realms — Linux PAM, Proxmox VE auth server (internal password store), LDAP/Active Directory, OpenID Connect — compose with RBAC (roles bound to paths like /vms/101 or pools) and API tokens (scoped, separable from user password, ideal for automation). TOTP/WebAuthn two-factor is built in.
  • Firewall: distributed, stateful, three levels (datacenter, node, guest) with security groups, IP sets, and log rendering; rules compile to iptables today with an nftables implementation in tech preview (8.2+) slated to replace it. SDN integration will auto-generate IP sets from VNets.

Engineering Direction

Two multi-year migrations shape the codebase: backend components are moving from Perl to Rust through the perlmod interop layer (new services like PBS and PDM are Rust-first), and the ExtJS web interface is moving to the Rust/Yew-based Proxmox widget toolkit that already powers the PVE mobile UI, PDM, and Mail Gateway quarantine. The "Cattle and Pets" project targets large estates: cluster-wide operations center, semi-automated ordered Ceph upgrades, bulk guest operations, large-cluster GUI performance.

Threat Model

Surface Reality Control
Management plane (8006, SSH 22) The hypervisor IS the management product — console access is root-equivalent Dedicated mgmt VLAN, PAM/PVE realms + TOTP/WebAuthn 2FA, RBAC least-privilege, API tokens instead of passwords for automation
Cluster bus (corosync 5405-5412) A node joining the corosync ring can influence config replication Isolated cluster NIC/VLAN; corosync supports encryption keys (/etc/corosync/authkey); restrict ring to cluster nodes
Guest escape QEMU/LXC share the host kernel (LXC especially) KVM for hard multi-tenancy; LXC for trusted workloads; apply AppArmor (LXC default), keep kernel/QEMU patched via subscription
Backups PBS chunks are the crown jewels; theft of unencrypted chunks exposes data Enable PBS encryption at the datastore/datastore-key level; protect PBS separately; test restore-path key custody
Quorum attacks / misoperation pvecm expected N overrides split-brain protection; HA fencing can power-cycle nodes Treat expected-quorum overrides as break-glass; watchdog discipline; disarm-ha only for planned windows
Supply chain Debian + Proxmox repos; no-subscription repo is slower-tested Enterprise repo (any subscription incl. Community tier) for tested rollouts; offline updates for air-gapped sites

Guest Storage and Backup Data Flow

flowchart LR
    G["Guest (VM / CT)<br/>virtio-scsi-single + iothread<br/>discard=on, ssd=1"]
    subgraph NodeS["Node storage layer"]
        QM["qm / pct volume layer<br/>(qemu-server config in /etc/pve)"]
        PLUG["pvesm storage plugins<br/>storage.cfg per cluster"]
        QM --> PLUG
    end
    subgraph Backends
        Z["ZFS zvol<br/>+ pvesr replication"]
        C["Ceph RBD<br/>hyperconverged"]
        F["file-based<br/>(qcow2 / volume chain)"]
    end
    subgraph Backup["Backup pipeline (vzdump)"]
        SNAP["snapshot mode<br/>(storage snapshot or chain)"]
        FLEECE["fleecing image<br/>(absorbs guest writes)"]
        PBS["PBS chunk store<br/>dedup + zstd + encryption"]
    end
    G --> QM
    PLUG --> Z & C & F
    SNAP --> FLEECE --> PBS
    Z & C & F -.snapshot source.- SNAP

Three consequences worth internalizing: the config file in /etc/pve is the single authority for what a guest's disks are (fixing "wrong disk" problems means fixing the config, not the hypervisor); snapshot semantics differ per backend, which is why 9.x is investing in volume chains to make file-based storages behave like ZFS/Ceph; and the fleecing image is what makes PBS backups invisible to busy guests — without it, a backup window is an I/O ceiling.

Live Migration

Migration moves a running guest between nodes: QEMU iterates memory while tracking dirty pages, copies them to the target, then pauses the source and transfers the residual delta — sub-second stop time in healthy clusters. Design constraints follow directly:

  • Same CPU vendor, same architecture. The guest sees a vCPU model; migrating Intel to AMD (or x86 to arm64) would fault on unsupported instructions. Custom CPU models (9.2 GUI) exist precisely to pin a portable feature set across heterogeneous-but-same-vendor nodes.
  • Shared storage vs local storage. With shared storage (Ceph, NFS, FC) only RAM/CPU state moves. With local storages, --with-local-disks adds a storage-migration phase (NBD copy of disks while running) — workable, but it is a data copy and slows with disk size.
  • HA interplay: HA-managed guests migrate through the CRS; the 9.2 load balancer uses the same machinery continuously, so a healthy cluster is always gently re-balancing by choice.

Performance Levers (and why they exist)

  • virtio-scsi-single + iothread + discard=on + ssd=1 — the standard disk quartet on ZFS/Ceph: one request queue per disk that can thread independently, TRIM passthrough so thin layers reclaim space, and SSD semantics so guest schedulers don't over-optimize.
  • CPU type: host exposes all host features (fastest, blocks cross-vendor migration); x86-64-vX baselines for portability; custom models for the middle ground.
  • Ballooning trades guest RAM against host pressure — set a realistic minimum, since ballooning shrinks guest-visible memory under host contention.
  • Multiqueue on virtio-net for high-PPS guests (queues should roughly match guest vCPUs).
  • NUMA + CPU pinning for latency-sensitive or large VMs on multi-socket hosts; taskset-style vCPU pinning trades scheduler flexibility for cache locality.
  • pveperf is the shipped first-guess tool: if fsync latency is poor, no guest tuning will save you — fix the storage layer first.

Container or VM? The Trust Boundary

LXC containers are kernel-shared: they boot in milliseconds, cost ~nothing, and suit trusted workloads (services, CI runners, homelab apps) — but a container escape lands on the hypervisor kernel, and the mitigation (unprivileged mapping, AppArmor, seccomp, no unnecessary nesting) is a hardening posture, not a boundary. KVM guests have a hardware boundary. The operational rule: default to unprivileged LXC for infrastructure you control; choose KVM whenever the workload is untrusted, licensed-per-VM, or needs its own kernel (SELinux policies, custom sysctls, out-of-tree modules).

Why It Won the VMware Exodus

The 2024 Broadcom licensing shock made thousands of vSphere estates re-platform. PVE's win was combination, not individual superiority: native ESXi import wizard (registered VMs, disks, network config — HN 245 points), per-socket pricing independent of cores, all-features-free AGPL, and a data path (PBS) that replaced Veeam for many. The trade: vCenter-equivalent multi-site management arrived only with PDM 1.0 (Dec 2025), and very large estates still lack some operational polish the "Cattle and Pets" roadmap targets.

Sources