Skip to content

Architecture

OpenStack is a distributed cloud operating system composed of independent services that communicate through a shared message bus (RabbitMQ) and a central identity service (Keystone). Each service owns its database and exposes a REST API. This microservices architecture enables incremental adoption but increases operational complexity compared to single-daemon platforms like OpenNebula.

See also: infrastructure/openstack/index, infrastructure/openstack/explanation, infrastructure/openstack/how-to-guides

Component Overview

graph TD
    subgraph Users["User Interfaces"]
        HORIZON["Horizon<br/>(Web Dashboard)"]
        CLI["OpenStack CLI"]
    end

    subgraph Core["Core Services"]
        KEYSTONE["Keystone<br/>(Identity)"]
        NOVA["Nova<br/>(Compute)"]
        NEUTRON["Neutron<br/>(Networking)"]
        CINDER["Cinder<br/>(Block Storage)"]
        GLANCE["Glance<br/>(Image)"]
        SWIFT["Swift<br/>(Object Storage)"]
    end

    subgraph Orchestration["Orchestration"]
        HEAT["Heat<br/>(Orchestration)"]
        MAGNUM["Magnum<br/>(Container Infra)"]
    end

    subgraph Infrastructure["Shared Infrastructure"]
        MQ["RabbitMQ<br/>(Message Bus)"]
        DB["MariaDB / Galera<br/>(Database)"]
        MEMCACHED["Memcached<br/>(Cache)"]
    end

    HORIZON --> KEYSTONE
    CLI --> KEYSTONE
    KEYSTONE --> DB
    NOVA --> MQ
    NOVA --> DB
    NEUTRON --> MQ
    NEUTRON --> DB
    CINDER --> MQ
    CINDER --> DB
    GLANCE --> DB
    SWIFT --> DB
    HEAT --> MQ
    HEAT --> DB
    NOVA --> NEUTRON
    NOVA --> CINDER
    NOVA --> GLANCE
    MAGNUM --> HEAT

Core Services

Keystone (Identity)

Keystone is the central authentication, authorization, and service catalog for OpenStack:

  • Authentication: Validates user identity via passwords, tokens, or federated identity (SAML, OIDC)
  • Authorization: Role-Based Access Control (RBAC) with per-project policies
  • Service Catalog: Registry of all OpenStack service endpoints (each service registers its API URL)
  • Multi-domain: Supports multiple identity backends (SQL, LDAP, Active Directory)
  • Token types: Fernet tokens (lightweight, symmetric-key) or JWT tokens
  • Every API request in OpenStack starts with Keystone token validation

Nova (Compute)

Nova manages the lifecycle of virtual machine instances:

  • nova-api: Accepts and validates REST API requests
  • nova-scheduler: Selects compute hosts based on filters and weights (similar to Kubernetes scheduling)
  • nova-conductor: Mediates database access for compute nodes (security proxy)
  • nova-compute: Runs on each hypervisor node, manages VM lifecycle via libvirt (KVM) or other drivers
  • nova-novncproxy / nova-serialproxy: Console access to instances
  • Supports live migration, resize, snapshot, and evacuate operations
  • Integrates with Neutron for networking, Cinder for volumes, Glance for images

Neutron (Networking)

Neutron provides network connectivity as a service between interface devices managed by other OpenStack services:

  • Networks and subnets: Virtual L2 segments with IP address management
  • Routers: Virtual L3 routing between subnets, with SNAT for external access
  • Floating IPs: One-to-one NAT from external networks to VMs
  • Security groups: Per-port stateful firewall rules
  • Plugins: ML2 (Modular Layer 2) with drivers for Open vSwitch, Linux bridge, SR-IOV, and vendor SDNs
  • Agents: L2 agent (per compute node), L3 agent (routing), DHCP agent, metadata agent

Cinder (Block Storage)

Cinder provides persistent block storage volumes to instances:

  • Volume creation, attachment, snapshot, and cloning
  • Backend drivers: LVM (local), Ceph RBD, NetApp, Dell EMC, AWS EBS, and dozens more
  • Volume types with extra specs (performance tiers, replication)
  • Availability zone awareness for volume placement
  • Backup to external backends (Swift, Ceph, POSIX)

Glance (Image)

Glance manages VM disk images:

  • Stores, discovers, and retrieves bootable disk images
  • Supports multiple formats: QCOW2, RAW, VHD, VMDK, ISO, AKI/ARI (kernel/ramdisk)
  • Backend stores: local filesystem, Swift, Ceph, S3, HTTP
  • Image properties and metadata for scheduling hints
  • Image sharing across projects

Swift (Object Storage)

Swift provides highly available, eventually consistent object storage:

  • Stores unstructured data objects organized in containers and accounts
  • Designed for durability and horizontal scalability (no single point of failure)
  • Ring-based data placement across storage nodes
  • Supports large objects (segmented uploads), versioning, and expiration
  • Independent of the SQL database (uses its own consistency layer)

Orchestration and Supporting Services

Heat (Orchestration)

Heat provisions OpenStack resources using declarative templates (HOT format -- Heat Orchestration Template):

  • Define stacks of interconnected resources (instances, networks, volumes, and others)
  • Supports auto-scaling via Ceilometer alarms
  • Handles dependency ordering and rollback on failure
  • Environment files separate parameter values from templates

Horizon (Dashboard)

Horizon is the web-based management interface:

  • Dashboard for managing instances, volumes, networks, and images
  • Project and user administration via Keystone integration
  • Extensible via Django plugins for additional services
  • Serves as the primary GUI for cloud operators and tenants

Magnum (Container Infrastructure)

Magnum provisions and manages Kubernetes, Swarm, and Mesos clusters on OpenStack:

  • Uses Heat templates to orchestrate cluster creation
  • Integrates with Neutron for cluster networking
  • Manages cluster lifecycle (create, update, delete)
  • Certificates and node group management

Shared Infrastructure

RabbitMQ (Message Bus)

All OpenStack services communicate asynchronously through RabbitMQ:

  • Implements AMQP 0-9-1 protocol
  • Topics-based routing (for example, notifications.info, nova)
  • Supports quorum queues for durability
  • HA via mirrored queues or quorum queues across a cluster

MariaDB / Galera (Database)

Each service runs its own database, typically MariaDB with Galera clustering for HA:

  • Galera provides multi-master synchronous replication
  • Each service has its own database schema (nova_api, nova_cell0, neutron, cinder, glance, keystone, heat)
  • Connection routed through a load balancer (HAProxy) with health checks

Memcached

Memcached provides caching for Keystone tokens and service catalog lookups:

  • Reduces database load for repeated authentication
  • Deployed as a pool of instances behind a load balancer
  • All services use memcache_servers configuration for shared caching

Request Flow: Instance Creation

sequenceDiagram
    actor User
    participant Horizon as Horizon / CLI
    participant Keystone as Keystone
    participant Nova as Nova API
    participant Sched as Nova Scheduler
    participant Compute as Nova Compute
    participant Glance as Glance
    participant Neutron as Neutron
    participant MQ as RabbitMQ

    User->>Horizon: Launch instance
    Horizon->>Keystone: authenticate (get token)
    Keystone-->>Horizon: token
    Horizon->>Nova: POST /servers (token + flavor + image)
    Nova->>Keystone: validate token
    Nova->>MQ: publish create_instance message
    Nova-->>Horizon: instance BUILD status

    Sched->>MQ: consume schedule message
    Sched->>Sched: filter + weigh compute hosts
    Sched->>MQ: publish selected host

    Compute->>MQ: consume build message for this host
    Compute->>Glance: download image
    Glance-->>Compute: image data
    Compute->>Neutron: create ports + allocate IPs
    Neutron-->>Compute: port info (MAC, IP)
    Compute->>Compute: create VM via libvirt/KVM
    Compute->>Nova: update instance status ACTIVE
    Nova-->>Horizon: instance ACTIVE
    Horizon-->>User: instance ready

Service Port Reference

Service Port Protocol
Keystone 5000 HTTP (API v3)
Nova API 8774 HTTP
Neutron 9696 HTTP
Cinder 8776 HTTP
Glance 9292 HTTP
Swift 8080 HTTP (proxy)
Heat 8004 HTTP
Horizon 80/443 HTTP/HTTPS
RabbitMQ 5672 AMQP
MariaDB 3306 MySQL
Memcached 11211 TCP

References

How It Works

Internal mechanisms, VM lifecycle, network data paths, and service interactions.

VM Provisioning Flow

sequenceDiagram
    participant User as User / Horizon
    participant KS as Keystone
    participant Nova_API as Nova API
    participant Sched as Nova Scheduler
    participant MQ as RabbitMQ
    participant Compute as Nova Compute
    participant Neutron as Neutron
    participant Glance as Glance
    participant Cinder as Cinder

    User->>KS: Authenticate (token)
    KS-->>User: Token
    User->>Nova_API: POST /servers (flavour, image, network)
    Nova_API->>KS: Validate token
    Nova_API->>Glance: Check image exists
    Nova_API->>Neutron: Allocate port
    Nova_API->>MQ: Schedule request
    MQ->>Sched: Pick host
    Sched->>Sched: Filter: RAM, CPU, disk, AZ
    Sched->>Sched: Weigh: balance, spread
    Sched->>MQ: Host selected
    MQ->>Compute: Build instance
    Compute->>Glance: Download image
    Compute->>Cinder: Attach volume (if any)
    Compute->>Neutron: Plug port → OVN
    Compute->>Compute: Launch KVM/QEMU domain
    Compute->>Nova_API: Instance ACTIVE

Neutron Networking (OVN)

flowchart TB
    subgraph Tenant["Tenant Network"]
        VM1["VM 1\n(10.0.0.2)"]
        VM2["VM 2\n(10.0.0.3)"]
    end

    subgraph OVN["OVN Data Path"]
        LS["Logical Switch\n(tenant subnet)"]
        LR["Logical Router\n(inter-subnet)"]
        GW["Gateway Router\n(external)"]
        SNAT["SNAT\n(floating IP)"]
    end

    subgraph Physical["Physical Network"]
        ExtNet["External Network\n(provider bridge)"]
    end

    VM1 --> LS
    VM2 --> LS
    LS --> LR
    LR --> GW
    GW --> SNAT
    SNAT --> ExtNet

    style OVN fill:#ef3e42,color:#fff

Ceph Integration

OpenStack Service Ceph Layer Purpose
Cinder RBD (RADOS Block Device) Persistent VM volumes
Glance RBD or RGW (Object) VM image storage
Nova RBD (ephemeral disks) Live migration enabler
Swift RGW (RADOS Gateway) S3-compatible object store
Manila CephFS Shared file systems

Multi-Region Architecture

flowchart TB
    subgraph Global["Global Services"]
        KS_G["Keystone\n(shared identity)"]
    end

    subgraph Region1["Region 1"]
        Nova1["Nova"]
        Neutron1["Neutron"]
        Cinder1["Cinder"]
        Compute1["Compute Hosts"]
    end

    subgraph Region2["Region 2"]
        Nova2["Nova"]
        Neutron2["Neutron"]
        Cinder2["Cinder"]
        Compute2["Compute Hosts"]
    end

    KS_G --> Region1
    KS_G --> Region2

Sources

Benchmarks

Scope

Performance characteristics, scaling limits, and resource consumption for OpenStack.

Nova (Compute) Performance

Operation Time Notes
VM boot (local) 10-30s Depends on image size
VM boot (Ceph) 15-45s RBD clone is fast
Live migration 5-60s Depends on memory size
Snapshot 10-120s Copy-on-write with Ceph

Scaling Benchmarks

Dimension Tested Notes
Compute nodes 500+ Production deployments
VMs per compute 100+ Depends on host resources
Total VMs 50,000+ Large cloud deployments
API requests/sec 1,000+ Keystone auth bottleneck

Neutron (Networking)

Feature OVS Performance OVN Performance
Port creation 5-10s 2-5s
Network create 1-3s 1-2s
Floating IP 2-5s 1-3s
East-west throughput 8-9 Gbps 9-10 Gbps

Sourcing Status

Unsourced Performance Data

The performance numbers in this document are estimated from vendor documentation, community benchmarks, and engineering judgment. They do not represent controlled benchmarks with documented test conditions. Specific hardware configurations, software versions, and test methodologies were not recorded.

Use these figures as rough guidance only. For production capacity planning, run your own benchmarks against your specific workload and infrastructure.

Sources

Security

Authentication: Keystone

Keystone is the central identity service for all OpenStack components:

Feature Details
Auth backends SQL (local), LDAP, Active Directory, SAML 2.0, OpenID Connect
Token types Fernet (symmetric, ephemeral), JWT (asymmetric)
Token scopes Project-scoped, domain-scoped, system-scoped
Federation Mapping external identity groups to Keystone roles/projects
MFA TOTP and application credentials with MFA rules 
graph TB
    subgraph Users
        Admin["Cloud Admin"]
        Tenant["Tenant Admin"]
        App["Application / Service"]
    end
    subgraph Keystone["Keystone Identity"]
        AuthN["Authentication\n(SQL/LDAP/OIDC)"]
        Token["Token Engine\n(Fernet/JWT)"]
        Policy["Policy Engine\n(policy.yaml)"]
        Catalog["Service Catalog"]
    end
    subgraph Services["OpenStack Services"]
        Nova["Nova"]
        Neutron["Neutron"]
        Cinder["Cinder"]
        Glance["Glance"]
        Swift["Swift"]
    end
    Admin --> AuthN
    Tenant --> AuthN
    App --> AuthN
    AuthN --> Token
    Token --> Policy
    Policy --> Catalog
    Catalog --> Services

Keystone Hardening

  • Rotate Fernet keys regularly (keystone-manage fernet_rotate)
  • Set short token lifetimes (1 hour for unscoped, 12 hours for scoped)
  • Disable the admin_token auth method in production
  • Enforce password complexity and rotation policies
  • Use scoped tokens (never unscoped) for API access
  • Enable audit middleware (CADF format) for all Keystone operations
  • Set file permissions on /etc/keystone/keystone.conf to 0600

Authorization: RBAC and Policy

Role-Based Access Control

OpenStack uses a policy engine (policy.yaml per service) to enforce RBAC:

Default Role Scope Typical Permissions
admin System Full access across all projects
member Project Create/manage resources in own project
reader Project Read-only access to project resources
Custom roles Variable Fine-grained per-service permissions

Project Isolation

Projects provide hard multi-tenancy boundaries:

  • Each project has isolated resources (VMs, networks, volumes, images)
  • Users can belong to multiple projects with different roles per project
  • Quotas enforce resource limits per project
  • Network namespaces prevent cross-project traffic

Secrets Management: Barbican

Barbican is OpenStack's key management service:

Capability Description
Symmetric keys AES encryption keys for Cinder volumes and Swift objects
Asymmetric keys RSA/EC key pairs for TLS and signing
Certificates X.509 certificate storage and lifecycle
Passphrases Generic secret storage
HSM integration PKCS#11 backends (Thales nCipher, AT&T ATAE)

Barbican Best Practices

  • Use the KMIP or Dogtag backend for enterprise deployments
  • Restrict API access via policy.json
  • Enable secret ACLs for cross-project secret sharing
  • Store TLS certificates and private keys in Barbican (not on filesystem)
  • Integrate with cert-manager for automated certificate lifecycle

Network Security: Neutron

Security Groups

Neutron security groups act as distributed stateful firewalls applied at the port level:

# Default-deny security group with explicit allow rules
openstack security group rule create --protocol tcp --dst-port 443 \
    --remote-ip 10.0.0.0/8 allow-https my-security-group
  • Default policy: deny all ingress, allow all egress
  • Stateful connection tracking (established connections auto-allowed)
  • Support for remote groups (allow traffic from another security group)
  • Port security and anti-spoofing enabled by default

Network Isolation

Mechanism Level Use Case
Projects L2/L3 Hard tenant isolation
VLANs L2 Simple network segmentation
VXLAN/GRE L2 overlay Scalable multi-tenant isolation
SR-IOV L2 Hardware-level NIC partitioning
OVS/DPDK L2/L3 High-performance programmable switching

Encryption

TLS Everywhere

Encrypt all inter-service communication:

  • API endpoints: TLS termination at HAProxy/Nginx, pass-through to services
  • RabbitMQ: TLS for all AMQP connections between services
  • MySQL/Galera: TLS for database connections and replication
  • Memcached: SASL authentication with TLS
  • Etcd: TLS for all client and peer connections

Encryption at Rest

Service Encryption Method
Cinder LUKS encryption via encrypt_key_id (Barbican integration)
Swift At-rest encryption with Barbican-managed keys
Nova Encrypted ephemeral storage (LUKS)
Glance Image encryption (less common. Rely on Cinder for volume-backed instances)

Compliance and Audit

CIS Benchmarks for OpenStack

CIS OpenStack Benchmark covers:

  • API endpoint configuration and TLS enforcement
  • Keystone hardening (token lifetimes, password policies)
  • Network security (security groups, segmentation)
  • File permissions (config files set to 0600/0640)
  • Database security (Galera TLS, restricted access)
  • Logging and monitoring (CADF audit events)

Audit Logging

  • Keystone CADF events: Track authentication, token operations, role assignments
  • Nova audit: Instance creation/deletion, metadata changes
  • Neutron audit: Security group changes, network creation
  • Centralized logging: Forward to ELK/Loki/Grafana for analysis

Hardening Checklist

  • Disable admin_token auth in Keystone production config
  • Rotate Fernet keys on a regular schedule
  • Enable TLS on all API endpoints, RabbitMQ, and database connections
  • Use Barbican for all secrets and key management
  • Apply default-deny security groups to all new ports
  • Configure project quotas to prevent resource exhaustion
  • Enable CADF audit logging across all services
  • Set config file permissions to 0600 (Keystone, Nova, and others)
  • Use LDAP/OIDC for user authentication (not local SQL)
  • Enable password complexity and rotation policies in Keystone
  • Implement network segmentation (separate management, tenant, storage networks)
  • Patch all services regularly — follow OpenStack security advisories
  • Run CIS benchmark compliance checks with OpenSCAP

Known Pitfalls

Pitfall Risk Mitigation
admin_token enabled Unauthenticated admin access Disable in production
Unencrypted RabbitMQ Inter-service credential interception Enable TLS on all AMQP connections
Overly broad security groups VM exposure to unauthorized traffic Default-deny with explicit allow rules
Unrotated Fernet keys Compromised token encryption Regular key rotation schedule
Shared MySQL credentials Database access escalation Per-service accounts with least privilege
No audit logging Undetected unauthorized actions Enable CADF events, centralize logs