Architecture¶

Kubernetes follows a declarative, state-driven architecture. A cluster consists of a control plane (one or more machines running the management components) and worker nodes (machines running user workloads). All state is stored in etcd, and every component communicates through the API server.

Cluster Architecture Overview¶

graph TD
    subgraph ControlPlane["Control Plane"]
        API["kube-apiserver<br/>:6443"]
        ETCD["etcd<br/>:2379-2380"]
        SCHED["kube-scheduler<br/>:10259"]
        CM["kube-controller-manager<br/>:10257"]
        DNS["CoreDNS"]
    end

    subgraph Node1["Worker Node 1"]
        KUBELET1["kubelet<br/>:10250"]
        PROXY1["kube-proxy"]
        CRI1["CRI Runtime<br/>(containerd/CRI-O)"]
        POD1A["Pod A"]
        POD1B["Pod B"]
    end

    subgraph Node2["Worker Node 2"]
        KUBELET2["kubelet<br/>:10250"]
        PROXY2["kube-proxy"]
        CRI2["CRI Runtime<br/>(containerd/CRI-O)"]
        POD2A["Pod C"]
        POD2B["Pod D"]
    end

    API <--> ETCD
    SCHED --> API
    CM --> API
    DNS --> API
    KUBELET1 --> API
    PROXY1 --> API
    KUBELET2 --> API
    PROXY2 --> API
    KUBELET1 --> CRI1
    CRI1 --> POD1A
    CRI1 --> POD1B
    KUBELET2 --> CRI2
    CRI2 --> POD2A
    CRI2 --> POD2B

Control Plane Components¶

kube-apiserver¶

The API server is the front end for the Kubernetes control plane. It exposes the Kubernetes HTTP API on port 6443 and is the only component that communicates directly with etcd.

Validates and stores all API objects (pods, services, deployments, and others)
Serves the REST API consumed by kubectl, controllers, and external tools
Handles authentication, authorization (RBAC), and admission controllers
Scales horizontally by running multiple replicas behind a load balancer
All other control plane components watch the API server, not etcd directly

etcd¶

etcd is a distributed, consistent key-value store that holds all cluster state:

Stores every object definition, cluster config, and dynamic state
Uses the Raft consensus algorithm for HA (quorum of 3 or 5 members)
Listens on TCP 2379 (client) and 2380 (peer communication)
Only the API server communicates with etcd directly
Backups are critical. Use etcdctl snapshot save regularly

etcd Quorum

An etcd cluster requires a majority (quorum) to accept writes. A 3-node cluster tolerates 1 failure. A 5-node cluster tolerates 2. Never run production etcd with fewer than 3 members.

kube-scheduler¶

The scheduler watches for newly created pods that have no assigned node and selects an appropriate node for them:

Filtering: Eliminates nodes that do not meet requirements (resources, taints, node selectors)
Scoring: Ranks remaining nodes using priority functions (resource balance, affinity, anti-affinity)
Binding: Writes the selected node name to the pod object via the API server
Supports custom schedulers and scheduler extensions
Listens on port 10259 for health probes

kube-controller-manager¶

Runs the core reconciliation controllers in a single binary:

Node Controller: Monitors node health, responds to node failures
ReplicaSet Controller: Maintains desired replica count for pods
Deployment Controller: Manages rolling updates and rollbacks
StatefulSet Controller: Ordered creation/deletion of stateful pods
Job Controller: Manages batch job completion
EndpointSlice Controller: Populates service endpoint data
Service Account Controller: Creates default service accounts for new namespaces
Namespace Controller: Cleans up resources when namespaces are deleted

Each controller is a reconciliation loop that watches the API server for changes and drives current state toward desired state.

CoreDNS¶

CoreDNS provides cluster-wide DNS resolution for services and pods:

Resolves service names to cluster IPs (for example, my-service.my-namespace.svc.cluster.local)
Supports DNS-based service discovery for headless services
Configurable via a Corefile (custom DNS zones, forwarding, rewrites)
Deployed as a Deployment in the kube-system namespace

Node Components¶

kubelet¶

The kubelet runs on every node and is responsible for pod lifecycle:

Watches the API server for pods assigned to its node
Pulls container images via the CRI runtime
Reports node and pod status back to the API server
Executes liveness, readiness, and startup probes
Mounts volumes (CSI) and configures pod networking (CNI)
Listens on port 10250 for API server health checks and exec/log requests

kube-proxy¶

kube-proxy maintains network rules on every node for service routing:

Implements the Service abstraction by managing iptables or IPVS rules
Routes traffic from service ClusterIP to backend pod IPs
Supports three modes: iptables (default), IPVS (high scale), userspace (legacy)
Watches the API server for changes to services and endpoints

CRI Runtime (containerd / CRI-O)¶

The container runtime interface (CRI) decouples kubelet from specific runtimes:

containerd: Most common runtime. CNCF graduated project
CRI-O: Lightweight runtime optimized for Kubernetes
Both implement the CRI API (gRPC) that kubelet calls
Container image pulling, container creation, and execution are delegated here
Dockershim was removed in Kubernetes v1.24

Container Network Interface (CNI)¶

CNI plugins handle pod network configuration:

Each pod gets its own IP address
Plugins: Calico, Cilium, Flannel, Weave, AWS VPC CNI, GKE Dataplane V2
Responsible for pod-to-pod connectivity within and across nodes
NetworkPolicy enforcement is plugin-dependent

Container Storage Interface (CSI)¶

CSI provides a standard interface for exposing storage to containers:

Replaces in-tree volume plugins (migration ongoing since v1.25)
Third-party drivers: AWS EBS CSI, GCE PD CSI, Azure Disk CSI, Ceph CSI
Handles volume provisioning, attachment, mounting, snapshots, and cloning

Container Runtime Interface (CRI)¶

The CRI is the gRPC API between kubelet and the container runtime:

Defines services: RuntimeService (container lifecycle) and ImageService (image management)
Enables pluggable runtimes without modifying kubelet source code
Supports both container runtimes and sandbox runtimes (Kata, gVisor)

Request Flow: Pod Creation¶

sequenceDiagram
    actor User
    participant API as kube-apiserver
    participant etcd as etcd
    participant CM as controller-manager
    participant Sched as kube-scheduler
    participant Kubelet as kubelet
    participant CRI as CRI runtime

    User->>API: kubectl apply -f pod.yaml
    API->>API: authenticate + authorize (RBAC)
    API->>API: run admission controllers (mutating, validating)
    API->>etcd: persist pod spec (nodeName="")
    etcd-->>API: confirmed
    API-->>User: accepted

    CM->>API: watch for new pods
    Note over CM: ReplicaSet controller ensures desired count

    Sched->>API: watch for unassigned pods
    API->>Sched: notify pod with nodeName=""
    Sched->>Sched: filter + score nodes
    Sched->>API: bind pod to node (set nodeName)
    API->>etcd: persist binding

    Kubelet->>API: watch for pods on its node
    API->>Kubelet: pod assigned to this node
    Kubelet->>CRI: pull image + create container
    CRI-->>Kubelet: container started
    Kubelet->>API: update pod status (Running)
    API->>etcd: persist status

Key Ports¶

Component	Port	Protocol	Purpose
kube-apiserver	6443	TCP	Kubernetes API
etcd	2379-2380	TCP	Client and peer communication
kubelet	10250	TCP	API server health, exec, logs
kube-scheduler	10259	TCP	Health probes
kube-controller-manager	10257	TCP	Health probes
kube-proxy	10256	TCP	Health probes

References¶

How It Works¶

Desired-state reconciliation, pod lifecycle, scheduling, networking, and storage internals.

Desired-State Reconciliation Loop¶

sequenceDiagram
    participant User as User / CI
    participant API as kube-apiserver
    participant ETCD as etcd
    participant Ctrl as Controller Manager
    participant Sched as Scheduler
    participant KL as kubelet (Node)
    participant CRI as containerd

    User->>API: kubectl apply -f deployment.yaml
    API->>ETCD: Store desired state
    Ctrl->>API: Watch: new Deployment
    Ctrl->>API: Create ReplicaSet
    Ctrl->>API: Create Pod specs
    Sched->>API: Watch: unscheduled Pods
    Sched->>Sched: Score nodes (resources, affinity, taints)
    Sched->>API: Bind Pod → Node
    KL->>API: Watch: Pod assigned to my node
    KL->>CRI: Create container sandbox
    CRI->>CRI: Pull image, start containers
    KL->>API: Update Pod status: Running

    loop Reconciliation
        Ctrl->>API: Watch: actual vs desired state
        Ctrl->>Ctrl: If replicas < desired → create more Pods
        Ctrl->>Ctrl: If replicas > desired → delete surplus
    end

Pod Lifecycle¶

stateDiagram-v2
    [*] --> Pending: Pod created
    Pending --> Running: Container started
    Running --> Succeeded: All containers exit 0
    Running --> Failed: Container exits non-zero
    Running --> Unknown: Node unreachable
    Failed --> [*]: Not restarted
    Succeeded --> [*]: Job complete
    Unknown --> Running: Node recovers
    Unknown --> Failed: Node dead (grace period)

    state Running {
        [*] --> Init: Init containers run sequentially
        Init --> Ready: Readiness probe passes
        Ready --> NotReady: Readiness probe fails
        NotReady --> Ready: Probe passes again
    }

Networking Model¶

The 4 Networking Rules¶

Pod-to-Pod: Every Pod gets its own IP. All Pods can communicate without NAT.
Pod-to-Service: Services provide stable virtual IPs (ClusterIP) backed by iptables/IPVS rules.
External-to-Service: LoadBalancer, NodePort, or Ingress/Gateway API expose services.
Pod-to-External: Pods can get to external networks via SNAT.

flowchart TB
    subgraph Cluster["Kubernetes Cluster"]
        subgraph Node1["Node 1"]
            P1["Pod A\n10.244.1.2"]
            P2["Pod B\n10.244.1.3"]
            KP1["kube-proxy\n(iptables/IPVS)"]
        end

        subgraph Node2["Node 2"]
            P3["Pod C\n10.244.2.2"]
            P4["Pod D\n10.244.2.3"]
            KP2["kube-proxy"]
        end

        SVC["Service: my-svc\nClusterIP: 10.96.0.10\n→ Pod A, Pod C"]

        CNI["CNI Plugin\n(Calico/Cilium/Flannel)\nPod-to-Pod routing"]
    end

    External["External\nTraffic"] -->|"LoadBalancer /\nNodePort"| SVC
    SVC -->|"iptables rules"| P1
    SVC -->|"iptables rules"| P3
    P1 <-->|"CNI overlay"| P3
    P2 <-->|"CNI overlay"| P4

    style Cluster fill:#326ce5,color:#fff

Storage Architecture¶

flowchart LR
    Pod["Pod"] --> PVC["PersistentVolumeClaim\n(request: 10Gi)"]
    PVC --> PV["PersistentVolume\n(10Gi, RWO)"]
    PV --> SC["StorageClass\n(provisioner: ebs.csi)"]
    SC --> CSI["CSI Driver\n(EBS, GCE PD, Ceph)"]
    CSI --> Disk["Cloud Disk\nor Storage"]

    style PVC fill:#326ce5,color:#fff

Scheduling Algorithm¶

Phase	Operation
Filtering	Eliminate nodes that don't meet Pod requirements (resources, taints, affinity)
Scoring	Rank remaining nodes: LeastRequestedPriority, BalancedResourceAllocation, NodeAffinityPriority, PodTopologySpread
Binding	Assign Pod to highest-scoring node
Preemption	If no node fits, evict lower-priority Pods

Sources¶

Benchmarks¶

Scope

Kubernetes scalability limits, API server performance, etcd throughput, and scheduling benchmarks.

Official Scalability Targets (SIG Scalability)¶

Kubernetes officially tests and targets these limits per cluster:

Dimension	Target	Notes
Nodes	5,000	Tested by SIG Scalability
Pods	150,000	30 pods/node avg
Pods per node	110	Kubelet default `maxPods`
Services	10,000
Endpoints per Service	5,000	Beyond this, use EndpointSlices
Namespaces	10,000
ConfigMaps	30,000
Secrets	30,000
Total API objects	~300,000	etcd storage limit

API Server Performance¶

Metric	Target SLO	Notes
API request latency (mutating, P99)	< 1s	At 5000-node scale
API request latency (non-mutating, P99)	< 5s	For resource-list calls
API request latency (P50)	< 100ms	Typical operations
Startup latency (P99)	< 5s	Pod ready from API call

etcd Performance¶

Cluster Size	WAL fsync P99	Read latency P99	Write QPS	Storage
< 100 nodes	< 5ms	< 10ms	1,000	2Gi
100-500 nodes	< 10ms	< 25ms	5,000	4Gi
500-5000 nodes	< 10ms	< 50ms	10,000	8Gi

Disk Latency is Critical

etcd requires sequential writes with fsync. Any disk with > 10ms fsync latency will cause leader elections, cluster instability, and cascading failures.

Scheduling Performance¶

Scheduler Metric	Value	Conditions
Scheduling throughput	~100 pods/sec	Default scheduler, 5000-node cluster
Scheduling latency (P99)	< 100ms	Without complex affinity rules
Scheduling with affinity	20-50 pods/sec	Pod anti-affinity across nodes
Preemption overhead	+50-100ms	When preemption kicks in

Network Performance (CNI Comparison)¶

CNI	Pod-to-Pod Latency	Throughput (TCP)	Throughput (eBPF)	Encryption Overhead
Cilium	~50us	9.5 Gbps	9.8 Gbps (native)	15-20% (WireGuard)
Calico	~60us	9.2 Gbps	9.5 Gbps (eBPF mode)	20-25% (WireGuard)
Flannel (VXLAN)	~80us	8.5 Gbps	N/A	N/A (no native)
Host networking	~30us	10 Gbps	N/A	N/A

Real-World Scale References¶

Google GKE: Supports 15,000 nodes per cluster (managed)
AWS EKS: Up to 5,000 nodes with managed control plane
OpenAI: Runs 7,500-node clusters for ML training
Alibaba Cloud: Reported testing at 10,000+ nodes

Sourcing Status¶

Unsourced Performance Data

The performance numbers in this document are estimated from vendor documentation, community benchmarks, and engineering judgment. They do not represent controlled benchmarks with documented test conditions. Specific hardware configurations, software versions, and test methodologies were not recorded.

Use these figures as rough guidance only. For production capacity planning, run your own benchmarks against your specific workload and infrastructure.

Sources¶

Security¶

Kubernetes security follows the 4C's model: Cloud, Cluster, Container, and Code. This note covers cluster and container layers. It focuses on identity, access control, pod isolation, secrets management, and supply chain security.

Authentication and Identity¶

Users and Service Accounts¶

Kubernetes distinguishes between two types of identities:

Users: Managed externally (OIDC, LDAP, SAML). Kubernetes does not store user objects. Users authenticate via the API server using certificates, tokens, or external identity providers.
Service Accounts: Kubernetes-managed identities for pods. Each namespace has a default service account. Custom service accounts are used to grant pods specific permissions.

Service Account Tokens

Since Kubernetes 1.24, service account tokens are no longer auto-mounted. Use automountServiceAccountToken: false unless a pod explicitly needs API access. Tokens are short-lived JWTs issued by the API server and projected into pods via volume mounts.

Authentication Methods¶

Method	Use Case
X.509 client certs	Admin access, kubeconfig
OIDC (for example, Okta, Entra ID)	Enterprise SSO for human users
Service account tokens	Pod-to-API-server communication
Webhook token auth	Custom token validation
Bootstrap tokens	Node bootstrapping (kubeadm)

RBAC (Role-Based Access Control)¶

RBAC is the primary authorization mechanism in Kubernetes. It uses four API objects:

Object	Scope	Purpose
`Role`	Namespace	Defines permissions within a single namespace
`ClusterRole`	Cluster-wide	Defines permissions across all namespaces or cluster-scoped resources
`RoleBinding`	Namespace	Grants a Role (or ClusterRole) to subjects within a namespace
`ClusterRoleBinding`	Cluster-wide	Grants a ClusterRole to subjects across all namespaces

Subjects can be Users, Groups, or ServiceAccounts.

# Example: Read-only access to pods in a namespace
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: production
  name: pod-reader
rules:
- apiGroups: [""]
  resources: ["pods", "pods/log"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: read-pods
  namespace: production
subjects:
- kind: User
  name: developer
roleRef:
  kind: Role
  name: pod-reader
  apiGroup: rbac.authorization.k8s.io

RBAC Best Practices

Do not bind cluster-admin broadly. Use least-privilege Roles.
Audit existing bindings regularly with kubectl get rolebindings,clusterrolebindings -o wide.
Use namespace isolation to limit blast radius.

Pod Security Standards and Admission¶

Pod Security Standards define three privilege levels applied at the namespace level:

Level	Description
Privileged	Unrestricted. Allows host access, privileged containers, all capabilities
Baseline	Minimally restrictive. Prevents host namespace sharing, privileged ports, hostPath mounts
Restricted	Heavily restricted. Requires non-root, dropped capabilities, read-only root FS, seccomp

Enforcement is done via the built-in Pod Security Admission controller (GA since v1.25):

# Apply restricted policy to a namespace
kubectl label namespace production \
  pod-security.kubernetes.io/enforce=restricted \
  pod-security.kubernetes.io/enforce-version=latest \
  pod-security.kubernetes.io/audit=restricted \
  pod-security.kubernetes.io/warn=restricted

The admission controller supports three modes per namespace: - enforce: Blocks non-compliant pods - audit: Logs audit events for non-compliant pods - warn: Triggers warnings but does not block

NetworkPolicy¶

NetworkPolicies control traffic flow at the pod level:

Default: all pod-to-pod communication is allowed within a cluster
NetworkPolicy can restrict ingress and egress traffic by label selectors, IP blocks, and ports
Implemented by the CNI plugin (Calico, Cilium, Antrea, and others)
Without a CNI plugin that supports NetworkPolicy, the rules are ignored

# Example: Deny all ingress to a namespace, allow only from specific pods
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-all-ingress
  namespace: production
spec:
  podSelector: {}
  policyTypes:
  - Ingress
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-from-frontend
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: api
  policyTypes:
  - Ingress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: frontend
    ports:
    - port: 8080
      protocol: TCP

Default Deny

A best practice is to create a default-deny-all NetworkPolicy in every namespace, then layer specific allow rules on top.

Secrets Management¶

Kubernetes Secrets¶

Kubernetes Secrets store sensitive data (passwords, tokens, keys) as base64-encoded objects:

Stored in etcd. Enable encryption at rest via EncryptionConfiguration
Default: base64 encoding only (not encryption). Configure --encryption-provider-config on the API server.
Mounted into pods as files (tmpfs) or environment variables
Namespace-scoped. Not accessible across namespaces without explicit sharing

Encryption at Rest¶

Enable encryption by configuring the API server:

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
    - secrets
    providers:
    - aescbc: # Deprecated since K8s 1.28; prefer aesgcm or secretbox
        keys:
        - name: key1
          secret: <BASE64_ENCODED_SECRET>
    - identity: {}

Deprecated Provider

The aescbc encryption provider is deprecated since Kubernetes 1.28. For new deployments, use aesgcm or secretbox providers instead. See Encrypting Secret Data at Rest for current recommendations.

External Secrets Management¶

For production, integrate with external secret stores:

External Secrets Operator: Syncs secrets from AWS Secrets Manager, Azure Key Vault, GCP Secret Manager, HashiCorp Vault
Sealed Secrets: Encrypts secrets with a public key. Decrypts in-cluster with a private key
Vault Agent Injector: Injects Vault secrets as files into pods via mutation

Admission Controllers¶

Admission controllers intercept API requests after authentication and authorization but before object persistence:

Controller	Purpose
PodSecurity	Enforces Pod Security Standards
ResourceQuota	Limits resource consumption per namespace
LimitRanger	Sets default resource requests/limits
NamespaceLifecycle	Prevents creation in terminating namespaces
ServiceAccount	Automates service account token mounting

Policy Engines (OPA Gatekeeper / Kyverno)¶

External admission webhooks enforce custom policies:

OPA Gatekeeper: Uses Rego policies. Audit mode for dry-run, enforcement via webhook
Kyverno: Kubernetes-native policies (YAML, no new language). Mutation and validation rules
Both run as validating/mutating admission webhooks
Policies can enforce: image registry restrictions, label requirements, resource limits, security contexts

# Kyverno example: Require non-root containers
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-non-root
spec:
  validationFailureAction: Enforce
  rules:
  - name: check-runAsNonRoot
    match:
      any:
      - resources:
          kinds:
          - Pod
    validate:
      message: "Containers must run as non-root"
      pattern:
        spec:
          containers:
          - securityContext:
              runAsNonRoot: true

Image Policy and Supply Chain¶

Image Policy Webhook¶

The ImagePolicyWebhook admission controller can validate images against external policy:

Restrict registries (for example, only allow images from registry.example.com)
Require signed images
Enforce tag restrictions (for example, disallow :latest)

Sigstore / Cosign¶

Sign container images with Cosign (Sigstore)
Verify signatures at admission time using policy controllers
Kyverno and OPA Gatekeeper both support Cosign verification

Audit Logging¶

Kubernetes audit logging records API server requests for forensic analysis:

Configured via --audit-log-path and AuditPolicy on the API server
Four logging stages: RequestReceived, ResponseStarted, ResponseComplete, Panic
Log levels: None, Metadata, Request, RequestResponse
Essential for compliance (SOC 2, PCI-DSS) and incident investigation

# Audit policy example
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
- level: RequestResponse
  resources:
  - group: ""
    resources: ["secrets"]
- level: Metadata
  resources:
  - group: ""
    resources: ["pods", "services"]
- level: None
  users: ["system:kube-proxy"]

CIS Benchmarks¶

The Center for Internet Security publishes Kubernetes benchmarks covering:

Control plane node configuration (API server, scheduler, controller-manager, etcd)
Worker node configuration (kubelet, proxy)
Network policies and CNI security
RBAC and service account controls
Secret encryption and audit logging

Automated scanning tools: - kube-bench: Open-source CIS benchmark scanner (Aqua Security) - Trivy operator: Scans running workloads for misconfigurations and CVEs

Security Checklist¶

Area	Recommendation
Authentication	Use OIDC for human users. Disable client cert auth if not needed
RBAC	Apply least-privilege. Audit bindings regularly
Pod Security	Enforce `baseline` or `restricted` via Pod Security Admission
Network	Apply default-deny NetworkPolicy. Restrict egress
Secrets	Enable encryption at rest. Use external secret stores for production
Admission	Deploy Kyverno or OPA Gatekeeper for custom policies
Images	Sign and verify images. Restrict allowed registries
Audit	Enable audit logging with appropriate policy levels
Kubelet	Disable anonymous auth. Set `--read-only-port=0`
etcd	Enable TLS client authentication. Restrict network access