Skip to content

Architecture

Related Notes

index | architecture | operations | security

Overview

Flannel is a simple overlay network fabric designed for Kubernetes clusters. Developed by CoreOS (now Red Hat), it provides a flat network space where every pod gets a unique IP address reachable from any node. Flannel focuses on one task -- inter-node pod connectivity -- and deliberately does not include network policy enforcement, advanced routing, or observability features.

Flannel is the default CNI plugin in K3s and many lightweight Kubernetes distributions. It is best suited for small-to-medium clusters where simplicity matters more than advanced networking features.

See also: security for limitations and when to use Calico or Cilium instead.

Component Diagram

graph TB
    subgraph "Control Plane"
        BE["Subnet Manager<br/>(etcd or Kubernetes API)"]
    end

    subgraph "Node 1"
        FLAN1["flanneld daemon<br/>(kube-flannel DaemonSet)"]
        VTEP1["flannel.1<br/>(VXLAN VTEP interface)"]
        BRIDGE1["cni0<br/>(bridge interface)"]
        VETH1_1["veth1 (pod)"]
        VETH1_2["veth2 (pod)"]
    end

    subgraph "Node 2"
        FLAN2["flanneld daemon"]
        VTEP2["flannel.1"]
        BRIDGE2["cni0"]
        VETH2_1["veth1 (pod)"]
        VETH2_2["veth2 (pod)"]
    end

    BE -->|"subnet leases<br/>& network config"| FLAN1
    BE -->|"subnet leases<br/>& network config"| FLAN2
    FLAN1 -->|"manages"| VTEP1
    FLAN1 -->|"manages"| BRIDGE1
    BRIDGE1 --> VETH1_1
    BRIDGE1 --> VETH1_2
    FLAN2 -->|"manages"| VTEP2
    FLAN2 -->|"manages"| BRIDGE2
    BRIDGE2 --> VETH2_1
    BRIDGE2 --> VETH2_2
    VTEP1 <-->|"VXLAN encapsulation<br/>UDP port 8472"| VTEP2

Core Components

flanneld Daemon

flanneld is the sole per-node daemon, deployed as a Kubernetes DaemonSet (typically kube-flannel-ds). It handles all networking responsibilities on each node:

  • Subnet leasing -- claims a subnet lease from the subnet manager (etcd or Kubernetes API). Each node receives a /24 subnet by default from the configured pod CIDR (e.g., 10.244.0.0/16).
  • Route programming -- writes kernel routes so that traffic destined for remote pod subnets is forwarded through the appropriate backend (VXLAN tunnel, host gateway, or WireGuard tunnel).
  • Backend management -- creates and manages the tunnel interface (flannel.1 for VXLAN, flannel-wg for WireGuard) and updates forwarding rules when nodes join or leave.
  • Subnet file -- writes /run/flannel/subnet.env containing the allocated subnet, MTU, and IP masquerade settings. This file is consumed by the CNI bridge plugin.

Subnet Manager

The subnet manager is the coordination backend that stores the global network configuration and per-node subnet leases:

Backend Description When to use
etcd Stores network config at /coreos.com/network/ and per-node subnet leases. Requires a separate etcd cluster. Standalone etcd deployments, non-Kubernetes environments
Kubernetes API (--kube-subnet-mgr) Uses Kubernetes Node objects and a ConfigMap (kube-flannel-cfg) for network configuration. No external etcd needed. Standard Kubernetes clusters (default for K3s)

When using the Kubernetes backend, Flannel reads the cluster CIDR from the net-conf.json key in the kube-flannel-cfg ConfigMap:

{
  "Network": "10.244.0.0/16",
  "Backend": {
    "Type": "vxlan"
  }
}

CNI Plugin (bridge)

Flannel uses the standard CNI bridge plugin (not its own custom plugin). The CNI configuration is generated from the subnet file written by flanneld:

  1. flanneld writes /run/flannel/subnet.env with the node's subnet and MTU.
  2. The CNI config at /etc/cni/net.d/10-flannel.conflist references the flannel CNI plugin.
  3. The flannel CNI plugin reads subnet.env, creates a veth pair, and attaches one end to the cni0 bridge.
  4. The pod's interface gets an IP from the node's allocated subnet.

Overlay Network Diagram

sequenceDiagram
    participant PodA as Pod A (10.244.1.5)
    participant CNI1 as cni0 bridge (Node 1)
    participant VTEP1 as flannel.1 VTEP (Node 1)<br/>10.244.1.0/24
    participant NET as Physical Network<br/>(Underlay)
    participant VTEP2 as flannel.1 VTEP (Node 2)<br/>10.244.2.0/24
    participant CNI2 as cni0 bridge (Node 2)
    participant PodB as Pod B (10.244.2.8)

    Note over PodA,PodB: Pod A sends packet to Pod B on a different node
    PodA->>CNI1: Packet: src=10.244.1.5 dst=10.244.2.8
    CNI1->>VTEP1: Route lookup: 10.244.2.0/24 via flannel.1
    Note over VTEP1: VXLAN encapsulation<br/>Outer: Node1_IP -> Node2_IP<br/>Inner: 10.244.1.5 -> 10.244.2.8<br/>VNI: 1, UDP port: 8472
    VTEP1->>NET: Encapsulated VXLAN packet
    NET->>VTEP2: Packet arrives at Node 2
    Note over VTEP2: VXLAN decapsulation<br/>Extract inner packet
    VTEP2->>CNI2: Inner packet: 10.244.1.5 -> 10.244.2.8
    CNI2->>PodB: Deliver to pod

Backend Types

VXLAN encapsulates pod packets inside UDP packets. It is the recommended backend because it works on any network fabric without special support.

  • VNI -- defaults to 1 on Linux, must be >= 4096 on Windows.
  • UDP port -- defaults to 8472 on Linux (kernel default), must be 4789 on Windows.
  • DirectRouting -- when enabled, uses host-gw-style direct routes for nodes on the same L2 subnet and VXLAN only for cross-subnet traffic. Default: disabled.
  • GBP -- Group Based Policy for VXLAN. Default: disabled.
  • MTU -- defaults to the MTU of the external interface minus 50 bytes (VXLAN overhead).

host-gw

Creates static IP routes to remote pod subnets via the remote node's IP address. No encapsulation overhead.

  • Requirements -- all nodes must be on the same L2 subnet. No encapsulation means no MTU reduction.
  • Performance -- lower latency and higher throughput than VXLAN because there is no encapsulation/decapsulation.
  • Limitations -- cannot traverse routers or cross subnets. Only works on flat L2 networks.

WireGuard

Uses in-kernel WireGuard to encapsulate and encrypt all inter-node pod traffic.

  • Encryption -- all pod-to-pod traffic between nodes is encrypted automatically.
  • ListenPort -- defaults to 51820 (IPv4) and 51821 (IPv6).
  • PSK -- optional pre-shared key for post-quantum resistance.
  • Mode -- separate (default, independent IPv4/IPv6 tunnels), auto (single tunnel, auto-detect address family), ipv4 or ipv6 (single tunnel, forced address family).
  • PersistentKeepaliveInterval -- keepalive interval in seconds for NAT traversal. Default: disabled (0).
  • Requirements -- requires the WireGuard kernel module (wireguard.ko) on each node.

IPSec (legacy)

Uses IPsec (strongSwan) for encrypted tunnels. This backend is less commonly used and has largely been superseded by the WireGuard backend.

Subnet Allocation

graph TD
    CIDR["Cluster CIDR<br/>e.g. 10.244.0.0/16"]
    N1["Node 1 subnet<br/>10.244.1.0/24"]
    N2["Node 2 subnet<br/>10.244.2.0/24"]
    N3["Node 3 subnet<br/>10.244.3.0/24"]
    N4["Node N subnet<br/>10.244.N.0/24"]

    CIDR --> N1
    CIDR --> N2
    CIDR --> N3
    CIDR --> N4

    N1 -->|"pod IPs<br/>10.244.1.1 - 10.244.1.254"| P1["Pods on Node 1"]
    N2 -->|"pod IPs<br/>10.244.2.1 - 10.244.2.254"| P2["Pods on Node 2"]
  • Subnet size -- defaults to /24 (254 addresses per node). Configurable via the SubnetLen field in the network configuration (e.g., "SubnetLen": 26 for /26 blocks with 62 addresses). Must be smaller than the cluster CIDR prefix.
  • Lease renewal -- nodes renew their subnet lease periodically (default every 24 hours for Kubernetes backend, configurable via --subnet-lease-renew-margin).
  • Subnet file -- each node writes its allocated subnet to /run/flannel/subnet.env for the CNI plugin to consume.

Data Flow Summary

Scenario Path
Pod-to-Pod on same node veth -> cni0 bridge -> veth (no encapsulation)
Pod-to-Pod on different node (VXLAN) veth -> cni0 -> route -> flannel.1 (VXLAN encap) -> physical NIC -> flannel.1 (decap) -> cni0 -> veth
Pod-to-Pod on different node (host-gw) veth -> cni0 -> route -> physical NIC -> route -> cni0 -> veth
Pod-to-external veth -> cni0 -> iptables MASQUERADE (if --ip-masq=true) -> physical NIC

What Flannel Does NOT Provide

Feature gaps

Flannel is intentionally minimal. It does not include:

  • Network policies -- no Kubernetes NetworkPolicy or custom policy enforcement. You must add Calico or a policy controller alongside Flannel.
  • eBPF dataplane -- uses kernel VXLAN/host routes, no eBPF acceleration.
  • Observability -- no built-in flow logging, service map, or Hubble-equivalent.
  • Load balancing -- does not replace kube-proxy. Relies on standard kube-proxy for service routing.
  • Multi-cluster -- no built-in ClusterMesh or cross-cluster networking.
  • IPAM flexibility -- subnet size is configurable but block affinity, block borrowing, and per-pod subnet allocation (like Calico IPAM) are not supported.

Sources

How It Works

VXLAN overlay mechanics, subnet lease lifecycle, route programming, and backend internals.

VXLAN Overlay

sequenceDiagram
    participant PodA as Pod A (Node 1, 10.244.0.2)
    participant Flannel1 as flanneld (Node 1)
    participant VTEP1 as VXLAN VTEP (flannel.1)
    participant Network as Physical Network (UDP 8472)
    participant VTEP2 as VXLAN VTEP (flannel.1)
    participant PodB as Pod B (Node 2, 10.244.1.3)

    PodA->>VTEP1: Packet to 10.244.1.3
    VTEP1->>VTEP1: Encapsulate in VXLAN header
    VTEP1->>Network: UDP 8472 → Node 2
    Network->>VTEP2: Receive VXLAN packet
    VTEP2->>VTEP2: Decapsulate
    VTEP2->>PodB: Deliver original packet

VXLAN Encapsulation Detail

When Pod A sends a packet to Pod B on a different node:

  1. The packet leaves the pod's veth, enters the cni0 bridge on Node 1
  2. The kernel performs a route lookup: 10.244.1.0/24 via 10.244.1.0 dev flannel.1
  3. The flannel.1 VTEP encapsulates the original Ethernet frame inside a VXLAN header:
  4. Outer Ethernet: Node 1 MAC → Node 2 MAC (resolved via kernel FDB)
  5. Outer IP: Node 1 IP → Node 2 IP
  6. Outer UDP: Source port (hash of inner flow) → Destination port 8472
  7. VXLAN header: VNI 1 (Virtual Network Identifier)
  8. Inner frame: Original pod-to-pod Ethernet frame (10.244.0.2 → 10.244.1.3)
  9. The encapsulated packet traverses the physical network
  10. Node 2's kernel receives the UDP packet on port 8472, the VXLAN driver decapsulates it, and delivers the inner frame to the cni0 bridge

The VTEP forwarding database (FDB) is populated by flanneld when nodes join or leave. Each flanneld writes the mapping of remote subnet → remote VTEP MAC → remote node IP into the kernel FDB via bridge fdb add commands.

Subnet Lease Model

Each node claims a subnet lease from the cluster CIDR. The default subnet size is /24 (254 pod IPs per node), configurable via SubnetLen in the network configuration JSON.

Lease Lifecycle

stateDiagram-v2
    [*] --> Claiming: Node starts
    Claiming --> Active: Subnet assigned
    Active --> Active: Renew (every 24h default)
    Active --> Expired: Renewal fails (node down)
    Expired --> Claiming: Node restarts
    Expired --> Reclaimed: Another node claims subnet
    Reclaimed --> [*]: Subnet reassigned

Subnet Manager Backends

Backend Lease Storage Renewal Mechanism
etcd /coreos.com/network/subnets/ prefix etcd lease with TTL (default 24h)
Kubernetes API Node annotations + ConfigMap Periodic renewal via --subnet-lease-renew-margin

When using the Kubernetes backend (--kube-subnet-mgr), flanneld watches the kube-flannel-cfg ConfigMap for network configuration changes and writes its subnet assignment as an annotation on the Node object.

Route Programming

Flanneld programs kernel routes that direct pod traffic to the correct backend:

Route Purpose
10.244.0.0/24 dev cni0 Local pod subnet via bridge
10.244.1.0/24 via 10.244.1.0 dev flannel.1 onlink Remote subnet via VXLAN tunnel
10.244.2.0/24 via 10.244.2.0 dev flannel.1 onlink Another remote subnet

The onlink flag is critical -- it tells the kernel to install the route even though the gateway (e.g., 10.244.1.0) is not directly reachable via the flannel.1 interface. The VTEP resolves the gateway to the remote node's physical IP via the FDB.

host-gw Backend Mechanics

When using the host-gw backend instead of VXLAN:

  1. Flanneld writes static routes pointing remote pod subnets to the remote node's physical IP
  2. No encapsulation occurs -- the kernel forwards original pod packets directly via the physical network
  3. The physical network must be flat L2 (all nodes on the same subnet)
  4. MTU is preserved (no VXLAN overhead), resulting in higher throughput

Subnet File

Flanneld writes /run/flannel/subnet.env on startup and after each lease renewal:

FLANNEL_SUBNET=10.244.0.1/24
FLANNEL_MTU=1450
FLANNEL_IPMASQ=true

The CNI bridge plugin reads this file to configure the cni0 bridge interface with the correct subnet, MTU, and masquerade rules. The MTU is calculated as the physical interface MTU minus the backend overhead (50 bytes for VXLAN).

Sources

Benchmarks

Scope

Performance characteristics, scaling limits, and resource consumption for Flannel.

Backend Performance

Backend Throughput (10GbE) Latency Overhead Encryption
VXLAN 8-9 Gbps +80-100us No
host-gw 9.5+ Gbps +10-20us No
WireGuard 7-8 Gbps +100-200us Yes
UDP (deprecated) 5-6 Gbps +200-500us No

Resource Usage

Metric Per Node Notes
CPU (idle) < 10m Minimal overhead
CPU (active) 50-200m Under heavy traffic
Memory 30-50Mi Lightweight

Scaling Limits

Dimension Limit Notes
Nodes per cluster 5,000+ Same as K8s limit
Pods per node 110 K8s default, not Flannel limit
Subnet size /24 per node (default) Configurable via SubnetLen

Sourcing Status

Unsourced Performance Data

The performance numbers in this document are estimated from vendor documentation, community benchmarks, and engineering judgment. They do not represent controlled benchmarks with documented test conditions. Specific hardware configurations, software versions, and test methodologies were not recorded.

Use these figures as rough guidance only. For production capacity planning, run your own benchmarks against your specific workload and infrastructure.

Sources