Networking

Container Network Interface (CNI) plugins for Kubernetes — providing pod networking, network policy enforcement, encryption, and observability across cluster nodes.

← Knowledge Base

Topics

Plugin	Description
Calico	Pluggable CNI with BGP-based routing, eBPF/iptables/nftables data planes, and rich network policy for Kubernetes, VMs, and bare metal.
Cilium	eBPF-powered CNI with identity-based security policies, Hubble flow observability, Tetragon runtime enforcement, and kube-proxy replacement.
Flannel	Simple, lightweight VXLAN overlay network — the "it just works" CNI for vanilla Kubernetes with minimal operational complexity.

Comparisons

Comparison	Scope
CNI Comparison	Calico vs Cilium vs Flannel — performance, security model, eBPF support, and operational overhead

Landscape

The Kubernetes networking space is being reshaped by eBPF, which enables CNI plugins to implement packet forwarding, load balancing, network policy, and observability directly in the Linux kernel without iptables chains or userspace proxies. Cilium has led this shift and become the default CNI for major managed Kubernetes offerings (GKE Dataplane V2, AWS EKS Anywhere, Azure AKS with BYOCNI), while Calico has responded by adding its own eBPF data plane alongside its traditional iptables and nftables backends.

The sidecar-less networking trend from the service mesh world is bleeding into CNI territory — Cilium's Service Mesh mode and Calico's application-layer policy both push L7 functionality into the CNI itself, reducing the need for sidecar proxies. Network policy enforcement has matured from basic L3/L4 Kubernetes NetworkPolicy to identity-aware L7 policies (CiliumNetworkPolicy, Calico GlobalNetworkPolicy) that can filter by HTTP path, gRPC method, or DNS FQDN.

IPAM Diversity

IPAM strategies vary significantly across CNI plugins: host-scope IPAM for overlay networks (Flannel), cluster-scope IPAM with block borrowing for routed networks (Calico BGP), and cloud-provider IPAM that allocates ENIs or VPC IPs directly to pods (AWS VPC CNI, Azure CNI). The choice of IPAM strategy has downstream implications for VPC address space consumption, pod-to-external-service routing, and multi-cluster networking.

Encryption in transit has also moved into the CNI layer — Cilium supports transparent WireGuard and IPsec encryption between nodes, while Calico offers WireGuard encryption for both its eBPF and iptables data planes, providing mTLS-equivalent protection without a service mesh.

Key Concepts

CNI Plugin Interface

The Container Network Interface specification defines a JSON-based contract between the container runtime and network plugins. When a pod is scheduled, the kubelet invokes the CNI binary with ADD to attach a network interface and assign an IP, and DEL to clean up when the pod terminates.

CNI plugins can be chained — a primary plugin handles IP assignment and routing, while secondary plugins add bandwidth limiting, port mapping, or IPAM delegation. The spec is deliberately minimal, which is why CNI plugins vary enormously in capability — from Flannel's simple VXLAN overlay to Cilium's full eBPF networking stack with service mesh integration.

eBPF vs iptables

Performance Implications

iptables processes packets through a linear chain of rules, causing O(n) performance degradation as the number of services and network policies grows — problematic at 5,000+ services. eBPF attaches programs directly to kernel hooks (XDP, TC, socket), enabling O(1) lookups via hash maps. Cilium's eBPF kube-proxy replacement eliminates conntrack overhead and reduces per-packet latency. Calico's eBPF data plane similarly bypasses iptables, though it retains iptables as a fallback for environments with older kernels (pre-5.4).

The kernel version requirement is a practical constraint: eBPF features used by Cilium require Linux 5.4+ (basic operation) with 5.10+ recommended for full feature support. nftables (used by Calico as an alternative to iptables) offers better performance than legacy iptables through atomic rule replacement and native set/map data structures, but still cannot match eBPF's programmability.

Network Policy

Kubernetes-native NetworkPolicy resources define L3/L4 ingress and egress rules scoped to pods via label selectors. The API is intentionally limited — it lacks deny-all defaults, cluster-wide scope, and L7 filtering.

Extended policy capabilities by CNI:

• Calico: GlobalNetworkPolicy (cluster-wide, ordered priority, deny rules), NetworkSet (external CIDR grouping), DNS policy, and application-layer policy via Envoy sidecar
• Cilium: CiliumNetworkPolicy with L7 HTTP/gRPC/Kafka filtering, DNS-based FQDN rules, and identity-aware enforcement that survives pod IP changes
• Flannel: No native network policy support — requires pairing with Calico's policy-only mode (Canal) for NetworkPolicy enforcement

IPAM (IP Address Management)

The strategy for allocating IP addresses to pods across a cluster. Host-local IPAM assigns a /24 per node from a cluster CIDR (simple but wastes addresses). Calico's IPAM uses IPAM blocks with borrowing between nodes for efficient utilization.

Cloud-provider IPAM (AWS VPC CNI) assigns real VPC IPs to pods, enabling direct pod-to-VPC-resource communication without NAT but consuming VPC address space. Dual-stack (IPv4+IPv6) IPAM is now GA in Kubernetes 1.23+ and supported by all major CNIs.

Overlay vs Routed

Overlay networks (VXLAN, Geneve) encapsulate pod traffic in outer UDP headers, enabling pod networking across any underlying L3 infrastructure without node network cooperation. Routed networks (BGP peering, direct routing) advertise pod CIDRs to the physical network, avoiding encapsulation overhead but requiring L3 fabric cooperation.

Calico supports both modes — VXLAN for cloud environments without BGP support, and native BGP peering for on-premises networks where route reflectors can distribute pod routes. Flannel defaults to VXLAN overlay, prioritizing simplicity over performance. Cilium supports VXLAN, Geneve, and native routing modes, with its eBPF data plane optimizing encapsulation overhead when overlays are necessary.

Open Questions

• As eBPF becomes the dominant data plane, will there be a practical reason to deploy Flannel (pure overlay, no eBPF) outside of resource-constrained edge environments like k3s on Raspberry Pi clusters?
• How will the Kubernetes Gateway API's maturation affect CNI plugins that have been adding L7 functionality — will CNIs defer to Gateway API implementations or continue to build competing L7 capabilities?
• With Cilium now providing service mesh, Hubble observability, and Tetragon runtime security alongside CNI, does the "do one thing well" Unix philosophy apply, or is an integrated networking platform the right architectural choice?