Architecture

Structure of the bpf-developer-tutorial project: curriculum layout, the CO-RE toolchain it teaches, the eunomia-bpf compile/load pipeline used by early lessons, and the CI-driven compatibility verification system.

Curriculum Layout

The repository holds one directory per tool under src/ — 60 directories total as of August 2026. Each is self-contained: kernel-side C (*.bpf.c), user-space loader in C/Go/Rust depending on the lesson, its own README (English + Chinese variants at repo root), build files, and a .config metadata file that drives the compatibility matrix.

Tier	Lessons	Framework focus
Getting started	0-10	eunomia-bpf ecc/ecli; kprobe, fentry, uprobe, tracepoints; hash maps, perf event arrays, ring buffers, histograms
Advanced projects	11-21	libbpf user-space programs; USDT, memleak tracing, LSM security hooks, tc and XDP basics
In-depth: networking	23, 29, 41-42, 46, 50, 53	L7 socket filters, sockops, XDP tcpdump/load balancer/packet generator, TCX links, BPF Qdisc egress pacer
In-depth: tracing	30-33, 37, 39-40, 48, 52	sslsniff via uprobe, Go runtime tracing, wall-clock profiler, funclatency, Rust/nginx/MySQL tracing, energy monitoring, fsession
In-depth: security	19, 24-28, 34, 51, 54	BPF LSM detection/defense, process/file hiding, privilege escalation demos, syscall argument rewrite, TCP quarantine, exec image inspection
Kernel features	35-36, 38, 43 + features/*	user ring buffer, userspace runtimes (bpftime), BTF-uprobe CO-RE extension, custom kfuncs, arena, iterators, token, workqueues, dynptr, struct_ops
Schedulers (sched_ext)	44-45	minimal BPF scheduler; scx_nest implementation (kernel 6.12+)
GPU/XPU	47 + xpu/*	CUDA event tracing via uprobes, CUPTI flamegraph profiler, GPU/NPU kernel driver monitoring
Platform	22, 49, cgroup	Android deployment, HID-BPF device fixes, cgroup policy control

Two non-lesson assets round out the set: a bpftrace tutorial port (src/bpftrace-tutorial) for one-liner-style learning, and lesson 18's curated research-paper index.

The CO-RE Toolchain Taught by the Tutorial

All lessons follow Compile Once, Run Everywhere principles: C code compiled with clang against a BTF-generated vmlinux.h, producing relocatable BPF bytecode that survives kernel-type changes across kernel versions without recompilation.

flowchart TB
    subgraph Author["Authoring (lesson src/)"]
        SRC["minimal.bpf.c<br/>SEC(\"tp/syscalls/sys_enter_write\")"]
        VMLINUX["vmlinux.h<br/>(BTF-derived types, CO-RE relocations)"]
    end

    subgraph Toolchain["eunomia-bpf compiler"]
        ECC["ecc<br/>(clang wrapper)"]
        PKG["package.json / Wasm module<br/>+ generated CLI args & export headers"]
    end

    subgraph Distribution["Distribution"]
        OCI["OCI registry<br/>ghcr.io/eunomia-bpf/execve"]
        GH_PAGES["GitHub Pages JSON packages"]
    end

    subgraph Loader["Load path"]
        ECLI["ecli CLI"]
        BPFLOADER["bpf-loader-rs"]
        LIBBPF["libbpf syscalls<br/>bpf()/perf_event_open()"]
    end

    subgraph Kernel["Linux kernel"]
        VERIFIER["BPF verifier"]
        JIT["JIT compiler"]
        HOOKS["Hook sites:<br/>tracepoints · kprobe/fentry · uprobe<br/>XDP/tc · sched_ext · BPF LSM"]
        MAPS["Maps:<br/>hash · ringbuf · perf events<br/>histograms"]
    end

    OUTPUT["User space output:<br/>events, histograms, CLI"]

    SRC --> ECC
    VMLINUX --> ECC
    ECC --> PKG
    PKG --> OCI
    PKG --> GH_PAGES
    GH_PAGES --> ECLI
    OCI --> ECLI
    ECLI --> BPFLOADER
    BPFLOADER --> LIBBPF
    LIBBPF --> VERIFIER
    VERIFIER --> JIT
    JIT --> HOOKS
    HOOKS --> MAPS
    MAPS --> OUTPUT

Key design points visible across lessons:

• Kernel/user separation: every program splits into kernel logic plus a user-space loader (*_user.c, or Go/Rust equivalents) handling attach, event polling, and display.
• Progressive data plumbing: lesson order deliberately walks perf event array (lesson 7) before ring buffer (lesson 8) before histograms (lesson 9), then user ring buffers for kernel-bound async (lesson 35).
• Minimal-example discipline: lesson 1's complete tracepoint program fits in ~25 lines including license declaration, demonstrating bpf_get_current_pid_tgid() filtering and bpf_printk output to /sys/kernel/debug/tracing/trace_pipe.

Later tiers swap the harness while keeping this shape: libbpf-bootstrap's Makefile workflow from lesson 11, cilium/ebpf for Go from the starter templates, libbpf-rs from lesson 12 onward, and bpftime/wasm-bpf as alternative load-and-execute runtimes profiled in lesson 36.

Runtime Event Flow

The full lifecycle taught by the early course arc, from artifact to observed data:

sequenceDiagram
    participant U as User terminal (ecli)
    participant L as bpf-loader-rs / libbpf
    participant K as Kernel (verifier -> JIT)
    participant T as Tracepoint tp/syscalls/sys_enter_write
    participant M as Maps / trace_pipe
    U->>L: sudo ./ecli run package.json
    L->>K: bpf() syscall: load program + maps
    K->>K: verifier proof pass, JIT to native
    L->>T: attach via perf_event_open / link_create
    Note over T,M: every matched kernel event fires the BPF function
    T->>M: bpf_printk -> trace_pipe<br/>(later lessons: bpf_perf_event_output,<br/>bpf_ringbuf_output)
    M->>U: events streamed until Ctrl+C detaches

Hook Taxonomy Across the Curriculum

Hook class	Attach point	Representative lessons
Static tracepoints	SEC("tp/syscalls/*"), subsystem events	1, 7-11
kprobe / fentry	arbitrary kernel functions	2 vs. 3 (same tool rebuilt with the modern mechanism)
uprobe / USDT	user-space functions (bash readline, SSL, JVM GC)	5, 15, 30, 37
Networking	XDP ingress, tc classifier, sockops, TCX, Qdisc	20-21, 29, 41-42, 50, 53
Security hooks	BPF LSM (security_*), cgroup device control	19, 54, cgroup
Scheduler ops	sched_ext struct_ops dispatch loop	44-45
GPU/user-space	CUDA library uprobes, CUPTI subscriber, bpftime VM	47, xpu/flamegraph, 36

The deliberate rebuild of the same unlink monitor with kprobe (lesson 2) then fentry (lesson 3) is the curriculum's clearest teaching device for migration pressure off legacy probe types.

Data Plumbing Evolution

The event-export mechanisms are introduced in dependency order, each replacing the previous one's limits:

1. bpf_printk (lesson 1) — zero-setup debugging; shared global pipe, three format args max.
2. perf event array (lesson 7, execsnoop) — structured events pushed on firing; no drop-safe ordering guarantee across CPUs.
3. ring buffer (lesson 8, exitsnoop) — single shared mmap'd buffer, kernel 5.8+, better memory efficiency.
4. histogram maps (lesson 9, runqlat) — in-kernel log2 bucketing via bpf_log2l() on recorded latencies; only aggregate buckets cross the kernel/user boundary, not raw events.
5. user ring buffer (lesson 35) — reverses direction: async user-to-kernel commands without a syscall wake-up per message.

Sched_ext Track

Lessons 44-45 teach struct_ops-based CPU scheduling on kernel 6.12+, where a BPF program implements a scheduling class alongside CFS. The org cites over one million machines already running sched_ext policies in production (Meta's scx_layered among them) as adoption evidence. Lesson 45 reproduces the upstream scx_nest policy: idle-CPU nesting heuristics that trade slight latency for reduced wake-up spread on multi-core hosts.

Framework Comparison Frame

Lesson 1 codifies the decision matrix the curriculum teaches implicitly:

Framework	Model	Trade-off emphasized
BCC	Python front-end, runtime compilation per target	Rich helpers, but heavy dependencies and repeated compiles
libbpf / CO-RE	Ahead-of-time compiled object, skeleton headers	One compile runs everywhere; steeper C conventions
cilium/ebpf (Go)	Go-native management of BPF objects	Idiomatic Go apps, still needs C kernel side
libbpf-rs	Rust ergonomic wrapper over libbpf	Safety in user space; same CO-RE kernel objects
eunomia-bpf	Kernel-code-only authorship, packaged artifacts	Fastest path from .bpf.c to distributable tool

Compatibility Verification System

The compatibility matrix is machine-generated from each lesson's .config metadata rather than hand-maintained prose. For every lesson it records minimum kernel version (with provenance basis: docs statement vs. required-feature introduction vs. repository baseline), architectures, BTF requirement, exact CONFIG_* kernel options, hardware prerequisites, root requirements, and test status.

Status distribution across the ~53 tracked rows (August 2026):

• CI runtime (~21): built and executed on runners — includes early tracing lessons, memleak, tc.
• CI build (~24): compiled but not exercised — most sched_ext/XDP/HID/feature lessons need real devices or newer kernels.
• Not in CI (~5): LSM-connect, XDP, Android, nginx, MySQL — hardware or environment constraints.
• Docs only (3): introductions and reading lists.

This is why a kernel-version spread of 4.8 → 7.0 (fsession latency, lesson 52) coexists with confident instructions: each claim traces back to declared metadata plus automated evidence where feasible.

Architectural nuance captured by the matrix

Lesson 3 shows genuinely split baselines (fentry supported on x86_64 since 5.5, arm64 needing 6.0) — exactly the kind of divergence a hand-written tutorial would flatten incorrectly.

Platform Portability Notes

Lesson 22 documents Android as a supported target (kernel 5.15 baseline, x86_64 emulator validated in the matrix) using the kernel's built-in eBPF userspace via libbpf from an app/native process — noting every shipped Android phone already runs eBPF internally for network/power management. Lesson 49 covers HID-BPF (kernel 6.3+): input-device fixups loaded through hid_bpf without patching drivers, the clearest example in the course of eBPF as a general driver-extension mechanism rather than pure observability.

Research Grounding

Lesson 18 ties the practical track to systems research the organization considers foundational: XRP in-kernel storage functions (OSDI '22 Best Paper), Jitterbug formally verified BPF JITs (OSDI '20), Electrode eBPF-accelerated Paxos (+128% throughput, NSDI '23), BMC in-kernel Memcached caching (up to 18x throughput, NSDI '21), hXDP FPGA offload (OSDI '20), and lambda-IO computational storage (FAST '23). Maintainer affiliation with an OSDI 2025 publication underwrites the depth of the sched_ext/GPU material.

Benchmarks

The tutorial itself publishes no performance numbers — appropriate for instructional material. Performance-relevant claims made inside lessons attribute to their sources: bpftime's ~10x-uprobe-speedup figure comes from the org's own benchmarking, and paper-linked speedups listed above carry venue citations. Treat both as upstream claims, not independently reproduced here.