Architecture¶

Structure of the bpf-developer-tutorial project: curriculum layout, the CO-RE toolchain it teaches, the eunomia-bpf compile/load pipeline used by early lessons, and the CI-driven compatibility verification system.

Curriculum Layout¶

The repository holds one directory per tool under src/ — 60 directories total as of August 2026. Each is self-contained: kernel-side C (*.bpf.c), user-space loader in C/Go/Rust depending on the lesson, its own README (English + Chinese variants at repo root), build files, and a .config metadata file that drives the compatibility matrix.

Tier	Lessons	Framework focus
Getting started	0-10	eunomia-bpf `ecc`/`ecli`. Kprobe, fentry, uprobe, tracepoints. Hash maps, perf event arrays, ring buffers, histograms
Advanced projects	11-21	libbpf user-space programs. USDT, memleak tracing, LSM security hooks, tc and XDP basics
In-depth: networking	23, 29, 41-42, 46, 50, 53	L7 socket filters, sockops, XDP tcpdump/load balancer/packet generator, TCX links, BPF Qdisc egress pacer
In-depth: tracing	30-33, 37, 39-40, 48, 52	sslsniff via uprobe, Go runtime tracing, wall-clock profiler, funclatency, Rust/nginx/MySQL tracing, energy monitoring, fsession
In-depth: security	19, 24-28, 34, 51, 54	BPF LSM detection/defense, process/file hiding, privilege escalation demos, syscall argument rewrite, TCP quarantine, exec image inspection
Kernel features	35-36, 38, 43 + features/*	user ring buffer, userspace runtimes (bpftime), BTF-uprobe CO-RE extension, custom kfuncs, arena, iterators, token, workqueues, dynptr, struct_ops
Schedulers (sched_ext)	44-45	minimal BPF scheduler. scx_nest implementation (kernel 6.12+)
GPU/XPU	47 + xpu/*	CUDA event tracing via uprobes, CUPTI flamegraph profiler, GPU/NPU kernel driver monitoring
Platform	22, 49, cgroup	Android deployment, HID-BPF device fixes, cgroup policy control

Two non-lesson assets round out the set: a bpftrace tutorial port (src/bpftrace-tutorial) for one-liner-style learning, and lesson 18's curated research-paper index.

The CO-RE Toolchain Taught by the Tutorial¶

All lessons follow Compile Once, Run Everywhere principles: C code compiled with clang against a BTF-generated vmlinux.h, producing relocatable BPF bytecode that survives kernel-type changes across kernel versions without recompilation.

flowchart TB
    subgraph Author["Authoring (lesson src/)"]
        SRC["minimal.bpf.c<br/>SEC(\"tp/syscalls/sys_enter_write\")"]
        VMLINUX["vmlinux.h<br/>(BTF-derived types, CO-RE relocations)"]
    end

    subgraph Toolchain["eunomia-bpf compiler"]
        ECC["ecc<br/>(clang wrapper)"]
        PKG["package.json / Wasm module<br/>+ generated CLI args & export headers"]
    end

    subgraph Distribution["Distribution"]
        OCI["OCI registry<br/>ghcr.io/eunomia-bpf/execve"]
        GH_PAGES["GitHub Pages JSON packages"]
    end

    subgraph Loader["Load path"]
        ECLI["ecli CLI"]
        BPFLOADER["bpf-loader-rs"]
        LIBBPF["libbpf syscalls<br/>bpf()/perf_event_open()"]
    end

    subgraph Kernel["Linux kernel"]
        VERIFIER["BPF verifier"]
        JIT["JIT compiler"]
        HOOKS["Hook sites:<br/>tracepoints · kprobe/fentry · uprobe<br/>XDP/tc · sched_ext · BPF LSM"]
        MAPS["Maps:<br/>hash · ringbuf · perf events<br/>histograms"]
    end

    OUTPUT["User space output:<br/>events, histograms, CLI"]

    SRC --> ECC
    VMLINUX --> ECC
    ECC --> PKG
    PKG --> OCI
    PKG --> GH_PAGES
    GH_PAGES --> ECLI
    OCI --> ECLI
    ECLI --> BPFLOADER
    BPFLOADER --> LIBBPF
    LIBBPF --> VERIFIER
    VERIFIER --> JIT
    JIT --> HOOKS
    HOOKS --> MAPS
    MAPS --> OUTPUT

Key design points visible across lessons:

Kernel/user separation: every program splits into kernel logic plus a user-space loader (*_user.c, or Go/Rust equivalents) handling attach, event polling, and display.
Progressive data plumbing: lesson order deliberately walks perf event array (lesson 7) before ring buffer (lesson 8) before histograms (lesson 9), then user ring buffers for kernel-bound async (lesson 35).
Minimal-example discipline: lesson 1's complete tracepoint program fits in ~25 lines including license declaration, demonstrating bpf_get_current_pid_tgid() filtering and bpf_printk output to /sys/kernel/debug/tracing/trace_pipe.

Later tiers swap the harness and keep this shape: libbpf-bootstrap's Makefile workflow from lesson 11, cilium/ebpf for Go from the starter templates, libbpf-rs from lesson 12 onward, and bpftime/wasm-bpf as alternative load-and-execute runtimes profiled in lesson 36.

Runtime Event Flow¶

The full lifecycle taught by the early course arc, from artifact to observed data:

sequenceDiagram
    participant U as User terminal (ecli)
    participant L as bpf-loader-rs / libbpf
    participant K as Kernel (verifier -> JIT)
    participant T as Tracepoint tp/syscalls/sys_enter_write
    participant M as Maps / trace_pipe
    U->>L: sudo ./ecli run package.json
    L->>K: bpf() syscall: load program + maps
    K->>K: verifier proof pass, JIT to native
    L->>T: attach via perf_event_open / link_create
    Note over T,M: every matched kernel event fires the BPF function
    T->>M: bpf_printk -> trace_pipe<br/>(later lessons: bpf_perf_event_output,<br/>bpf_ringbuf_output)
    M->>U: events streamed until Ctrl+C detaches

Hook Taxonomy Across the Curriculum¶

Hook class	Attach point	Representative lessons
Static tracepoints	`SEC("tp/syscalls/*")`, subsystem events	1, 7-11
kprobe / fentry	arbitrary kernel functions	2 vs. 3 (same tool rebuilt with the modern mechanism)
uprobe / USDT	user-space functions (bash readline, SSL, JVM GC)	5, 15, 30, 37
Networking	XDP ingress, tc classifier, sockops, TCX, Qdisc	20-21, 29, 41-42, 50, 53
Security hooks	BPF LSM (`security_*`), cgroup device control	19, 54, cgroup
Scheduler ops	sched_ext struct_ops dispatch loop	44-45
GPU/user-space	CUDA library uprobes, CUPTI subscriber, bpftime VM	47, xpu/flamegraph, 36

The deliberate rebuild of the same unlink monitor with kprobe (lesson 2) then fentry (lesson 3) is the curriculum's clearest teaching device for migration pressure off legacy probe types.

Data Plumbing Evolution¶

The event-export mechanisms are introduced in dependency order, each replacing the previous one's limits:

bpf_printk (lesson 1) — zero-setup debugging. Shared global pipe, three format args max.
perf event array (lesson 7, execsnoop) — structured events pushed on firing. No drop-safe ordering guarantee across CPUs.
ring buffer (lesson 8, exitsnoop) — single shared mmap'd buffer, kernel 5.8+, better memory efficiency.
histogram maps (lesson 9, runqlat) — in-kernel log2 bucketing via bpf_log2l() on recorded latencies. Only aggregate buckets cross the kernel/user boundary, not raw events.
user ring buffer (lesson 35) — reverses direction: async user-to-kernel commands without a syscall wake-up per message.

Sched_ext Track¶

Lessons 44-45 teach struct_ops-based CPU scheduling on kernel 6.12+, where a BPF program implements a scheduling class alongside CFS. The org cites over one million machines already running sched_ext policies in production (Meta's scx_layered among them) as adoption evidence. Lesson 45 reproduces the upstream scx_nest policy: idle-CPU nesting heuristics that trade slight latency for reduced wake-up spread on multi-core hosts.

Framework Comparison Frame¶

Lesson 1 codifies the decision matrix the curriculum teaches implicitly:

Framework	Model	Trade-off emphasized
BCC	Python front-end, runtime compilation per target	Rich helpers, but heavy dependencies and repeated compiles
libbpf / CO-RE	Ahead-of-time compiled object, skeleton headers	One compile runs everywhere. Steeper C conventions
cilium/ebpf (Go)	Go-native management of BPF objects	Idiomatic Go apps, still needs C kernel side
libbpf-rs	Rust ergonomic wrapper over libbpf	Safety in user space. Same CO-RE kernel objects
eunomia-bpf	Kernel-code-only authorship, packaged artifacts	Fastest path from `.bpf.c` to distributable tool

Compatibility Verification System¶

The compatibility matrix is machine-generated from each lesson's .config metadata rather than hand-maintained prose. For every lesson it records minimum kernel version (with provenance basis: docs statement vs. required-feature introduction vs. repository baseline), architectures, BTF requirement, exact CONFIG_* kernel options, hardware prerequisites, root requirements, and test status.

Status distribution across the ~53 tracked rows (August 2026):

CI runtime (~21): built and executed on runners — includes early tracing lessons, memleak, tc.
CI build (~24): compiled but not exercised — most sched_ext/XDP/HID/feature lessons need real devices or newer kernels.
Not in CI (~5): LSM-connect, XDP, Android, nginx, MySQL — hardware or environment constraints.
Docs only (3): introductions and reading lists.

This is why a kernel-version spread of 4.8 → 7.0 (fsession latency, lesson 52) coexists with confident instructions: each claim traces back to declared metadata plus automated evidence where feasible.

Architectural nuance captured by the matrix

Lesson 3 shows genuinely split baselines (fentry supported on x86_64 since 5.5, arm64 needing 6.0) — exactly the kind of divergence a hand-written tutorial can flatten incorrectly.

Platform Portability Notes¶

Lesson 22 documents Android as a supported target (kernel 5.15 baseline, x86_64 emulator validated in the matrix) using the kernel's built-in eBPF userspace via libbpf from an app/native process. It notes that every shipped Android phone already runs eBPF internally for network/power management. Lesson 49 covers HID-BPF (kernel 6.3+): input-device fixups loaded through hid_bpf without patching drivers, the clearest example in the course of eBPF as a general driver-extension mechanism rather than pure observability.

Research Grounding¶

Lesson 18 ties the practical track to systems research the organization considers foundational: XRP in-kernel storage functions (OSDI '22 Best Paper), Jitterbug formally verified BPF JITs (OSDI '20), Electrode eBPF-accelerated Paxos (+128% throughput, NSDI '23), BMC in-kernel Memcached caching (up to 18x throughput, NSDI '21), hXDP FPGA offload (OSDI '20), and lambda-IO computational storage (FAST '23). Maintainer affiliation with an OSDI 2025 publication underwrites the depth of the sched_ext/GPU material.

Benchmarks¶

The tutorial itself publishes no performance numbers — appropriate for instructional material. Performance-relevant claims made inside lessons attribute to their sources: bpftime's ~10x-uprobe-speedup figure comes from the org's own benchmarking, and paper-linked speedups listed above carry venue citations. Treat both as upstream claims, not independently reproduced here.

Security¶

Security considerations for using this tutorial resource: the privilege model its exercises require, isolation practices for running them, supply-chain notes for consuming its distributed artifacts, and dual-use warnings for specific lessons. Content sourced from lesson materials read August 2026 plus documented kernel eBPF privilege semantics.

Privilege Model¶

Every executable lesson requires root. On modern kernels (5.8+), the equivalent fine-grained capabilities replace blanket root for production-style loading:

Capability	Grants	Relevant to
`CAP_BPF`	program/map load, most `bpf()` operations	every lesson
`CAP_PERFMON`	performance monitoring, uprobes/kprobes attach	tracing lessons 1-14, 30-40
`CAP_NET_ADMIN`	XDP/tc/netfilter-class attachments	networking lessons 20-21, 41-42, 50, 53
`CAP_SYS_ADMIN`	legacy catch-all on older kernels, BTF write paths, LSM hook attachment path enablement	lesson 19 (BPF LSM)

Make sure that you know whether your distro disables unprivileged eBPF entirely (recommended hardening state, because early lessons assume privileged loading anyway):

sysctl kernel.unprivileged_bpf_disabled

Lesson 0's framing

The tutorial presents the verifier as the safety boundary: programs undergo static proof before execution. This prevents kernel crashes, memory unsafety, and information leaks regardless of caller privileges. JIT compilation then translates bytecode to native code with W^X discipline. These guarantees protect the kernel — not the data an authorized tracer collects.

Data Sensitivity While Learning¶

Several lessons decode genuinely sensitive material by design:

lesson 30 (sslsniff) attaches uprobes to OpenSSL/BoringSSL/GnuTLS entry points and emits TLS plaintext — running it captures any HTTPS traffic of local applications.
lessons 5, 15, 37 capture user-space function arguments (shell input, JVM GC telemetry, Rust binaries).
lesson 39-40 record nginx request URLs and MySQL queries verbatim.

Treat these like credential-dumping tools: run only on machines you own or are explicitly authorized to instrument. Shared hosts, employer laptops, and CI runners shared across teams are poor practice targets even when technically permitted by root access.

Isolation Practice¶

Disposable VM preferred — the compatibility matrix shows Ubuntu x86_64/arm64 runners as the validated baseline. A snapshot-resettable VM (for example, a stock Ubuntu cloud image) contains mistakes cleanly. Enable nested virtualization note applies if testing XDP over virtual links.
Android lesson (22) requires a device or emulator — the emulator route keeps kernel experiments off any personal handset.
Networking lessons bind to real interfaces (biopattern wants block devices, XDP lessons name a NIC). Prefer lab bridges or veth pairs so host connectivity does not become collateral damage.
bleeding-edge kernels — lessons gated at 6.19/7.0 features imply custom-mainline test VMs, never production or daily-driver systems.

Dual-Use Lesson Warnings¶

A deliberate course arc teaches offense-shaped primitives so defenders understand attacker technique:

Lesson	Technique demonstrated	Handling caution
24-hide	hiding PIDs/files from user space	run in isolated namespace/kernel only
25-signal	terminating arbitrary PIDs from inside BPF (`bpf_send_signal`)	scope PID filters before loading
26-sudo	privilege escalation via file-content manipulation	textbook demonstration environment only
27-replace	transparent tampering of file reads	same
34-syscall	rewriting live syscall arguments	can corrupt unrelated processes without filters
51-tcp-quarantine	severing established connections precisely	disruptive by intent. Lab networks

None of these lessons publish exploit chains beyond well-known kernel-programmability concepts. The value is defender familiarity. Mirror upstream's own positioning: authorization and lawful use are assumed prerequisites.

Artifact Supply Chain¶

ecli consumes either locally compiled packages or remote ones:

$ sudo ./ecli run ghcr.io/eunomia-bpf/execve:latest

Remote mode executes third-party-compiled bytecode under your kernel account. For anything security-sensitive:

prefer compiling locally from the repository you inspected (ecc minimal.bpf.c) over pulling ghcr.io images.
treat --no-cache refetches like dependency updates — review what changed.
the OCI contents include exported event headers/config JSON. Inspect package.json alongside the .bpf.c it claims to match.

This follows the vault-wide reference-first rule: the repository source above is the trusted origin for any tool behavior described in these notes.

Lifecycle Residue¶

Lesson 28 teaches that pinned, linked-away-from-process programs can outlive their launcher. After experiments, examine residue rather than assuming teardown:

sudo bpftool prog show
sudo bpftool map show
sudo rm -f /sys/fs/bpf/<pinned-path>   # only pins you created

Orphaned kprobes silently keep collecting — a privacy leak in exactly the spirit the tracing lessons demonstrate.

Questions¶

Do the org's newer BPF-token features (features/bpf_token lesson) map onto a delegated least-privilege workflow for classroom/shared-lab settings?
Is there upstream guidance yet for verifying signature/provenance of OCI-published compiled tools?