Skip to content

Architecture

How a bpftrace script becomes kernel-executing probes and returned data: the compile chain, probe providers, the PERCPU map/aggregation engine, and the sync/async split that governs performance.

Script-to-Kernel Pipeline

bpftrace is a compiler with an attached runtime, not an interpreter. Upstream language documentation states the contract plainly: LLVM compiles scripts into eBPF bytecode, and the tool interacts with the Linux BPF subsystem through libbpf and bcc.

flowchart TB
    subgraph Frontend["Compiler front end"]
        BT["script.bt / -e one-liner<br/>probe clauses + actions"]
        CLANG["Clang-based parser<br/>(C preprocessor passes, #include)"]
        SEMA["Semantic analysis + AST<br/>builtins: pid / comm / args / retval / nsecs"]
    end

    subgraph Backend["LLVM backend"]
        IR["LLVM IR + optimization passes"]
        OBJ["BPF ELF objects embedded in the bpftrace binary"]
    end

    subgraph Runtime["Runtime (libbpf + bcc)"]
        ATTACH["Probe attachment engine"]
        MAPS["PERCPU maps:<br/>count() / sum() / hist() / lhist()<br/>associative arrays @key"]
        EVENTS["Event buffer:<br/>print(), printf(), exit() emissions"]
    end

    KERNEL["Linux kernel<br/>verifier -> JIT -> attached programs"]

    BT --> CLANG --> SEMA --> IR --> OBJ
    OBJ --> ATTACH
    ATTACH --> KERNEL
    ATTACH --> MAPS
    MAPS --> EVENTS
    EVENTS --> OUTPUT["stdout text / ASCII histograms"]

Design consequences worth internalizing:

  • The Clang front end means C-style types and preprocessor includes work inside scripts — typed probe arguments come from the same headers CO-RE tools use.
  • Programs are compiled once per invocation and loaded as real BPF objects; there is no per-event interpretation overhead at run time.
  • Aggregation happens in-kernel: hist() buckets, counters, and sums update in PERCPU map memory on every hit; only the reduced result crosses to user space.

Probe Providers

Probes select where programs attach. Verified syntax families from upstream docs and tutorial material:

Provider Address form Example from docs
tracepoint tracepoint:subsystem:event tracepoint:syscalls:sys_enter_openat
kprobe kprobe:function kprobe:tcp_* (wildcard class match shown upstream)
kretprobe kretprobe:function kretprobe:vfs_read { @bytes = hist(retval); }
uprobe / USDT user function markers used for library/application entry points
interval interval:unit:n interval:ms:100 { @ = count(); }
profile sampling timers periodic stack capture

Discovery is part of the architecture — -l 'tracepoint:syscalls:sys_enter_*' enumerates attachable events before any script exists, with grep-style filtering downstream.

Providers map onto distinct kernel mechanisms, which is why upstream's required-config list fans out the way it does:

Provider Kernel mechanism Required config (from dependency_support.md)
tracepoint static instrumentation hook table CONFIG_BPF_EVENTS=y, CONFIG_FTRACE_SYSCALLS=y
kprobe / kretprobe ftrace function-event registration CONFIG_KPROBES=y, CONFIG_KPROBE_EVENTS=y, CONFIG_FUNCTION_TRACER=y
uprobe / USDT inode-backed user-space breakpoints CONFIG_UPROBES=y, CONFIG_UPROBE_EVENTS=y
profile / interval BPF program re-triggered by timing base JIT surface (CONFIG_BPF_JIT=y)

Builtins available inside action blocks include process identity (pid, comm, tid, cgroup, uid, username), time sources (nsecs), probe metadata (func, probe), and per-provider argument access (args, retval) — all documented in the upstream language reference.

Session Structure

A script session composes three block kinds seen throughout upstream examples:

  • BEGIN runs once at load — canonical use is printing a banner (printf("Tracing open syscalls... Hit Ctrl-C to end.\n");).
  • Probe clauses carry the recording logic; comma-separated targets share one body when they consume identical arguments.
  • Drain points are any clause that reads or clears state. exit() terminates collection from inside the kernel context; residual buffered events flush on teardown.

The vendored one-liner tutorial inside the tutorial repo teaches exactly this shape first, making it the lowest-friction on-ramp to everything else in the eBPF ecosystem.

Map and Aggregation Engine

The stdlib distinguishes two cost classes explicitly (upstream stdlib notes):

Efficient writes, expensive sync reads. Maps like @ = count(); and @ = sum(x); are backed by thread-safe PERCPU maps: each CPU increments its slot locklessly, avoiding cross-CPU contention on hot paths.

sequenceDiagram
    participant CPUs as Kernel CPUs (hit path)
    participant PM as PERCPU map slots
    participant R as Reader interval (async)
    CPUs->>PM: lock-free local-slot increment
    Note over PM: zero cross-CPU coordination during tracing
    R->>PM: print(@) reads slots (cheap async emission)
    PM-->>R: reduced ASCII histogram / total printed

Sync reads behave differently: iterating every CPU's slot inline is documented as costly (if (@ > 10) style checks inside intervals). The idiomatic shape pairs cheap producers on interval:ms ticks with a slower interval:s consumer doing async print(@) then clear(@).

Aggregation primitives: count(), sum(), plus max/min/avg/sum parameterized tseries(@v, 1s, 5, "avg") windows for time-bucketed series, lhist(value, min, max, step) linear histograms, and log2-power hist() for latency distributions. Associative maps key on builtins (@open[comm] = count();) without extra syntax.

Cost Model by Operation

Explicit economics documented upstream, worth treating as design law when composing scripts:

Operation Class Documented behavior
@ = count(); / @ = sum(x); on hit path write thread-safe PERCPU increment, contention-free
print(@) from an interval clause read, async cheap emission of reduced map contents
inline if (@ > 10) style checks read, sync explicitly flagged expensive — iterates CPUs
printf() / string formatting in actions output asynchronous through the event buffer
clear(@) / zero(@) maintenance resets aggregation windows between drains

The intended cadence falls straight out of the table: instrument hot paths with pure writes, schedule drains at human-auditable intervals, and never let user-space judgment calls execute synchronously against live maps mid-trace.

Async Event Delivery

Actions like printf(), map print(), and exit() are asynchronous by design — they enqueue into an event channel drained by the CLI rather than executing synchronously in program context. This is what keeps output ordering relaxed but probe execution cheap. Two practical patterns fall out:

  1. Fire-and-forget logging: unconditional printf inside kprobes.
  2. Consumer loops: producers write maps continuously; a second interval clause drains them.
flowchart LR
    P1["kretprobe:vfs_read<br/>@bytes = hist(retval)"] --> M[("@bytes PERCPU map")]
    P2["interval:ms:100<br/>@ = count()"] --> M2[("@ PERCPU map")]
    M2 --> C["interval:s:10<br/>print(@); clear(@)"]
    M --> O["process exit / EOF drain"] --> S["stdout histogram"]

Distribution Modes

Two packaging paths exist upstream, both verified against current docs:

  1. Distro packages — the README's primary path: apt install bpftrace (Debian/Ubuntu family), dnf install bpftrace (Fedora/CentOS). The distribution ships a binary whose Clang/LLVM/libbpf build matrix already matches that distro's kernels.
  2. Source builds — canonical CMake flow (cmake -B build -DCMAKE_BUILD_TYPE=Release && make -C build -j$(nproc)) after dependency installation, or Nix flakes including git submodules via .?submodules=1.

An ahead-of-time mode additionally compiles scripts into redistributable bundles intended for environments that lack build tooling; bundle compatibility across kernel minor versions is an open operational question noted in this topic's Questions.

The two paths imply different freshness envelopes: distro packages lag upstream releases by design (stability over recency), while source/Nix builds track v0.26.1-era behavior (June 2026 current) at the cost of owning the dependency matrix — Clang, LLVM, libbpf, and bcc versions must cohere with each other and with the target kernel. Production fleets typically split: packaged binary on hosts, source-built environment on the analysis workstation where new probe types get explored first.

Benchmarks

No first-party throughput numbers are published in the referenced docs, and none are invented here. Documented performance characteristics that matter operationally: PERCPU aggregation writes avoid contention; synchronous map traversal is explicitly flagged expensive; async event emission keeps printf-style output off the instrumented path's critical section. Treat absolute overhead as workload-dependent and measure with the profile provider itself when it matters.