Architecture¶

How a bpftrace script becomes kernel-executing probes and returned data: the compile chain, probe providers, the PERCPU map/aggregation engine, and the sync/async split that governs performance.

Script-to-Kernel Pipeline¶

bpftrace is a compiler with an attached runtime, not an interpreter. Upstream language documentation states the contract plainly: LLVM compiles scripts into eBPF bytecode, and the tool interacts with the Linux BPF subsystem through libbpf and bcc.

flowchart TB
    subgraph Frontend["Compiler front end"]
        BT["script.bt / -e one-liner<br/>probe clauses + actions"]
        CLANG["Clang-based parser<br/>(C preprocessor passes, #include)"]
        SEMA["Semantic analysis + AST<br/>builtins: pid / comm / args / retval / nsecs"]
    end

    subgraph Backend["LLVM backend"]
        IR["LLVM IR + optimization passes"]
        OBJ["BPF ELF objects embedded in the bpftrace binary"]
    end

    subgraph Runtime["Runtime (libbpf + bcc)"]
        ATTACH["Probe attachment engine"]
        MAPS["PERCPU maps:<br/>count() / sum() / hist() / lhist()<br/>associative arrays @key"]
        EVENTS["Event buffer:<br/>print(), printf(), exit() emissions"]
    end

    KERNEL["Linux kernel<br/>verifier -> JIT -> attached programs"]

    BT --> CLANG --> SEMA --> IR --> OBJ
    OBJ --> ATTACH
    ATTACH --> KERNEL
    ATTACH --> MAPS
    MAPS --> EVENTS
    EVENTS --> OUTPUT["stdout text / ASCII histograms"]

Design consequences worth internalizing:

The Clang front end means C-style types and preprocessor includes work inside scripts — typed probe arguments come from the same headers CO-RE tools use.
Programs are compiled once per invocation and loaded as real BPF objects. There is no per-event interpretation overhead at run time.
Aggregation happens in-kernel: hist() buckets, counters, and sums update in PERCPU map memory on every hit. Only the reduced result crosses to user space.

Probe Providers¶

Probes select where programs attach. Verified syntax families from upstream docs and tutorial material:

Provider	Address form	Example from docs
tracepoint	`tracepoint:subsystem:event`	`tracepoint:syscalls:sys_enter_openat`
kprobe	`kprobe:function`	`kprobe:tcp_*` (wildcard class match shown upstream)
kretprobe	`kretprobe:function`	`kretprobe:vfs_read { @bytes = hist(retval); }`
uprobe / USDT	user function markers	used for library/application entry points
interval	`interval:unit:n`	`interval:ms:100 { @ = count(); }`
profile	sampling timers	periodic stack capture

Discovery is part of the architecture — -l 'tracepoint:syscalls:sys_enter_*' enumerates attachable events before any script exists, with grep-style filtering downstream.

Providers map onto distinct kernel mechanisms, which is why upstream's required-config list fans out the way it does:

Provider	Kernel mechanism	Required config (from dependency_support.md)
tracepoint	static instrumentation hook table	`CONFIG_BPF_EVENTS=y`, `CONFIG_FTRACE_SYSCALLS=y`
kprobe / kretprobe	ftrace function-event registration	`CONFIG_KPROBES=y`, `CONFIG_KPROBE_EVENTS=y`, `CONFIG_FUNCTION_TRACER=y`
uprobe / USDT	inode-backed user-space breakpoints	`CONFIG_UPROBES=y`, `CONFIG_UPROBE_EVENTS=y`
profile / interval	BPF program re-triggered by timing	base JIT surface (`CONFIG_BPF_JIT=y`)

Builtins available inside action blocks include process identity (pid, comm, tid, cgroup, uid, username), time sources (nsecs), probe metadata (func, probe), and per-provider argument access (args, retval) — all documented in the upstream language reference.

Session Structure¶

A script session composes three block kinds seen throughout upstream examples:

BEGIN runs once at load — canonical use is printing a banner (printf("Tracing open syscalls... Hit Ctrl-C to end.\n");).
Probe clauses carry the recording logic. Comma-separated targets share one body when they consume identical arguments.
Drain points are any clause that reads or clears state. exit() terminates collection from inside the kernel context. Residual buffered events flush on teardown.

The vendored one-liner tutorial inside the tutorial repo teaches exactly this shape first. This makes it the lowest-friction on-ramp to everything else in the eBPF ecosystem.

Map and Aggregation Engine¶

The stdlib distinguishes two cost classes explicitly (upstream stdlib notes):

Efficient writes, expensive sync reads. Maps like @ = count(); and @ = sum(x); are backed by thread-safe PERCPU maps: each CPU increments its slot locklessly. This prevents cross-CPU contention on hot paths.

sequenceDiagram
    participant CPUs as Kernel CPUs (hit path)
    participant PM as PERCPU map slots
    participant R as Reader interval (async)
    CPUs->>PM: lock-free local-slot increment
    Note over PM: zero cross-CPU coordination during tracing
    R->>PM: print(@) reads slots (cheap async emission)
    PM-->>R: reduced ASCII histogram / total printed

Sync reads behave differently: iterating every CPU's slot inline is documented as costly (if (@ > 10) style checks inside intervals). The idiomatic shape pairs cheap producers on interval:ms ticks with a slower interval:s consumer doing async print(@) then clear(@).

Aggregation primitives: count(), sum(), plus max/min/avg/sum parameterized tseries(@v, 1s, 5, "avg") windows for time-bucketed series, lhist(value, min, max, step) linear histograms, and log2-power hist() for latency distributions. Associative maps key on builtins (@open[comm] = count();) without extra syntax.

Cost Model by Operation¶

Explicit economics documented upstream, worth treating as design law when composing scripts:

Operation	Class	Documented behavior
`@ = count();` / `@ = sum(x);` on hit path	write	thread-safe PERCPU increment, contention-free
`print(@)` from an interval clause	read, async	cheap emission of reduced map contents
inline `if (@ > 10)` style checks	read, sync	explicitly flagged expensive — iterates CPUs
`printf()` / string formatting in actions	output	asynchronous through the event buffer
`clear(@)` / `zero(@)`	maintenance	resets aggregation windows between drains

The intended cadence falls straight out of the table: instrument hot paths with pure writes, schedule drains at human-auditable intervals, and never let user-space judgment calls execute synchronously against live maps mid-trace.

Async Event Delivery¶

Actions like printf(), map print(), and exit() are asynchronous by design — they enqueue into an event channel drained by the CLI rather than executing synchronously in program context. This is what keeps output ordering relaxed but probe execution cheap. Two practical patterns fall out:

Fire-and-forget logging: unconditional printf inside kprobes.
Consumer loops: producers write maps continuously. A second interval clause drains them.

flowchart LR
    P1["kretprobe:vfs_read<br/>@bytes = hist(retval)"] --> M[("@bytes PERCPU map")]
    P2["interval:ms:100<br/>@ = count()"] --> M2[("@ PERCPU map")]
    M2 --> C["interval:s:10<br/>print(@); clear(@)"]
    M --> O["process exit / EOF drain"] --> S["stdout histogram"]

Distribution Modes¶

Two packaging paths exist upstream, both verified against current docs:

Distro packages — the README's primary path: apt install bpftrace (Debian/Ubuntu family), dnf install bpftrace (Fedora/CentOS). The distribution ships a binary whose Clang/LLVM/libbpf build matrix already matches that distro's kernels.
Source builds — canonical CMake flow (cmake -B build -DCMAKE_BUILD_TYPE=Release && make -C build -j$(nproc)) after dependency installation, or Nix flakes including git submodules via .?submodules=1.

An ahead-of-time mode additionally compiles scripts into redistributable bundles intended for environments that lack build tooling. Bundle compatibility across kernel minor versions is an open operational question noted in this topic's Questions.

The two paths imply different freshness envelopes: distro packages lag upstream releases by design (stability over recency), while source/Nix builds track v0.26.1-era behavior (June 2026 current) at the cost of owning the dependency matrix — Clang, LLVM, libbpf, and bcc versions must cohere with each other and with the target kernel. Production fleets typically split: packaged binary on hosts, source-built environment on the analysis workstation where new probe types get explored first.

Benchmarks¶

No first-party throughput numbers are published in the referenced docs, and none are invented here. Documented performance characteristics that matter operationally: PERCPU aggregation writes prevent contention. Synchronous map traversal is explicitly flagged expensive. Async event emission keeps printf-style output off the instrumented path's critical section. Treat absolute overhead as workload-dependent and measure with the profile provider itself when it matters.

Security¶

Security posture of operating bpftrace: capability requirements, what tracing exposes, third-party script risk, and kernel hardening interactions. Privilege semantics follow documented kernel eBPF behavior. Feature facts trace to upstream dependency_support.md and language docs verified August 2026.

Capability Model¶

bpftrace loads and attaches real BPF programs. As a result, every probe class requires elevated privileges at runtime:

Capability	Needed for	Notes
`CAP_BPF` (5.8+)	program/map load via bpf() syscall	pairs with one of the below. Alone insufficient for attach
`CAP_PERFMON`	kprobes, uprobes, tracepoints, sampling timers	replaces legacy `CAP_SYS_ADMIN` on modern kernels
`CAP_SYS_ADMIN`	pre-5.8 kernels and certain debugfs attach paths	still required in some distro/hypervisor combos
root	packaging default — sudo invocation as shown upstream docs	simplest correct posture

Make sure that you know whether unprivileged BPF is disabled (a hardened baseline, and bpftrace assumes privileged operation anyway):

sysctl kernel.unprivileged_bpf_disabled

Container runtimes need explicit capability export (--privileged or targeted caps). Namespace-confined root without perf/BPF grants fails at load, not at parse.

Data Exposure While Tracing¶

A tracing language is a data-exfiltration primitive by construction:

str(args.filename) prints filesystem paths — including tokens passed to open calls.
uprobe/USDT probes capture user-space function arguments: credentials buffers, request bodies, serialization inputs.
kretprobe histograms (hist(retval)) leak distributions that are themselves sensitive on multi-tenant hosts.

Operate bpftrace only on systems you own or are authorized to instrument. Incident-tooling usage must be governed by the same approvals as packet capture — same blast radius, different layer.

Third-Party Script Risk¶

.bt scripts look inert but compile to arbitrary-kernel-access programs under your privileges. Treat a borrowed script like a root shell:

Read every probe clause before running — make sure that the target functions match the claimed purpose.
Watch action bodies for unexpected writes or broad wildcards (kprobe:*-class matches instrument far more than needed).
Prefer vendoring community scripts into a reviewed repo over curl-piping from gists. Treat updates as code review events.

The Clang front end accepts C preprocessor includes — complex upstream scripts can pull headers that materially change what they access, so review with expanded context (-d-style dry parsing exists precisely for this audit step).

Kernel Hardening Interactions¶

Hardened hosts intentionally restrict what bpftrace needs. Expect friction and document exceptions rather than loosening globally:

Lockdown integrity mode commonly gates debugfs/tracing interfaces several providers depend on (CONFIG_DEBUG_FS=y is on the required list).
secureBoot + signed-module policies can block unsigned BPF loads depending on distribution policy hooks.
Vendor kernels (cloud images, LTS-minus builds) frequently ship with CONFIG_KPROBE_EVENTS or CONFIG_UPROBE_EVENTS off — the readiness grep lives in Operations.
auditd/seccomp profiles for service accounts will see unusual bpf() syscall traffic when agents run. Allowlist deliberately.

For permanent fleet deployment prefer purpose-built daemons compiled against libbpf over ad-hoc bpftrace sessions — smaller privilege surface, reviewed binaries, no scripting interpreter present on hosts.

Overhead as an Availability Concern¶

Aggressive probing is a self-inflicted outage vector on busy systems. Mitigations grounded in documented mechanics:

Keep hit-path actions minimal. Use PERCPU map aggregation instead of per-event user-space wakeups.
Drain maps on slow intervals asynchronously (print(@) consumer pattern from the stdlib docs) — sync CPU iteration inside hot clauses is explicitly flagged expensive.
Sample (profile, ms-scale interval:) rather than record when the question tolerates estimation.
Cap concurrency of parallel sessions. Overlapping full-kernel kprobes multiply already-paid costs.

Lifecycle Hygiene¶

Ctrl-C tears down attached programs, but crashed sessions can strand resources. After abnormal exits:

sudo bpftool prog show && sudo bpftool map show

Standalone bpftrace rarely pins objects persistently. Residue most often appears when wrapping automation kills sessions mid-load.

Questions¶

Do hardened distros (Fedora lockdown defaults, Ubuntu FIPS profiles) ship turnkey "trace-ops" role definitions mapping bpftrace needs to exact capabilities?
What policy linting exists for .bt review — static checks for wildcard breadth, sync-read anti-patterns, or PII-shaped argument access?