From 067350789615a0ff216e5248ab53a1dff0b252bf Mon Sep 17 00:00:00 2001 From: Necco Ceresani Date: Thu, 3 Sep 2026 10:12:57 -0400 Subject: [PATCH] count dropped execs instead of losing them silently Jacob's two seeding fixes made a third bug visible. With the scope finally covering pre-existing subtrees, a fork storm produces far more events than before, and most of them were disappearing with nothing on screen to say so: 96,000 execs fired across 24 workers, ~36,000 shown, ~60,000 gone. The loss itself is a real limit rather than a defect. Ring-buffer delivery is bound by bytes moved, not events, and a record carries a 1 KiB argument window, so throughput caps around 2,700 execs/s. Measured with an empty callback, so it is the runtime's ceiling and not this script's per-event work (normalize is ~4ms per 40k records, model.add ~12ms; neither is the bottleneck). A `make -j24` can outrun it. A normal npm install, at tens per second, cannot come close. What was wrong is that it was silent, so every count read as a total when it was a floor. bpf_ringbuf_output's return value was discarded. A stats_map ARRAY now counts emitted and dropped, JS polls it, and a non-zero drop count appears on the verdict line and in the headless report, which also says outright that the numbers below it are a floor. test/drops.sh asserts the accounting closes: captured + dropped must be at least what actually ran, drops must be non-zero at storm scale (otherwise the drop path silently goes untested), and the report must explain what the number means. Verified: 36,490 + 59,536 = 96,026 against 96,000 fired. Two things tried and rejected, recorded so they are not retried: - Shrinking ARGV_BUF to 256 raises the ceiling to ~10,200/s, a 3.8x gain, but truncates 12% of real records (measured against the checked-in captures: p50 argv is 48 bytes, p99 is 1026). Wrong trade for a tool whose job is showing you what ran. - Emitting a variable-length record (header + args_len, mean 143 bytes instead of a fixed 1076) delivers nothing at all. The ring is bound with `btf_struct: "exec_event"`, so the consumer decodes fixed-size records and a short write is not seen. Left in a comment at the emit site. Also removes setCgroup and its unread target_cgid filter. Nothing ever called it, so container mode has always been a plain pid subtree; now that the existing tree is seeded from the process graph, the tgid set IS the scope and a second overlapping filter would only be a way for the two to disagree. --- README.md | 9 ++++- src/bpf/exectop.bpf.c | 46 +++++++++++++++++++++- src/components/verdict.jsx | 11 +++++- src/main.jsx | 4 +- src/probes/capture.js | 24 ++++++++--- src/probes/exec.js | 32 ++++++++++++--- src/probes/probe.js | 1 + test/drops.sh | 81 ++++++++++++++++++++++++++++++++++++++ 8 files changed, 190 insertions(+), 18 deletions(-) create mode 100755 test/drops.sh diff --git a/README.md b/README.md index 31db4df..6a2f2e1 100644 --- a/README.md +++ b/README.md @@ -290,12 +290,16 @@ A BPF program that loads on your laptop can be rejected by an older kernel's ver `make veristat` loads every program through the verifier on your own kernel and reports per-program complexity. [`.github/workflows/kernel-matrix.yml`](.github/workflows/kernel-matrix.yml) builds the object and boots each kernel in its matrix in a VM, failing if any verifier rejects it. Run the same matrix locally on Linux with KVM using `make veristat-matrix`. -The aggregation has its own suite, which needs no kernel: +Three suites, split by what they can reach: ```sh -node test/heuristics.test.mjs +node test/heuristics.test.mjs # folding + outlier scoring — no kernel needed +bash test/capture.sh # the kernel seam: seeding, fork propagation, threads +bash test/drops.sh # ring-buffer drop accounting under a fork storm ``` +The first replays recorded captures through `lib/model.js`, so it runs anywhere node does. The other two need a Linux box with the probe built, because they exercise the path from kernel to traced set to ring buffer, which is where two whole-subtree bugs lived: a scope that only ever spread forward through `fork`, so anything already running was invisible, and a parent lookup keyed by tid against a map keyed by tgid, which silently dropped every process spawned by a pre-existing thread. Neither was reachable from a fixture replay. + It runs the folding and the outlier scoring against five recorded captures in `test/` and asserts both halves of the claim: three benign builds produce no findings, and the adversarial ones produce the expected findings. The captures are real probe output rather than synthesized fixtures, deliberately: an earlier synthetic version of this suite passed while the heuristics were badly wrong. ## Try it without real traffic @@ -325,6 +329,7 @@ Container mode additionally needs Docker reachable from the host running the pro > `exectop` is observability, not enforcement. It tells you what was launched; it does not stop, delay, or modify anything. For a kernel-enforced boundary around what a process can touch, [`agent-lock`](https://github.com/yeet-src/agent-lock) is the sibling that blocks rather than reports. - **Anything that doesn't exec.** A dependency that does its damage inside Node, Python, or the JVM without launching a program is invisible here. Fetching a URL with `fetch()` looks like nothing; fetching it with `curl` is a row. This is the boundary that matters most when reasoning about what the findings panel can and cannot catch. +- **Execs beyond about 2,700 a second.** Delivery from the kernel is bound by bytes moved rather than events, and a record carries a 1 KiB argument window, so a parallel fork storm outruns the ring buffer. Measured: 96,000 execs fired across 24 workers, 36,000 captured and 60,000 dropped. The count is not lost, it is reported: the verdict line shows `59,431 dropped` and the headless report says the numbers below it are a floor. Normal builds are nowhere near this (a real `npm install` is tens per second), but a `make -j24` can reach it. - **Arguments past 1 KiB.** The kernel copies a fixed 1024-byte window of the argument blob and marks the record truncated. A very long compiler invocation is cut off; the program, its flags and the timing stay correct. - **Children that already existed** when you attach with `--pid` or `--container`. Membership propagates at `fork`, so a process that forked before you attached is outside the set until it forks again. Launch mode has no such gap, which is the reason to prefer it. - **Which files a process touched, or what it sent.** This is process launches only. For file access see [`agent-lock`](https://github.com/yeet-src/agent-lock), for HTTP see [`container-traffic`](https://github.com/yeet-src/container-traffic), for raw packets [`pktscope`](https://github.com/yeet-src/pktscope). diff --git a/src/bpf/exectop.bpf.c b/src/bpf/exectop.bpf.c index 4ef805b..efc55a0 100644 --- a/src/bpf/exectop.bpf.c +++ b/src/bpf/exectop.bpf.c @@ -46,9 +46,15 @@ struct exec_event { __u8 args[ARGV_BUF]; }; + struct { __uint(type, BPF_MAP_TYPE_RINGBUF); - __uint(max_entries, 1 << 22); // 4 MiB — exec storms are bursty + // 32 MiB. A record is ~1 KiB (the argv window dominates), so 4 MiB held + // only ~4000 events and a parallel fork storm outran it in under a second: + // measured 7k of 36k execs arriving, the rest dropped silently. This is + // the cheapest lever, and the drop counter above reports what still slips + // through rather than letting a burst quietly truncate the picture. + __uint(max_entries, 1 << 25); } events SEC(".maps"); // The traced set: tgid -> depth below the root. Seeded from userspace with the @@ -82,6 +88,23 @@ struct { __type(value, __u64); } fork_ts SEC(".maps"); +// Event counters, read by JS. `emitted` and `dropped` are the two numbers that +// matter: a burst that outruns the ring buffer loses execs, and losing them +// silently is worse than the loss. Measured on a 12-way fork storm, 36k execs +// in a few seconds, only ~7k arrived — 80% gone with nothing on screen to say +// so. The UI now shows a dropped count when it is non-zero. +struct counters { + __u64 emitted; + __u64 dropped; +}; + +struct { + __uint(type, BPF_MAP_TYPE_ARRAY); + __uint(max_entries, 1); + __type(key, __u32); + __type(value, struct counters); +} stats_map SEC(".maps"); + // Set from JS before the subscription opens (probe.bss). When non-zero, only // tasks in this cgroup are eligible — the cgroup is the scope and `traced` // narrows it to the actual subtree. When zero, `traced` alone decides. @@ -102,6 +125,9 @@ static __always_inline int in_scope(__u32 tgid, __u32 *depth_out) struct traced_val *d = bpf_map_lookup_elem(&traced, &k); if (!d) return 0; + // target_cgid is inert: the traced tgid set is the scope, seeded from the + // process graph and grown at fork. Kept as a knob for a future + // cgroup-only mode, and it reads as 0 unless something patches it. if (target_cgid && bpf_get_current_cgroup_id() != target_cgid) return 0; *depth_out = d->depth; @@ -198,6 +224,22 @@ int on_exec(struct trace_event_raw_sched_process_exec *ctx) e->args_len = (n == 0) ? (__u32)len : 0; } - bpf_ringbuf_output(&events, e, sizeof(*e), 0); + // A variable-length emit (HDR_BYTES + args_len) was tried here to raise + // throughput, since the ring is bound by bytes moved and the median argv + // blob is only 48 bytes against a 1024-byte field. It does not work: the + // ring buffer is bound on the JS side with `btf_struct: "exec_event"`, so + // the consumer decodes fixed-size records and a short write delivers + // nothing at all (measured: 0 events received). Raising the ceiling would + // mean a smaller ARGV_BUF, which truncates 12% of real records, or a + // second smaller event type. Left as a documented limit instead: the + // drop counter reports what is lost rather than hiding it. + long sent = bpf_ringbuf_output(&events, e, sizeof(*e), 0); + struct counters *c = bpf_map_lookup_elem(&stats_map, &zero); + if (c) { + if (sent < 0) + __sync_fetch_and_add(&c->dropped, 1); + else + __sync_fetch_and_add(&c->emitted, 1); + } return 0; } diff --git a/src/components/verdict.jsx b/src/components/verdict.jsx index 781e994..a9cf90e 100644 --- a/src/components/verdict.jsx +++ b/src/components/verdict.jsx @@ -2,9 +2,9 @@ // before you read anything else — the total, the rate, a sparkline, the // dominant behavior, and whether anything looks out of place. import { Box, Text } from "yeet:tui"; -import { C_BAD, C_DIM, C_FAINT, C_OK, C_TITLE, BUCKET, fmtAge, fmtCount, fmtRate, pad, sparkline } from "@/lib/format.js"; +import { C_BAD, C_DIM, C_FAINT, C_OK, C_TITLE, C_WARN, BUCKET, fmtAge, fmtCount, fmtRate, pad, sparkline } from "@/lib/format.js"; -export default ({ tick, stats, buckets, outliers, idle, width: W }) => ( +export default ({ tick, stats, buckets, outliers, idle, dropped, width: W }) => ( {() => { @@ -59,6 +59,13 @@ export default ({ tick, stats, buckets, outliers, idle, width: W }) => ( if (quiet != null && quiet > 10) { runs.push({` · quiet for ${fmtAge(quiet)}`}); } + // A dropped exec means the counts below are a floor, not a total. Say + // so where the totals are, rather than letting a burst quietly + // truncate the picture. + const lost = dropped?.() ?? 0; + if (lost > 0) { + runs.push({` · ${fmtCount(lost)} dropped`}); + } // Clear the rest of the line (see above — no erase-in-line exists). const used = runs.reduce((n, r) => n + String(r?.props?.children ?? "").length, 0); runs.push({" ".repeat(Math.max(0, W() - used - 2))}); diff --git a/src/main.jsx b/src/main.jsx index aa7d49d..43853ce 100644 --- a/src/main.jsx +++ b/src/main.jsx @@ -19,7 +19,7 @@ */ import { Box, Text, computed, mount, signal } from "yeet:tui"; import { - buckets, flashes, folds, idle, outliers, paused, seedExisting, seedRoot, setCgroup, setPaused, stats, status, tick, + buckets, dropped, flashes, folds, idle, outliers, paused, seedExisting, seedRoot, setPaused, stats, status, tick, } from "@/probes/exec.js"; import { commOf, containerRoot, descendantsOf, listContainers, procTable } from "@/lib/scope.js"; import { C_FAINT, C_DIM } from "@/lib/format.js"; @@ -211,7 +211,7 @@ const Root = (size) => { return ( - + diff --git a/src/probes/capture.js b/src/probes/capture.js index 95cc246..9a2c901 100644 --- a/src/probes/capture.js +++ b/src/probes/capture.js @@ -9,7 +9,7 @@ // // It also writes the raw normalized stream to /tmp/exectop-capture.json so a // run can be replayed against changed heuristics without re-running the load. -import { HashMap, RingBuf } from "yeet:bpf"; +import { ArrayMap, HashMap, RingBuf } from "yeet:bpf"; import { control } from "./probe.js"; import { normalize } from "../lib/argv.js"; import { createModel } from "../lib/model.js"; @@ -17,6 +17,18 @@ import { descendantsOf, procTable } from "../lib/scope.js"; const traced = new HashMap(control, "traced"); const events = new RingBuf(control, "events"); +const statsMap = new ArrayMap(control, "stats_map"); + +// Kernel-side drop count. A non-zero value means the ring filled faster than +// it drained, so every number in the report is a floor rather than a total. +const dropped = async () => { + try { + const c = await statsMap.lookup(0); + return Number(c?.dropped ?? 0); + } catch { + return 0; + } +}; const root = Number(yeet.args?._?.[0] ?? 0); const secs = Number(yeet.args?._?.[1] ?? 30); @@ -42,14 +54,16 @@ const sub = await events.subscribe((w) => { model.add(e); }); -setTimeout(() => { - report(); +setTimeout(async () => { + await report(); yeet.exit(); }, secs * 1000); -function report() { +async function report() { const total = model.total; - console.log(`\n=== ${total} execs in ${model.elapsed.toFixed(0)}s (${model.rate().toFixed(1)}/s) ===\n`); + const lost = await dropped(); + console.log(`\n=== ${total} execs in ${model.elapsed.toFixed(0)}s (${model.rate().toFixed(1)}/s)${lost ? ` — ${lost} DROPPED` : ""} ===\n`); + if (lost) console.log(`[warn] ${lost} execs were dropped: the ring buffer filled faster than it drained, so every count below is a floor.\n`); console.log("-- doing --"); for (const b of model.buckets()) { diff --git a/src/probes/exec.js b/src/probes/exec.js index c020d69..9fe0480 100644 --- a/src/probes/exec.js +++ b/src/probes/exec.js @@ -2,7 +2,7 @@ // ring buffer once, feeds the pure aggregation in lib/model.js, and exposes // what the UI reads. The only BPF-aware module besides probe.js. import { signal } from "yeet:tui"; -import { DataSec, HashMap, RingBuf } from "yeet:bpf"; +import { ArrayMap, DataSec, HashMap, RingBuf } from "yeet:bpf"; import { control } from "./probe.js"; import { normalize } from "../lib/argv.js"; import { createModel, foldKey } from "../lib/model.js"; @@ -10,6 +10,18 @@ import { createModel, foldKey } from "../lib/model.js"; const events = new RingBuf(control, "events"); const traced = new HashMap(control, "traced"); const bss = new DataSec(control, "probe.bss"); +const statsMap = new ArrayMap(control, "stats_map"); + +// How many execs the kernel emitted vs dropped. A drop means the ring buffer +// filled faster than userspace drained it, so the numbers on screen are a +// floor rather than a count. Polled rather than streamed: it only needs to be +// right when it is read. +// A SIGNAL, not a plain variable. The verdict line reads this inside a thunk, +// and a thunk only re-renders when a signal it read changes — a plain +// function returning a mutated local never triggers a repaint, so the warning +// was computed correctly and never drawn. +export const droppedCount = signal(0); +export const dropped = () => droppedCount.get(); const model = createModel(); @@ -56,10 +68,11 @@ export async function seedExisting(rows) { return n; } -// Narrow to a cgroup (0 = the pid subtree alone, no cgroup filter). -export async function setCgroup(cgid) { - await bss.patch({ target_cgid: cgid }); -} +// Note: there is deliberately no cgroup narrowing. An earlier design set a +// target_cgid in .bss and filtered on it in-kernel, but nothing ever called it, +// so container mode was always a plain pid subtree. Now that the existing tree +// is seeded from the process graph, the tgid set IS the scope, and a second +// overlapping filter would only add a way for the two to disagree. // One subscription, driving one model. // @@ -105,6 +118,15 @@ await events.subscribe((w) => { // per event would spend the budget on work the eye cannot see. The flash decay // also needs a heartbeat, so tick advances even when the stream is briefly // quiet but flashes are still fading. +// Poll the kernel counters once a second. Cheap, and only used to warn. +setInterval(async () => { + try { + const c = await statsMap.lookup(0); + const n = Number(c?.dropped ?? 0); + if (n !== droppedCount.get()) droppedCount.set(n); + } catch { /* counters are advisory; never break the UI over them */ } +}, 1000); + setInterval(() => { const fading = flashes.size > 0; // Tick at least once a second even with nothing arriving, so the idle line diff --git a/src/probes/probe.js b/src/probes/probe.js index ba91d84..b974043 100644 --- a/src/probes/probe.js +++ b/src/probes/probe.js @@ -18,6 +18,7 @@ async function load() { .bind("events", { kind: "ringbuf", btf_struct: "exec_event" }) .bind("traced", { kind: "hash" }) .bind("fork_ts", { kind: "hash" }) + .bind("stats_map", { kind: "array" }) .bind("probe.bss", { kind: "data" }) .start(); } catch (err) { last = err; } diff --git a/test/drops.sh b/test/drops.sh new file mode 100755 index 0000000..a1a69da --- /dev/null +++ b/test/drops.sh @@ -0,0 +1,81 @@ +#!/usr/bin/env bash +# Ring-buffer drop accounting. +# +# Delivery is bound by bytes moved, not events: a 1 KiB record caps the ring at +# roughly 2,700 execs/s. A parallel fork storm outruns that by an order of +# magnitude, and the events that do not fit are lost. That is a real limit and +# it is fine; losing them SILENTLY is not, because every count on screen then +# reads as a total when it is a floor. +# +# This asserts the accounting closes: captured + dropped == what actually ran. +# Run it in the VM after `make`. +set -uo pipefail +cd "$(dirname "$0")/.." + +WORKERS=${WORKERS:-24} +PER=${PER:-4000} +EXPECT=$((WORKERS * PER)) + +g="\033[32m"; r="\033[31m"; d="\033[2m"; z="\033[0m" +fail=0 +ok() { printf "${g} ok %s${z}\n" "$1"; } +bad() { printf "${r} FAIL %s${z}\n" "$1"; fail=$((fail+1)); } + +cat > /tmp/exectop-storm.sh </dev/null 2>&1 &1) +CAPTURED=$(printf '%s' "$OUT" | grep -oE '^=== [0-9]+ execs' | grep -oE '[0-9]+' | head -1) +DROPPED=$(printf '%s' "$OUT" | grep -oE '— [0-9]+ DROPPED' | grep -oE '[0-9]+' | head -1) +DROPPED=${DROPPED:-0} +CAPTURED=${CAPTURED:-0} +TOTAL=$((CAPTURED + DROPPED)) + +echo +echo "captured=$CAPTURED dropped=$DROPPED sum=$TOTAL expected>=$EXPECT" +echo + +# The point of the test: nothing vanishes unaccounted for. Shell overhead adds +# a little (seq, sleep, the subshells), so the sum is a floor, and the slack is +# for scheduling, not for losses. +if [ "$TOTAL" -ge "$EXPECT" ]; then + ok "accounting closes: captured + dropped >= $EXPECT" +else + bad "accounting leaks: $TOTAL < $EXPECT — $((EXPECT - TOTAL)) execs vanished uncounted" +fi + +# A storm this size must actually overrun the ring. If it does not, either the +# machine got much faster or the storm is no longer a storm, and the drop path +# is then untested rather than passing. +if [ "$DROPPED" -gt 0 ]; then + ok "drops are reported, not silent ($DROPPED)" +else + bad "no drops at $EXPECT execs — the drop path went untested; raise WORKERS/PER" +fi + +# And the warning has to reach the reader. +if printf '%s' "$OUT" | grep -q "were dropped"; then + ok "the report says the counts are a floor" +else + bad "drops counted but never explained in the output" +fi + +echo +if [ "$fail" -eq 0 ]; then + printf "${g}PASS — every exec is either captured or counted as dropped${z}\n" +else + printf "${r}FAIL — %d check(s) failed${z}\n" "$fail" +fi +exit "$fail"