diff --git a/.agents/skills/bpf-tutorial-writing-style/SKILL.md b/.agents/skills/bpf-tutorial-writing-style/SKILL.md index d51a2d1b..71cc61fe 100644 --- a/.agents/skills/bpf-tutorial-writing-style/SKILL.md +++ b/.agents/skills/bpf-tutorial-writing-style/SKILL.md @@ -1,6 +1,6 @@ --- name: bpf-tutorial-writing-style -description: Style checklist for English and Chinese bpf-developer-tutorial README files. Use while drafting or reviewing a tutorial so it reads like the established series, preserves technical detail, and presents complete source clearly. +description: Style checklist for English and Chinese bpf-developer-tutorial README files. Use while drafting or reviewing a tutorial so it reads like the established series, preserves technical detail, and presents complete kernel source clearly. --- # BPF Tutorial Writing Style @@ -32,8 +32,9 @@ For advanced tutorials (40+), follow this exact section order: 5. **Code Implementation**: - First introduce the overall processing logic - - Show complete source code for each component (header, BPF program, user-space loader) - - After each complete code block, explain the key parts with paragraph style (not bullet lists) + - Show complete kernel/BPF source and the core headers needed to understand it + - Include the complete user-space loader when it is concise; when it is long, use focused excerpts that cover the normal user workflow and link to the lesson directory for the full file + - After each code block, explain the key parts with paragraph style (not bullet lists) - Focus on logic and advanced features, not basic syntax 6. **Additional Concepts** (if needed): Deeper explanation of specific features, edge cases, or semantics @@ -87,8 +88,9 @@ Use this Chinese paragraph only as a voice reference for connected rhythm, restr ## Teach from complete source -- Introduce one component, show its complete source in an ordinary Markdown fence, then explain the important logic before moving to the next component. -- Keep source byte-exact. Preserve comments and commands; use focused excerpts only after the complete block. +- Introduce one component, show its complete kernel/BPF source in an ordinary Markdown fence, then explain the important logic before moving to the next component. +- Show core headers completely. Include a complete user-space file when it remains readable; otherwise use focused excerpts that preserve the public control flow without making the tutorial a source dump. +- Keep every complete source block byte-exact. Preserve comments and commands in excerpts, and link once to the lesson directory for omitted user-space details. - Use neither `
` nor HTML synchronization markers. - Link to the complete lesson once through its GitHub directory. Avoid an opening catalog of individual files. - Every Markdown link uses an absolute `https://github.com/...` target. Omit a link when no stable GitHub target exists. diff --git a/.agents/skills/find-bpf-tutorial-topic/SKILL.md b/.agents/skills/find-bpf-tutorial-topic/SKILL.md index 48df5dfc..e54c5137 100644 --- a/.agents/skills/find-bpf-tutorial-topic/SKILL.md +++ b/.agents/skills/find-bpf-tutorial-topic/SKILL.md @@ -1,6 +1,6 @@ --- name: find-bpf-tutorial-topic -description: Audit bpf-developer-tutorial coverage, research current Linux BPF work and real open-source eBPF projects, maintain the repository tutorial candidate registry, and rank the next lesson by reader value, eBPF leverage, reproducibility, distinctness, teaching clarity, ecosystem evidence, maturity, and maintenance cost. Use when deciding what eBPF tutorial to write next, checking whether an idea is already covered, comparing feature-driven and scenario-driven topics, refreshing the topic roadmap, or asking which candidate has the highest value. +description: Audit bpf-developer-tutorial coverage, research current Linux BPF work and real open-source eBPF projects, maintain the repository tutorial candidate registry, and rank the next lesson by reader value, operational usefulness, eBPF leverage, reproducibility, distinctness, teaching clarity, ecosystem evidence, maturity, and maintenance cost. Use when deciding what eBPF tutorial to write next, checking whether an idea is already covered or practically usable, comparing feature-driven and scenario-driven topics, refreshing the topic roadmap, or asking which candidate has the highest value. --- # Find the Next BPF Tutorial Topic @@ -43,26 +43,42 @@ Convert each project feature into a small independent lesson. Teach the mechanis ## Define the runnable lesson before scoring -For every serious candidate, write five short fields: +For every serious candidate, write seven short fields: - the concrete question a reader can reproduce; - the event, packet, request, task, or workload followed end to end; - the BPF hook, map, helper, kfunc, or program type that makes the answer possible; +- the exact command a reader would realistically run and the decision or action its output enables; +- the natural lifecycle: single-pass action, bounded diagnostic, or persistent monitor, including how it stops and cleans up; - the deterministic fixture and observable success output; - the closest existing lesson and the new knowledge this candidate adds. Defer an idea when these fields remain vague. A helper name alone is not a tutorial topic. +## Apply the practicality gate before scoring + +Walk through the proposed public workflow as an operator, separately from the deterministic test harness: + +- A naturally one-shot operation, such as an iterator scan or atomic state change, may run once and exit. An observability or enforcement monitor must attach before independent workloads, remain useful for its natural lifetime, and stop cleanly by signal, duration, or an explicit completion condition. +- The fixture may be artificial, but the public command must not exist only to launch `/bin/true` or another toy trigger. A launch-scoped command is acceptable only when launch scoping is itself the lesson's real use case. +- Require an explicit readiness signal before an independent workload starts; fixed sleeps do not prove that attachment or setup finished. +- Define target scope and filters, concurrent-event or admission behavior, failure and drop visibility, cleanup after normal exit and signals, and what a reader does with the result. +- For asynchronous work, define shutdown as: stop admission, wait for completed work rather than merely started callbacks, drain output, report stable health, then destroy resources. +- Reject a design whose output merely proves that a helper ran. The output must answer the stated operational question or drive the stated action. + +Redesign or defer a candidate when the fixture is its only plausible user, when the lifecycle contradicts the scenario, or when safe cleanup and bounded resource behavior cannot be explained and tested. + ## Score value Score from evidence, then subtract costs. Keep the breakdown in working notes and place the final score in `TUTORIAL_IDEAS.md`. | Dimension | Maximum | Question | |---|---:|---| -| Reader problem | 25 | Does it answer a recurring, consequential question? | -| eBPF leverage | 20 | Does eBPF provide visibility or control that ordinary tools cannot provide as cleanly? | +| Reader problem | 20 | Does it answer a recurring, consequential question? | +| Operational usefulness | 15 | Is the public command and lifecycle useful outside the test fixture? | +| eBPF leverage | 15 | Does eBPF provide visibility or control that ordinary tools cannot provide as cleanly? | | Coverage gap | 15 | Does it add a distinct mechanism or scenario to this repository? | -| Reproducibility | 15 | Can one bounded local or KVM fixture prove the result? | +| Reproducibility | 10 | Can one bounded local or KVM fixture prove the result? | | Teaching clarity | 10 | Can the lesson follow one understandable path and show decisive output? | | Ecosystem evidence | 10 | Do maintained projects or upstream tests validate the use case? | | Maturity | 5 | Are the required kernel and userspace interfaces stable enough to maintain? | @@ -76,6 +92,8 @@ Newness breaks close ties; it does not replace reader value. Favor a stable less A candidate can become `ready` only when: - it has a concrete reproducible problem and expected output; +- its documented invocation and lifecycle match how a reader would use the tool outside the fixture; +- its target scope, concurrency or admission bound, failure visibility, and cleanup path are explicit; - eBPF is central to the solution; - the repository does not already teach the same flow; - the example can fit one coherent tutorial; diff --git a/.agents/skills/find-bpf-tutorial-topic/agents/openai.yaml b/.agents/skills/find-bpf-tutorial-topic/agents/openai.yaml index 245d5367..61caccb0 100644 --- a/.agents/skills/find-bpf-tutorial-topic/agents/openai.yaml +++ b/.agents/skills/find-bpf-tutorial-topic/agents/openai.yaml @@ -1,4 +1,4 @@ interface: display_name: "Find BPF Tutorial Topic" - short_description: "Rank the next high-value eBPF tutorial topic" - default_prompt: "Use $find-bpf-tutorial-topic to audit coverage and recommend the next eBPF tutorial." + short_description: "Rank practical, high-value eBPF tutorial topics" + default_prompt: "Use $find-bpf-tutorial-topic to audit coverage, practical usability, and recommend the next eBPF tutorial." diff --git a/.agents/skills/write-bpf-production-tutorial/SKILL.md b/.agents/skills/write-bpf-production-tutorial/SKILL.md index df86af6d..8e212337 100644 --- a/.agents/skills/write-bpf-production-tutorial/SKILL.md +++ b/.agents/skills/write-bpf-production-tutorial/SKILL.md @@ -1,6 +1,6 @@ --- name: write-bpf-production-tutorial -description: Write or rewrite one bilingual bpf-developer-tutorial lesson with pinned Claude Opus 4.5, then verify the result against the implementation. Use when creating README.md and README.zh.md, preserving full source, testing the example, or preparing tutorial changes for review. +description: Design, write, or rewrite one practical bilingual bpf-developer-tutorial lesson with pinned Claude Opus 4.5, then verify the public workflow and prose against the implementation. Use when creating README.md and README.zh.md, checking whether a tutorial tool has a realistic CLI and lifecycle, preserving complete kernel source, testing the example, or preparing tutorial changes for review. --- # Write a BPF Tutorial @@ -17,18 +17,35 @@ Read these inputs before writing: - the current README pair when revising an existing lesson; - primary upstream sources for versions and feature semantics. -Collect the facts the reader needs: the problem, why traditional approaches fail, the kernel/user-space flow, feature versions, requirements, commands, real output, cleanup, limits, and references. Keep every claim grounded in the code, tests, captured output, or a primary source. Preserve an existing draft before a from-scratch rewrite. +Collect the facts the reader needs: the problem, why traditional approaches fail, the kernel/user-space flow, feature versions, requirements, intended public commands, real output, concurrency or admission behavior, cleanup, limits, and references. Keep every claim grounded in the code, tests, captured output, or a primary source. Preserve an existing draft before a from-scratch rewrite. Build and run the example when the environment supports it. Use `$test-bpf-tutorial-kvm` for kernel features that need the repository's KVM environment. Runtime details support the tutorial; local workspace paths, VM names, shared repositories, caches, prompts, and agent traces stay private. -## 2. Structure requirements +## 2. Pass the practical-design gate + +Validate the tool as an operator before writing prose. Keep its public workflow separate from the deterministic fixture: + +- State the operational question, the exact command a reader would run, the independent workload or target, the useful output, and how the tool stops. +- Classify the lifecycle. A scan or atomic control action may be one-shot; a tracer or monitor must attach before the workload and remain active until a signal, duration, or real completion condition. +- Do not turn a blocked toy child or `/bin/true` fixture into the public CLI merely because it removes a test race. Launch-scoped tracing is valid only when it is the intended real workflow. +- Emit and test an explicit readiness signal before starting an independent workload. Do not use a fixed sleep as proof that setup or attachment completed. +- Check target scope and filters, concurrent state, admission bounds, drop and failure counters, exit status, normal cleanup, signal cleanup, and destructive-action safety in proportion to the lesson. +- For asynchronous work, stop admission first, wait for completed work rather than merely entered callbacks, drain output, report stable health, and only then destroy resources. +- Execute the documented command as written. The test must prove the real lifecycle plus one relevant failure or cleanup path, not only that the BPF program loaded. +- Describe the current public workflow directly. Do not narrate removed flags, old child-command modes, or other migration history unless backward compatibility is itself the lesson. +- Keep repository tests out of the reader-facing tutorial path. Use them as private validation evidence, but teach the normal command, independent workload, useful tool output, and shutdown sequence. +- Show only output emitted by the documented tool in public examples. Never include harness lines such as `TEST-*` or `PASS`, fixture setup, test assertions, or local test-infrastructure provenance. + +If the implementation only demonstrates a helper but is awkward or misleading as a tool, revise the implementation and test before asking the writer to explain it. Do not let polished prose overclaim an impractical design. + +## 3. Structure requirements For advanced tutorials (40+), follow this exact section order as defined in `$bpf-tutorial-writing-style`: 1. **Title + Introduction**: Concrete problem scenario, link to source 2. **Background / Why This Approach**: Explain traditional approaches and their limitations, then what eBPF enables 3. **High-Level Mechanism**: How the feature works before showing code -4. **Code Implementation**: Complete source blocks, then paragraph explanations +4. **Code Implementation**: Complete kernel/BPF source and core headers, plus a complete user-space loader when concise or focused user-space excerpts when long, then paragraph explanations 5. **Additional Concepts** (if needed) 6. **Compilation and Execution**: AFTER code analysis, not before 7. **Summary + Call to Action** @@ -39,7 +56,7 @@ The "Background / Why" section is critical. It must explain: - Why each doesn't work well - What the eBPF approach enables -## 3. Let Opus write +## 4. Let Opus write Claude Opus writes all reader-facing tutorial prose. Use the exact model ID `claude-opus-4-5-20251101`. Stop if that model is unavailable instead of substituting another model. @@ -55,22 +72,24 @@ Do not paste a second checklist, paragraph plan, fact inventory, or review rubri Run Claude from the repository root with the pinned model and permission to read the repository and write the two README files. Opus does not commit or push. -## 4. Check the result +## 5. Check the result Inspect both files and the diff instead of trusting the model's final message. Confirm that: - both languages are complete and tell the same technical story; +- the documented command, target, lifecycle, signals, concurrency bounds, cleanup, and health output match the practical design gate; - the opening reads like a tutorial rather than an abstract or feature list; - there is a "Why" section explaining traditional approaches and their limitations; - the high-level mechanism is explained BEFORE code sections; - compilation/execution is AFTER code analysis; -- every core source file appears once in a complete ordinary Markdown fence; +- every kernel/BPF source and core header appears once in a complete ordinary Markdown fence; a concise user-space loader should also be complete, while a long loader may use focused excerpts that cover the normal public control flow; - code, commands, output, versions, requirements, cleanup, and limits agree with the repository; +- sample output comes from the normal public workflow and contains no test-harness or migration/deprecation narration; - the opening source link points only to the lesson directory; - every published link is an absolute `https://github.com/...` URL; - no local infrastructure, prompt, model, agent, or trace detail appears in public text. -Check source blocks with the bundled `sync-source-blocks.py`, then run `git diff --check`, the lesson build, its tests, and the relevant runtime test. Treat a functional run as a functional run rather than a benchmark. +Check the required complete-source inventory with the bundled `sync-source-blocks.py`: always include kernel/BPF sources and core headers, and include user-space files only when the README presents them as complete. Then run `git diff --check`, the lesson build, its tests, the exact documented public command, and the relevant runtime test. Treat a functional run as a functional run rather than a benchmark. Read the finished pair once as an intermediate eBPF developer. If a concrete problem remains, give Opus a short defect list in the same session and let it revise the whole pair before returning. Keep prompts, responses, partial drafts, and failed runs; never delete real conversation or agent history. diff --git a/.agents/skills/write-bpf-production-tutorial/agents/openai.yaml b/.agents/skills/write-bpf-production-tutorial/agents/openai.yaml index 00427e27..ad3f247f 100644 --- a/.agents/skills/write-bpf-production-tutorial/agents/openai.yaml +++ b/.agents/skills/write-bpf-production-tutorial/agents/openai.yaml @@ -1,4 +1,4 @@ interface: display_name: "BPF Tutorial Writer" - short_description: "Write and test clear bilingual eBPF lessons" - default_prompt: "Use $write-bpf-production-tutorial to turn this eBPF example into a clear, tested bilingual tutorial." + short_description: "Design and test practical bilingual eBPF lessons" + default_prompt: "Use $write-bpf-production-tutorial to turn this eBPF example into a practical, tested bilingual tutorial." diff --git a/src/50-tcx/Makefile b/src/50-tcx/Makefile index c0b4c4b4..6c3ca89b 100644 --- a/src/50-tcx/Makefile +++ b/src/50-tcx/Makefile @@ -44,9 +44,16 @@ endef $(call allow-override,CC,$(CROSS_COMPILE)cc) $(call allow-override,LD,$(CROSS_COMPILE)ld) -.PHONY: all +.PHONY: all test all: $(APPS) +test: all + @if [ "$$(id -u)" -ne 0 ]; then \ + echo "test requires root; run it inside the tutorial KVM" >&2; \ + exit 1; \ + fi + python3 tests/test_tcx_demo.py ./tcx_demo + .PHONY: clean clean: $(call msg,CLEAN) diff --git a/src/50-tcx/README.md b/src/50-tcx/README.md index 3edc5d0b..960ee3a8 100644 --- a/src/50-tcx/README.md +++ b/src/50-tcx/README.md @@ -74,7 +74,7 @@ __u32 last_ifindex; SEC("tcx/ingress") int tcx_stats(struct __sk_buff *skb) { - stats_hits++; + __sync_fetch_and_add(&stats_hits, 1); last_len = skb->len; last_protocol = bpf_ntohs(skb->protocol); last_ifindex = skb->ifindex; @@ -84,7 +84,7 @@ int tcx_stats(struct __sk_buff *skb) SEC("tcx/ingress") int tcx_classifier(struct __sk_buff *skb) { - classifier_hits++; + __sync_fetch_and_add(&classifier_hits, 1); return TCX_PASS; } ``` @@ -107,7 +107,7 @@ For packet data access, you would use `bpf_skb_load_bytes()` or direct packet po ### Global Variables as Counters -We use global variables (`stats_hits`, `classifier_hits`, etc.) rather than BPF maps. The libbpf skeleton exposes these at `skel->bss->stats_hits`, simplifying user-space access. This approach works for single-CPU demonstrations; production code should use per-CPU arrays to avoid concurrent update races. +We use global variables (`stats_hits`, `classifier_hits`, etc.) rather than explicit BPF maps. The libbpf skeleton exposes these at `skel->bss->stats_hits`, simplifying user-space access. The hit counters use atomic additions because packets can arrive on multiple CPUs; the last-packet metadata remains a best-effort snapshot. ### Return Codes: `TCX_NEXT` vs `TCX_PASS` @@ -166,18 +166,25 @@ After attachment, the loader queries the kernel for the live chain state. The re This introspection capability is valuable for debugging multi-program pipelines and for tools that need to understand the current attachment state. -### Step 4: Generate Traffic and Verify +### Step 4: Stay Attached and Observe Traffic -The loader sends a UDP packet to `127.0.0.1:9` (the discard service) to trigger the chain, waits briefly, then reads the global variables: +After printing `READY` and the live chain, the loader stays attached. Generate traffic independently on the monitored interface, then press Ctrl-C (or use `--duration`) to stop. The loader detaches both links before reading a stable final snapshot: ```c -printf(" tcx_stats hits : %llu\n", - (unsigned long long)skel->bss->stats_hits); -printf(" tcx_classifier hits : %llu\n", - (unsigned long long)skel->bss->classifier_hits); +bpf_link__destroy(stats_link); +stats_link = NULL; +bpf_link__destroy(classifier_link); +classifier_link = NULL; + +printf("COUNTERS stats_hits=%llu classifier_hits=%llu last_ifindex=%u " + "last_protocol=0x%04x last_len=%u\n", + (unsigned long long)skel->bss->stats_hits, + (unsigned long long)skel->bss->classifier_hits, + skel->bss->last_ifindex, skel->bss->last_protocol, + skel->bss->last_len); ``` -If both counters show 1, the chain executed as intended: `tcx_stats` ran first (recording metadata, returning `TCX_NEXT`), then `tcx_classifier` ran (counting the hit, returning `TCX_PASS`). +Matching hit counts show that `tcx_stats` ran first (recording metadata and returning `TCX_NEXT`) and `tcx_classifier` then ran and returned `TCX_PASS`. Keeping the process alive also demonstrates TCX ownership: closing the link file descriptors on exit automatically detaches the programs. ## Building and Running @@ -198,37 +205,32 @@ make ### Run ```bash -sudo ./tcx_demo -i lo +sudo ./tcx_demo ``` -Expected output: +The default monitors loopback until Ctrl-C. In another terminal, generate ordinary loopback traffic, for example `ping -c 1 127.0.0.1`. The monitor first prints readiness and chain order, then prints counters when it exits: ```text -Attached TCX programs to lo (ifindex=1) +READY interface=lo ifindex=1 duration=until-signal TCX ingress chain revision: 2 slot 0: prog_id=812 link_id=901 slot 1: prog_id=811 link_id=900 - -Counters: - tcx_stats hits : 1 - tcx_classifier hits : 1 - last ifindex : 1 - last protocol : 0x0800 - last length : 46 +^C +COUNTERS stats_hits=... classifier_hits=... last_ifindex=1 last_protocol=0x0800 last_len=... ``` **Reading the output:** - **Revision 2**: The chain was modified twice: once when `tcx_classifier` attached (revision 1), and once when `tcx_stats` was inserted before it (revision 2). - **Slot ordering**: Slot 0 is `tcx_stats` (the program we inserted with `BPF_F_BEFORE`); slot 1 is `tcx_classifier`. -- **Protocol 0x0800**: IPv4 (the generated UDP packet). -- **Length 46**: The 20-byte payload "tcx tutorial packet" plus headers. +- **Protocol 0x0800**: IPv4 traffic traversed the ingress chain. +- **Matching hit counts**: Both programs observed the same packets in the intended order. ### Options -- `-i IFACE`: Attach to a different interface (default: `lo`) -- `-n`: Skip traffic generation (useful for examining attach/query behavior only) -- `-v`: Enable libbpf debug output to see the underlying BPF syscall sequence +- `-i, --interface IFACE`: Attach to a different interface (default: `lo`) +- `-d, --duration SEC`: Stop after a bounded interval; `0` waits for a signal (default: `0`) +- `-v, --verbose`: Enable libbpf debug output to see the underlying BPF syscall sequence ## Comparison with Lesson 20 (Classic TC) diff --git a/src/50-tcx/README.zh.md b/src/50-tcx/README.zh.md index 026d31b4..c9fb8bf9 100644 --- a/src/50-tcx/README.zh.md +++ b/src/50-tcx/README.zh.md @@ -74,7 +74,7 @@ __u32 last_ifindex; SEC("tcx/ingress") int tcx_stats(struct __sk_buff *skb) { - stats_hits++; + __sync_fetch_and_add(&stats_hits, 1); last_len = skb->len; last_protocol = bpf_ntohs(skb->protocol); last_ifindex = skb->ifindex; @@ -84,7 +84,7 @@ int tcx_stats(struct __sk_buff *skb) SEC("tcx/ingress") int tcx_classifier(struct __sk_buff *skb) { - classifier_hits++; + __sync_fetch_and_add(&classifier_hits, 1); return TCX_PASS; } ``` @@ -107,7 +107,7 @@ int tcx_classifier(struct __sk_buff *skb) ### 全局变量作为计数器 -我们使用全局变量（`stats_hits`、`classifier_hits` 等）而非 BPF map。libbpf skeleton 在 `skel->bss->stats_hits` 暴露这些变量，简化了用户态访问。这种方式适用于单 CPU 演示；生产代码应使用 per-CPU 数组来避免并发更新竞争。 +我们使用全局变量（`stats_hits`、`classifier_hits` 等）而非显式 BPF map。libbpf skeleton 在 `skel->bss->stats_hits` 暴露这些变量，简化了用户态访问。数据包可能在多个 CPU 上到达，因此命中计数使用原子加法；最后一个数据包的元数据仍是尽力而为的快照。 ### 返回码：`TCX_NEXT` 与 `TCX_PASS` @@ -166,18 +166,25 @@ err = bpf_prog_query_opts(ifindex, BPF_TCX_INGRESS, &query); 这种内省能力对调试多程序流水线很有价值，也适用于需要了解当前挂载状态的工具。 -### 第四步：发送流量并验证 +### 第四步：保持挂载并观察流量 -加载器向 `127.0.0.1:9`（discard 服务）发送一个 UDP 包来触发链，短暂等待后读取全局变量： +加载器打印 `READY` 和实时链后会保持挂载。请在另一个进程中向目标接口发送流量，然后按 Ctrl-C（或使用 `--duration`）停止。加载器先 detach 两个 link，再读取稳定的最终快照： ```c -printf(" tcx_stats hits : %llu\n", - (unsigned long long)skel->bss->stats_hits); -printf(" tcx_classifier hits : %llu\n", - (unsigned long long)skel->bss->classifier_hits); +bpf_link__destroy(stats_link); +stats_link = NULL; +bpf_link__destroy(classifier_link); +classifier_link = NULL; + +printf("COUNTERS stats_hits=%llu classifier_hits=%llu last_ifindex=%u " + "last_protocol=0x%04x last_len=%u\n", + (unsigned long long)skel->bss->stats_hits, + (unsigned long long)skel->bss->classifier_hits, + skel->bss->last_ifindex, skel->bss->last_protocol, + skel->bss->last_len); ``` -如果两个计数器都显示 1，说明链按预期执行了：`tcx_stats` 先运行（记录元数据，返回 `TCX_NEXT`），然后 `tcx_classifier` 运行（计数，返回 `TCX_PASS`）。 +相同的命中数说明 `tcx_stats` 先运行（记录元数据并返回 `TCX_NEXT`），随后 `tcx_classifier` 运行并返回 `TCX_PASS`。保持进程运行也能体现 TCX 所有权：退出时关闭 link 文件描述符，程序便会自动卸载。 ## 编译和运行 @@ -198,37 +205,32 @@ make ### 运行 ```bash -sudo ./tcx_demo -i lo +sudo ./tcx_demo ``` -预期输出： +默认会持续监控 loopback，直到按 Ctrl-C。可在另一个终端生成普通 loopback 流量，例如 `ping -c 1 127.0.0.1`。监控器先打印就绪状态和链顺序，退出时再打印计数： ```text -Attached TCX programs to lo (ifindex=1) +READY interface=lo ifindex=1 duration=until-signal TCX ingress chain revision: 2 slot 0: prog_id=812 link_id=901 slot 1: prog_id=811 link_id=900 - -Counters: - tcx_stats hits : 1 - tcx_classifier hits : 1 - last ifindex : 1 - last protocol : 0x0800 - last length : 46 +^C +COUNTERS stats_hits=... classifier_hits=... last_ifindex=1 last_protocol=0x0800 last_len=... ``` **输出解读：** - **Revision 2**：链被修改了两次，`tcx_classifier` 挂载时（revision 1），`tcx_stats` 插入到它前面时（revision 2）。 - **Slot 顺序**：Slot 0 是 `tcx_stats`（用 `BPF_F_BEFORE` 插入的程序）；slot 1 是 `tcx_classifier`。 -- **Protocol 0x0800**：IPv4（生成的 UDP 包）。 -- **Length 46**：20 字节的载荷 "tcx tutorial packet" 加上报头。 +- **Protocol 0x0800**：IPv4 流量经过了 ingress 链。 +- **相同命中数**：两个程序按预期顺序观察到了相同的数据包。 ### 选项 -- `-i IFACE`：挂载到不同的接口（默认：`lo`） -- `-n`：跳过流量生成（只查看挂载/查询行为） -- `-v`：启用 libbpf 调试输出，查看底层 BPF syscall 序列 +- `-i, --interface IFACE`：挂载到不同的接口（默认：`lo`） +- `-d, --duration SEC`：在限定时间后停止；`0` 表示等待信号（默认：`0`） +- `-v, --verbose`：启用 libbpf 调试输出，查看底层 BPF syscall 序列 ## 与第 20 课（经典 TC）的对比 diff --git a/src/50-tcx/tcx_demo.bpf.c b/src/50-tcx/tcx_demo.bpf.c index a4425f47..f84207b5 100644 --- a/src/50-tcx/tcx_demo.bpf.c +++ b/src/50-tcx/tcx_demo.bpf.c @@ -22,7 +22,7 @@ __u32 last_ifindex; SEC("tcx/ingress") int tcx_stats(struct __sk_buff *skb) { - stats_hits++; + __sync_fetch_and_add(&stats_hits, 1); last_len = skb->len; last_protocol = bpf_ntohs(skb->protocol); last_ifindex = skb->ifindex; @@ -32,6 +32,6 @@ int tcx_stats(struct __sk_buff *skb) SEC("tcx/ingress") int tcx_classifier(struct __sk_buff *skb) { - classifier_hits++; + __sync_fetch_and_add(&classifier_hits, 1); return TCX_PASS; } diff --git a/src/50-tcx/tcx_demo.c b/src/50-tcx/tcx_demo.c index b9ca980b..67160e50 100644 --- a/src/50-tcx/tcx_demo.c +++ b/src/50-tcx/tcx_demo.c @@ -1,12 +1,13 @@ // SPDX-License-Identifier: GPL-2.0 -#include #include +#include #include +#include #include #include #include #include -#include +#include #include #include #include @@ -14,12 +15,20 @@ static struct env { const char *ifname; + unsigned int duration; bool verbose; - bool no_trigger; } env = { .ifname = "lo", }; +static volatile sig_atomic_t exiting; + +static void handle_signal(int signal) +{ + (void)signal; + exiting = 1; +} + static int libbpf_print_fn(enum libbpf_print_level level, const char *format, va_list args) { if (level == LIBBPF_DEBUG && !env.verbose) @@ -30,57 +39,63 @@ static int libbpf_print_fn(enum libbpf_print_level level, const char *format, va static void usage(const char *prog) { fprintf(stderr, - "Usage: %s [-i IFACE] [-v] [-n]\n" - " -i IFACE attach TCX programs to interface (default: lo)\n" - " -v enable libbpf debug logs\n" - " -n do not generate loopback traffic automatically\n", + "Usage: %s [--interface IFACE] [--duration SEC] [--verbose]\n\n" + "Monitor a TCX ingress chain until Ctrl-C.\n\n" + "Options:\n" + " -i, --interface IFACE interface to monitor (default: lo)\n" + " -d, --duration SEC stop after SEC; 0 waits for a signal (default: 0)\n" + " -v, --verbose enable libbpf debug logs\n" + " -h, --help show this help\n", prog); } +static int parse_duration(const char *value) +{ + char *end = NULL; + unsigned long parsed; + + errno = 0; + parsed = strtoul(value, &end, 10); + if (errno || end == value || *end || parsed > 86400) { + fprintf(stderr, "invalid duration in seconds: %s\n", value); + return -EINVAL; + } + env.duration = parsed; + return 0; +} + static int parse_args(int argc, char **argv) { + static const struct option options[] = { + { "interface", required_argument, NULL, 'i' }, + { "duration", required_argument, NULL, 'd' }, + { "verbose", no_argument, NULL, 'v' }, + { "help", no_argument, NULL, 'h' }, + {}, + }; int opt; - while ((opt = getopt(argc, argv, "i:vn")) != -1) { + while ((opt = getopt_long(argc, argv, "i:d:vh", options, NULL)) != -1) { switch (opt) { case 'i': env.ifname = optarg; break; + case 'd': + if (parse_duration(optarg)) + return -EINVAL; + break; case 'v': env.verbose = true; break; - case 'n': - env.no_trigger = true; - break; + case 'h': + usage(argv[0]); + exit(0); default: return -EINVAL; } } - return 0; -} - -static int generate_loopback_traffic(void) -{ - struct sockaddr_in addr = { - .sin_family = AF_INET, - .sin_port = htons(9), - }; - const char payload[] = "tcx tutorial packet"; - int fd, err = 0; - - if (inet_pton(AF_INET, "127.0.0.1", &addr.sin_addr) != 1) - return -EINVAL; - - fd = socket(AF_INET, SOCK_DGRAM, 0); - if (fd < 0) - return -errno; - - if (sendto(fd, payload, sizeof(payload), 0, (struct sockaddr *)&addr, sizeof(addr)) < 0) - err = -errno; - - close(fd); - return err; + return optind == argc ? 0 : -EINVAL; } static void print_tcx_chain(int ifindex) @@ -109,6 +124,43 @@ static void print_tcx_chain(int ifindex) } } +static long long monotonic_milliseconds(void) +{ + struct timespec timestamp; + + if (clock_gettime(CLOCK_MONOTONIC, ×tamp)) + return -errno; + return timestamp.tv_sec * 1000LL + timestamp.tv_nsec / 1000000; +} + +static int wait_until_done(void) +{ + struct timespec interval = { .tv_nsec = 100000000 }; + long long deadline = 0; + long long now; + + if (env.duration) { + now = monotonic_milliseconds(); + if (now < 0) + return (int)now; + deadline = now + env.duration * 1000LL; + } + + while (!exiting) { + if (deadline) { + now = monotonic_milliseconds(); + if (now < 0) + return (int)now; + if (now >= deadline) + break; + } + if (nanosleep(&interval, NULL) && errno != EINTR) + return -errno; + } + + return 0; +} + int main(int argc, char **argv) { struct tcx_demo_bpf *skel = NULL; @@ -128,6 +180,8 @@ int main(int argc, char **argv) } libbpf_set_print(libbpf_print_fn); + signal(SIGINT, handle_signal); + signal(SIGTERM, handle_signal); skel = tcx_demo_bpf__open_and_load(); if (!skel) { @@ -161,28 +215,23 @@ int main(int argc, char **argv) } } - printf("Attached TCX programs to %s (ifindex=%d)\n", env.ifname, ifindex); + printf("READY interface=%s ifindex=%d duration=%s\n", env.ifname, ifindex, + env.duration ? "limited" : "until-signal"); + fflush(stdout); print_tcx_chain(ifindex); - if (!env.no_trigger && strcmp(env.ifname, "lo") == 0) { - err = generate_loopback_traffic(); - if (err) - fprintf(stderr, "failed to generate loopback traffic: %s\n", - strerror(-err)); - usleep(200000); - } else if (!env.no_trigger) { - printf("Generate traffic on %s and re-run with -n if you only want attach/query.\n", - env.ifname); - } - - printf("\nCounters:\n"); - printf(" tcx_stats hits : %llu\n", - (unsigned long long)skel->bss->stats_hits); - printf(" tcx_classifier hits : %llu\n", - (unsigned long long)skel->bss->classifier_hits); - printf(" last ifindex : %u\n", skel->bss->last_ifindex); - printf(" last protocol : 0x%04x\n", skel->bss->last_protocol); - printf(" last length : %u\n", skel->bss->last_len); + err = wait_until_done(); + bpf_link__destroy(stats_link); + stats_link = NULL; + bpf_link__destroy(classifier_link); + classifier_link = NULL; + + printf("COUNTERS stats_hits=%llu classifier_hits=%llu last_ifindex=%u " + "last_protocol=0x%04x last_len=%u\n", + (unsigned long long)skel->bss->stats_hits, + (unsigned long long)skel->bss->classifier_hits, + skel->bss->last_ifindex, skel->bss->last_protocol, + skel->bss->last_len); cleanup: bpf_link__destroy(stats_link); diff --git a/src/50-tcx/tests/test_tcx_demo.py b/src/50-tcx/tests/test_tcx_demo.py new file mode 100644 index 00000000..e7dfd436 --- /dev/null +++ b/src/50-tcx/tests/test_tcx_demo.py @@ -0,0 +1,115 @@ +#!/usr/bin/env python3 +"""End-to-end lifecycle test for the persistent TCX monitor.""" + +from __future__ import annotations + +import re +import select +import signal +import socket +import subprocess +import sys +import time + + +COUNTERS = re.compile( + r"COUNTERS stats_hits=(\d+) classifier_hits=(\d+) " + r"last_ifindex=(\d+) last_protocol=0x([0-9a-fA-F]+) last_len=(\d+)" +) + + +def wait_ready(process: subprocess.Popen[str]) -> list[str]: + lines: list[str] = [] + deadline = time.monotonic() + 8 + assert process.stdout is not None + + while time.monotonic() < deadline: + readable, _, _ = select.select([process.stdout], [], [], 0.25) + if readable: + line = process.stdout.readline() + if line: + lines.append(line) + if line.startswith("READY "): + return lines + if process.poll() is not None: + break + + process.kill() + remainder, _ = process.communicate() + raise AssertionError("TCX monitor did not become ready:\n" + "".join(lines) + remainder) + + +def generate_loopback_traffic() -> None: + receiver = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) + sender = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) + try: + receiver.bind(("127.0.0.1", 0)) + receiver.settimeout(1) + payload = b"tcx monitor lifecycle" + sender.sendto(payload, receiver.getsockname()) + received, _ = receiver.recvfrom(128) + assert received == payload + finally: + sender.close() + receiver.close() + + +def run_and_stop(binary: str, stop_signal: int) -> str: + process = subprocess.Popen( + [binary, "--duration", "30"], + text=True, + stdout=subprocess.PIPE, + stderr=subprocess.STDOUT, + bufsize=1, + ) + lines = wait_ready(process) + generate_loopback_traffic() + process.send_signal(stop_signal) + remainder, _ = process.communicate(timeout=5) + output = "".join(lines) + remainder + assert process.returncode == 0, output + return output + + +def main() -> int: + binary = sys.argv[1] if len(sys.argv) > 1 else "./tcx_demo" + + invalid = subprocess.run( + [binary, "--duration", "invalid"], + text=True, + stdout=subprocess.PIPE, + stderr=subprocess.STDOUT, + check=False, + ) + assert invalid.returncode != 0 + assert "invalid duration" in invalid.stdout + + first = run_and_stop(binary, signal.SIGTERM) + match = COUNTERS.search(first) + assert match, first + stats_hits, classifier_hits, ifindex, protocol, length = ( + int(value, 16) if index == 3 else int(value) + for index, value in enumerate(match.groups()) + ) + assert stats_hits > 0, first + assert classifier_hits == stats_hits, first + assert ifindex > 0, first + assert protocol == 0x0800, first + assert length > 0, first + assert len(re.findall(r"^ slot ", first, re.MULTILINE)) == 2, first + + # A clean second chain has exactly two slots, so neither first-run link leaked. + second = run_and_stop(binary, signal.SIGINT) + assert COUNTERS.search(second), second + assert len(re.findall(r"^ slot ", second, re.MULTILINE)) == 2, second + + ready = next(line for line in first.splitlines() if line.startswith("READY ")) + counters = next(line for line in first.splitlines() if line.startswith("COUNTERS ")) + print(ready) + print(counters) + print("PASS: READY lifecycle, external traffic, signal shutdown, and reattach succeeded") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/src/51-tcp-quarantine/Makefile b/src/51-tcp-quarantine/Makefile index 89a8863b..c6399a79 100644 --- a/src/51-tcp-quarantine/Makefile +++ b/src/51-tcp-quarantine/Makefile @@ -51,6 +51,10 @@ clean: $(Q)rm -rf $(OUTPUT) $(APPS) test: all + @if [ "$$(id -u)" -ne 0 ]; then \ + echo "test requires root; run it inside the tutorial KVM" >&2; \ + exit 1; \ + fi python3 tests/test_tcp_quarantine.py ./tcp_quarantine $(OUTPUT) $(OUTPUT)/libbpf $(BPFTOOL_OUTPUT): diff --git a/src/51-tcp-quarantine/README.md b/src/51-tcp-quarantine/README.md index b1ac11da..bd10035a 100644 --- a/src/51-tcp-quarantine/README.md +++ b/src/51-tcp-quarantine/README.md @@ -14,7 +14,7 @@ This seemingly simple requirement is surprisingly difficult to achieve with trad What we need is a kernel-level mechanism that can identify and destroy specific sockets atomically, with no race window. This is exactly what Linux 6.5 introduced with the `bpf_sock_destroy` kernel function (kfunc). -This tutorial builds a command-line tool that uses a BPF iterator to walk the kernel's TCP socket table, find established connections matching a specific IPv4 address and port, and destroy them on demand. You will learn how BPF iterators provide safe, locked traversal of kernel data structures, and how kfuncs like `bpf_sock_destroy` expose kernel operations to BPF programs. +This tutorial builds a command-line tool that uses a BPF iterator to walk the kernel's TCP socket table. A dry run lists every established connection matching a remote IPv4 address and port; apply mode additionally requires the selected local IPv4 address and port, then destroys only that complete 4-tuple. You will learn how BPF iterators provide safe, locked traversal of kernel data structures, and how kfuncs like `bpf_sock_destroy` expose kernel operations to BPF programs. > Complete source code: @@ -40,14 +40,14 @@ The `bpf_sock_destroy` kfunc, introduced in Linux 6.5 (commit `4ddbcb886268af8d1 The `tcp_quarantine` tool combines these two features into a practical workflow: -1. User space specifies a target IPv4 address and port, plus whether to actually destroy matches or just count them (dry-run mode). +1. User space specifies a remote IPv4 address and port. By default this is a dry run; apply mode also requires one local IPv4 address and port copied from the dry-run output. 2. The BPF program is loaded with these parameters baked into its read-only data section. 3. User space opens an iterator file descriptor and reads from it, triggering the kernel to invoke the BPF callback for each TCP socket. -4. For each socket, the BPF callback checks: Is this an IPv4 socket? Is it in the ESTABLISHED state? Does its destination match the target? -5. Matching sockets are counted. In apply mode, `bpf_sock_destroy` is called to tear them down. +4. For each socket, the BPF callback checks: Is this an IPv4 socket? Is it in the ESTABLISHED state? Does its remote endpoint match? In apply mode, does its local endpoint also match? +5. Matching sockets are printed and counted. Only an apply scan with all four endpoint fields equal calls `bpf_sock_destroy`. 6. After traversal completes, user space reads statistics from the BPF program's BSS section and reports results. -The entire matching-and-destroy sequence happens inside the kernel, with no window where socket state could change between inspection and action. +The final match and destroy happen in one iterator callback on the same socket. Dry-run and apply are separate scans, so a connection can disappear between them; in that case apply safely reports `matched=0`. ## Code Walkthrough @@ -73,7 +73,7 @@ struct quarantine_stats { #endif /* __TCP_QUARANTINE_H */ ``` -These five counters track: total sockets examined, how many were IPv4 and ESTABLISHED, how many matched the target destination, how many were successfully destroyed (apply mode only), and how many `bpf_sock_destroy` calls failed. +These five counters track: total sockets examined, how many were IPv4 and ESTABLISHED, how many matched the active selector, how many were successfully destroyed (apply mode only), and how many `bpf_sock_destroy` calls failed. ### BPF Iterator Program @@ -94,6 +94,8 @@ char LICENSE[] SEC("license") = "GPL"; const volatile __u32 target_addr; const volatile __u16 target_port; +const volatile __u32 target_local_addr; +const volatile __u16 target_local_port; const volatile bool apply; struct quarantine_stats stats; @@ -104,8 +106,11 @@ SEC("iter/tcp") int quarantine_tcp(struct bpf_iter__tcp *ctx) { struct sock_common *sk = ctx->sk_common; - __u32 dst_addr; + struct seq_file *seq = ctx->meta->seq; + __u32 dst_addr, local_addr; + __u32 dst_host, local_host; __u16 dst_port; + __u16 local_port; __u16 family; __u8 state; int err; @@ -124,8 +129,21 @@ int quarantine_tcp(struct bpf_iter__tcp *ctx) dst_port = BPF_CORE_READ(sk, skc_dport); if (dst_addr != target_addr || dst_port != bpf_htons(target_port)) return 0; + local_addr = BPF_CORE_READ(sk, skc_rcv_saddr); + local_port = BPF_CORE_READ(sk, skc_num); + if (apply && (local_addr != target_local_addr || + local_port != target_local_port)) + return 0; stats.matched++; + local_host = bpf_ntohl(local_addr); + dst_host = bpf_ntohl(dst_addr); + BPF_SEQ_PRINTF(seq, + "MATCH local=%u.%u.%u.%u:%u remote=%u.%u.%u.%u:%u\n", + local_host >> 24, (local_host >> 16) & 0xff, + (local_host >> 8) & 0xff, local_host & 0xff, local_port, + dst_host >> 24, (dst_host >> 16) & 0xff, + (dst_host >> 8) & 0xff, dst_host & 0xff, target_port); if (!apply) return 0; @@ -141,15 +159,15 @@ int quarantine_tcp(struct bpf_iter__tcp *ctx) Let's break down the key elements: -**Configuration variables**: The three `const volatile` variables (`target_addr`, `target_port`, `apply`) live in the BPF program's read-only data section (`.rodata`). User space writes to them after opening the skeleton but before loading the program. The `const volatile` pattern tells the BPF verifier these are compile-time constants while still allowing user space to set them. This enables the verifier to optimize code paths; for example, when `apply` is false, the entire destruction branch can be eliminated. +**Configuration variables**: The five `const volatile` variables hold the remote endpoint, optional local endpoint, and apply mode in the BPF program's read-only data section (`.rodata`). User space writes them after opening the skeleton but before loading the program. The verifier can eliminate the local-match and destruction path when `apply` is false. **Kfunc declaration**: The line `extern int bpf_sock_destroy(struct sock_common *sock) __ksym;` declares `bpf_sock_destroy` as an external kernel symbol. The `__ksym` annotation tells the loader to resolve this at load time by looking up the kernel function. **CO-RE field access**: `BPF_CORE_READ(sk, skc_family)` reads the `skc_family` field from the socket structure using BTF (BPF Type Format) information. This is part of CO-RE (Compile Once, Run Everywhere): the compiled BPF program contains relocation records that the loader patches based on the running kernel's BTF data. A program compiled on one kernel version will work on others even if structure field offsets differ. -**Byte order handling**: The destination address is stored in network byte order (big-endian), which matches what `inet_pton` produces, so we compare directly. The port is also stored in network byte order, so we convert our host-byte-order target port with `bpf_htons()` before comparing. +**Byte order handling**: Remote and local addresses match the network byte order produced by `inet_pton`. The remote `skc_dport` is in network byte order and is compared with `bpf_htons(target_port)`; local `skc_num` is already in host byte order. -**Three-stage filtering**: The callback applies progressively tighter filters. First it counts all sockets (`scanned`), then narrows to IPv4 ESTABLISHED sockets (`established`), then to those matching the exact destination (`matched`). This funnel structure lets you see at a glance whether the tool is scanning the right population. +**Progressive filtering**: The callback first counts all sockets (`scanned`), then IPv4 ESTABLISHED sockets (`established`), then remote matches. Dry-run prints every candidate as `MATCH local=... remote=...`. Apply mode rechecks the selected local endpoint before incrementing `matched` or destroying anything. There is deliberately no option that destroys every remote match. ### User-Space Loader @@ -171,12 +189,18 @@ Let's break down the key elements: #include "tcp_quarantine.skel.h" static struct env { - const char *destination; - unsigned int port; + const char *remote_argument; + const char *local_argument; bool apply; bool verbose; } env; +struct endpoint { + struct in_addr address; + unsigned int port; + char text[INET_ADDRSTRLEN + 7]; +}; + static int libbpf_print_fn(enum libbpf_print_level level, const char *format, va_list args) { @@ -188,15 +212,14 @@ static int libbpf_print_fn(enum libbpf_print_level level, const char *format, static void usage(const char *program) { fprintf(stderr, - "Usage: %s --destination IPv4 --port PORT [--apply] [--verbose]\n" + "Usage: %s [--apply LOCAL_IPV4:PORT] REMOTE_IPV4:PORT [--verbose]\n" "\n" - "Find established TCP client connections to an exact destination.\n" - "The default is a safe dry run; --apply destroys matching sockets.\n" + "List established TCP clients to REMOTE_IPV4:PORT.\n" + "The default is a dry run. Copy one listed local endpoint into\n" + "--apply to destroy only that exact IPv4 4-tuple.\n" "\n" "Options:\n" - " -d, --destination IPv4 exact remote IPv4 address\n" - " -p, --port PORT exact remote TCP port (1-65535)\n" - " -a, --apply destroy matching sockets\n" + " -a, --apply IPv4:PORT local endpoint selected from dry-run output\n" " -v, --verbose print libbpf diagnostics\n" " -h, --help show this help\n", program); @@ -215,31 +238,42 @@ static int parse_port(const char *value, unsigned int *port) return 0; } +static int parse_endpoint(const char *value, struct endpoint *endpoint) +{ + char address[INET_ADDRSTRLEN]; + const char *separator = strrchr(value, ':'); + size_t address_length; + + if (!separator || separator == value) + return -EINVAL; + address_length = separator - value; + if (address_length >= sizeof(address)) + return -EINVAL; + memcpy(address, value, address_length); + address[address_length] = '\0'; + if (inet_pton(AF_INET, address, &endpoint->address) != 1 || + parse_port(separator + 1, &endpoint->port)) + return -EINVAL; + snprintf(endpoint->text, sizeof(endpoint->text), "%s:%u", + address, endpoint->port); + return 0; +} + static int parse_args(int argc, char **argv) { static const struct option options[] = { - { "destination", required_argument, NULL, 'd' }, - { "port", required_argument, NULL, 'p' }, - { "apply", no_argument, NULL, 'a' }, + { "apply", required_argument, NULL, 'a' }, { "verbose", no_argument, NULL, 'v' }, { "help", no_argument, NULL, 'h' }, {}, }; int option; - while ((option = getopt_long(argc, argv, "d:p:avh", options, NULL)) != -1) { + while ((option = getopt_long(argc, argv, "a:vh", options, NULL)) != -1) { switch (option) { - case 'd': - env.destination = optarg; - break; - case 'p': - if (parse_port(optarg, &env.port)) { - fprintf(stderr, "invalid TCP port: %s\n", optarg); - return -EINVAL; - } - break; case 'a': env.apply = true; + env.local_argument = optarg; break; case 'v': env.verbose = true; @@ -252,15 +286,16 @@ static int parse_args(int argc, char **argv) } } - if (!env.destination || !env.port || optind != argc) + if (argc - optind != 1) return -EINVAL; + env.remote_argument = argv[optind]; return 0; } static int run_iterator(struct bpf_program *program) { struct bpf_link *link; - char buffer[256]; + char buffer[4096]; int iter_fd, length, err; link = bpf_program__attach_iter(program, NULL); @@ -277,8 +312,13 @@ static int run_iterator(struct bpf_program *program) goto cleanup; } - while ((length = read(iter_fd, buffer, sizeof(buffer))) > 0) - ; + while ((length = read(iter_fd, buffer, sizeof(buffer))) > 0) { + if (fwrite(buffer, 1, length, stdout) != (size_t)length) { + err = -EIO; + fprintf(stderr, "failed to print iterator output\n"); + break; + } + } if (length < 0) { err = -errno; fprintf(stderr, "failed while scanning TCP sockets: %s\n", strerror(errno)); @@ -293,7 +333,8 @@ cleanup: int main(int argc, char **argv) { struct tcp_quarantine_bpf *skel = NULL; - struct in_addr destination; + struct endpoint remote = {}; + struct endpoint local = {}; int err; err = parse_args(argc, argv); @@ -301,8 +342,14 @@ int main(int argc, char **argv) usage(argv[0]); return 1; } - if (inet_pton(AF_INET, env.destination, &destination) != 1) { - fprintf(stderr, "invalid IPv4 destination: %s\n", env.destination); + if (parse_endpoint(env.remote_argument, &remote)) { + fprintf(stderr, "invalid remote IPv4 endpoint: %s\n", + env.remote_argument); + return 1; + } + if (env.apply && parse_endpoint(env.local_argument, &local)) { + fprintf(stderr, "invalid local IPv4 endpoint: %s\n", + env.local_argument); return 1; } @@ -313,8 +360,10 @@ int main(int argc, char **argv) return 1; } - skel->rodata->target_addr = destination.s_addr; - skel->rodata->target_port = env.port; + skel->rodata->target_addr = remote.address.s_addr; + skel->rodata->target_port = remote.port; + skel->rodata->target_local_addr = local.address.s_addr; + skel->rodata->target_local_port = local.port; skel->rodata->apply = env.apply; err = tcp_quarantine_bpf__load(skel); @@ -331,9 +380,10 @@ int main(int argc, char **argv) if (err) goto cleanup; - printf("mode=%s destination=%s:%u scanned=%llu established=%llu " + printf("SUMMARY mode=%s remote=%s%s%s scanned=%llu established=%llu " "matched=%llu destroyed=%llu failed=%llu\n", - env.apply ? "apply" : "dry-run", env.destination, env.port, + env.apply ? "apply" : "dry-run", remote.text, + env.apply ? " local=" : "", env.apply ? local.text : "", skel->bss->stats.scanned, skel->bss->stats.established, skel->bss->stats.matched, skel->bss->stats.destroyed, skel->bss->stats.failed); @@ -350,7 +400,7 @@ The key workflow: 1. **Open the skeleton**: `tcp_quarantine_bpf__open()` parses the embedded BPF object but doesn't load it yet. -2. **Configure parameters**: We write to `skel->rodata->*` to set the target address, port, and apply flag. This must happen before `load()`. +2. **Configure parameters**: We write the parsed remote endpoint, optional local endpoint, and apply flag to `skel->rodata->*` before `load()`. 3. **Load the program**: `tcp_quarantine_bpf__load()` loads the BPF program into the kernel. The verifier runs, BTF relocations are applied, and kfuncs are resolved. @@ -358,7 +408,7 @@ The key workflow: 5. **Read results**: After iteration, we read statistics from `skel->bss->stats`. The BSS section is automatically memory-mapped, so these reads just access shared memory. -The iterator's `read()` loop discards the buffer contents because this tool doesn't produce output through the iterator interface; it only uses the traversal to drive the BPF callback. Other iterator programs might write data to the iterator (e.g., formatted socket information), but ours communicates exclusively through the BSS statistics. +The iterator callback writes each candidate with `BPF_SEQ_PRINTF`. The `read()` loop copies that iterator output to stdout, making the dry run actionable; aggregate results still come from BSS statistics. ## Network Namespace Boundary @@ -368,7 +418,7 @@ If you need to close a connection inside a container or another namespace, first ```bash sudo nsenter --net=/proc//ns/net -- ./tcp_quarantine \ - --destination 203.0.113.99 --port 443 --apply + 203.0.113.99:443 ``` Here `` is the PID of any process in the target network namespace. The `--net` flag changes only the network namespace while keeping the mount namespace intact, so the tool binary remains accessible. @@ -381,7 +431,7 @@ When `bpf_sock_destroy` tears down a socket, the kernel executes the protocol's - Writes may produce `EPIPE` with a `SIGPIPE` signal. - The exact behavior varies by kernel version and the timing of operations. -The test included with this tutorial accepts any of these as valid indicators that the connection was torn down. +Treat any of these outcomes as application-visible evidence that the connection was torn down; confirm the exact behavior expected by your application and kernel version. ## Building and Running @@ -396,45 +446,39 @@ make -j2 **Dry-run mode** (the default) scans for matches but doesn't destroy anything: ```bash -sudo ./tcp_quarantine --destination 127.0.0.1 --port 42063 +sudo ./tcp_quarantine 127.0.0.1:42063 ``` -**Apply mode** destroys matching sockets: +Copy one `MATCH` line's local endpoint into apply mode. This destroys only that complete 4-tuple: ```bash -sudo ./tcp_quarantine --destination 127.0.0.1 --port 42063 --apply +sudo ./tcp_quarantine --apply 127.0.0.1:55490 127.0.0.1:42063 ``` Command-line reference: ```text -Usage: ./tcp_quarantine --destination IPv4 --port PORT [--apply] [--verbose] +Usage: ./tcp_quarantine [--apply LOCAL_IPV4:PORT] REMOTE_IPV4:PORT [--verbose] Options: - -d, --destination IPv4 exact remote IPv4 address - -p, --port PORT exact remote TCP port (1-65535) - -a, --apply destroy matching sockets + -a, --apply IPv4:PORT local endpoint selected from dry-run output -v, --verbose print libbpf diagnostics -h, --help show this help ``` ### Example Session -The following output was captured on x86_64 running kernel `7.0.0-rc2+`. The automated test creates two loopback TCP connections: a target (which we'll destroy) and a control (which should survive). It verifies that traffic flows on both connections before running the tool: +Suppose a service has two established connections to the same remote listener. A normal dry-run lists both candidates; copying one local endpoint into the apply command selects only that connection: ```console -mode=dry-run destination=127.0.0.1:42063 scanned=6 established=4 matched=1 destroyed=0 failed=0 -mode=apply destination=127.0.0.1:42063 scanned=6 established=4 matched=1 destroyed=1 failed=0 -PASS: dry-run preserved both connections; apply destroyed only the target +MATCH local=127.0.0.1:55490 remote=127.0.0.1:42063 +MATCH local=127.0.0.1:55494 remote=127.0.0.1:42063 +SUMMARY mode=dry-run remote=127.0.0.1:42063 scanned=8 established=6 matched=2 destroyed=0 failed=0 +MATCH local=127.0.0.1:55490 remote=127.0.0.1:42063 +SUMMARY mode=apply remote=127.0.0.1:42063 local=127.0.0.1:55490 scanned=7 established=5 matched=1 destroyed=1 failed=0 ``` -The socket counts and port numbers vary between runs. What matters is: exactly one socket matched, dry-run left it intact, apply destroyed it successfully, and the control connection continued working. - -Run the automated test: - -```bash -sudo make test -``` +Socket counts and ports vary. What matters is that dry-run lists both remote matches without changing them, while apply reports `matched=1 destroyed=1` for the copied 4-tuple and leaves the other socket intact. ### Requirements @@ -451,14 +495,14 @@ If loading fails, the tool prints both the kernel error and a checklist of prere ## Summary -This tutorial demonstrated how to use BPF iterators and the `bpf_sock_destroy` kfunc to surgically terminate specific TCP connections. Unlike process killing, firewall rules, or user-space RST injection, this approach: +This tutorial demonstrated how to use BPF iterators and the `bpf_sock_destroy` kfunc to surgically terminate one selected TCP connection. Unlike process killing, firewall rules, or user-space RST injection, this approach: - Works atomically with no race window between inspection and action - Does not require guessing TCP sequence numbers - Does not disrupt unrelated connections on the same process - Is not affected by encryption or unusual network configurations -The tool as written handles single-pass traversal with exact matching on one IPv4 destination. Possible extensions include IPv6 support, wildcard matching, continuous monitoring mode, process/cgroup attribution, multi-namespace orchestration, and integration with threat-intelligence feeds. +The tool uses the complete IPv4 4-tuple as its smallest safe destructive selector. Possible extensions include IPv6 support, process/cgroup attribution, multi-namespace orchestration, and integration with threat-intelligence feeds; broad wildcard destruction is intentionally outside this tutorial. > To learn more about eBPF, visit our tutorial repository at or our website at . diff --git a/src/51-tcp-quarantine/README.zh.md b/src/51-tcp-quarantine/README.zh.md index 21bef47e..7d82377b 100644 --- a/src/51-tcp-quarantine/README.zh.md +++ b/src/51-tcp-quarantine/README.zh.md @@ -12,7 +12,7 @@ 我们需要的是一种内核级机制，能够原子地识别并销毁特定套接字，不存在竞态窗口。这正是 Linux 6.5 引入 `bpf_sock_destroy` 内核函数（kfunc）所提供的能力。 -本教程将构建一个命令行工具，使用 BPF 迭代器遍历内核的 TCP 套接字表，找出与指定 IPv4 地址和端口匹配的已建立连接，并按需销毁。你将学习 BPF 迭代器如何在持有锁的情况下安全遍历内核数据结构，以及 `bpf_sock_destroy` 这样的 kfunc 如何将内核操作暴露给 BPF 程序。 +本教程将构建一个命令行工具，使用 BPF 迭代器遍历内核的 TCP 套接字表。dry-run 列出所有匹配远端 IPv4 地址和端口的已建立连接；apply 模式还必须给出选中的本地 IPv4 地址和端口，因此只会销毁完整 4-tuple 对应的连接。你将学习 BPF 迭代器如何在持有锁的情况下安全遍历内核数据结构，以及 `bpf_sock_destroy` 这样的 kfunc 如何将内核操作暴露给 BPF 程序。 > 完整源码： @@ -38,14 +38,14 @@ Kfunc（内核函数）是 BPF 程序直接调用特定内核函数的一种机 `tcp_quarantine` 工具将这两个特性组合成一个实用的工作流程： -1. 用户态指定目标 IPv4 地址和端口，以及是真正销毁匹配项还是仅计数（dry-run 模式）。 +1. 用户态指定远端 IPv4 地址和端口。默认是 dry-run；apply 模式还要提供从 dry-run 输出复制的一组本地 IPv4 地址和端口。 2. BPF 程序加载时，这些参数被写入其只读数据段。 3. 用户态打开迭代器文件描述符并从中读取，触发内核为每个 TCP 套接字调用 BPF 回调。 -4. 对于每个套接字，BPF 回调检查：这是 IPv4 套接字吗？处于 ESTABLISHED 状态吗？目标地址匹配吗？ -5. 匹配的套接字被计数。在 apply 模式下，调用 `bpf_sock_destroy` 拆除它们。 +4. 对于每个套接字，BPF 回调检查：这是 IPv4 套接字吗？处于 ESTABLISHED 状态吗？远端端点匹配吗？apply 模式下，本地端点也匹配吗？ +5. 匹配的套接字会被打印并计数。只有 apply 扫描中四个端点字段全部相等时才调用 `bpf_sock_destroy`。 6. 遍历完成后，用户态从 BPF 程序的 BSS 段读取统计数据并报告结果。 -整个匹配和销毁过程都在内核中进行，检查和操作之间不存在套接字状态变化的窗口。 +最终匹配和销毁在同一个迭代器回调中针对同一个 socket 完成。dry-run 与 apply 是两次扫描，连接可能在两者之间消失；这时 apply 会安全地报告 `matched=0`。 ## 代码详解 @@ -71,7 +71,7 @@ struct quarantine_stats { #endif /* __TCP_QUARANTINE_H */ ``` -五个计数器分别记录：检查的套接字总数、IPv4 且 ESTABLISHED 状态的数量、与目标地址匹配的数量、成功销毁的数量（仅 apply 模式）、`bpf_sock_destroy` 调用失败的数量。 +五个计数器分别记录：检查的套接字总数、IPv4 且 ESTABLISHED 状态的数量、与当前选择器匹配的数量、成功销毁的数量（仅 apply 模式）、`bpf_sock_destroy` 调用失败的数量。 ### BPF 迭代器程序 @@ -92,6 +92,8 @@ char LICENSE[] SEC("license") = "GPL"; const volatile __u32 target_addr; const volatile __u16 target_port; +const volatile __u32 target_local_addr; +const volatile __u16 target_local_port; const volatile bool apply; struct quarantine_stats stats; @@ -102,8 +104,11 @@ SEC("iter/tcp") int quarantine_tcp(struct bpf_iter__tcp *ctx) { struct sock_common *sk = ctx->sk_common; - __u32 dst_addr; + struct seq_file *seq = ctx->meta->seq; + __u32 dst_addr, local_addr; + __u32 dst_host, local_host; __u16 dst_port; + __u16 local_port; __u16 family; __u8 state; int err; @@ -122,8 +127,21 @@ int quarantine_tcp(struct bpf_iter__tcp *ctx) dst_port = BPF_CORE_READ(sk, skc_dport); if (dst_addr != target_addr || dst_port != bpf_htons(target_port)) return 0; + local_addr = BPF_CORE_READ(sk, skc_rcv_saddr); + local_port = BPF_CORE_READ(sk, skc_num); + if (apply && (local_addr != target_local_addr || + local_port != target_local_port)) + return 0; stats.matched++; + local_host = bpf_ntohl(local_addr); + dst_host = bpf_ntohl(dst_addr); + BPF_SEQ_PRINTF(seq, + "MATCH local=%u.%u.%u.%u:%u remote=%u.%u.%u.%u:%u\n", + local_host >> 24, (local_host >> 16) & 0xff, + (local_host >> 8) & 0xff, local_host & 0xff, local_port, + dst_host >> 24, (dst_host >> 16) & 0xff, + (dst_host >> 8) & 0xff, dst_host & 0xff, target_port); if (!apply) return 0; @@ -139,15 +157,15 @@ int quarantine_tcp(struct bpf_iter__tcp *ctx) 关键要素解读： -**配置变量**：三个 `const volatile` 变量（`target_addr`、`target_port`、`apply`）位于 BPF 程序的只读数据段（`.rodata`）。用户态在打开 skeleton 之后、加载程序之前写入它们。`const volatile` 模式告诉 BPF 验证器这些是编译期常量，同时仍允许用户态设置。这使验证器能够优化代码路径，例如当 `apply` 为 false 时，整个销毁分支可以被消除。 +**配置变量**：五个 `const volatile` 变量在 BPF 程序的只读数据段（`.rodata`）中保存远端端点、可选本地端点和 apply 模式。用户态在打开 skeleton 之后、加载程序之前写入它们；当 `apply` 为 false 时，验证器可以消除本地匹配和销毁路径。 **Kfunc 声明**：`extern int bpf_sock_destroy(struct sock_common *sock) __ksym;` 这行将 `bpf_sock_destroy` 声明为外部内核符号。`__ksym` 注解告诉加载器在加载时通过查找内核函数来解析它。 **CO-RE 字段访问**：`BPF_CORE_READ(sk, skc_family)` 使用 BTF（BPF Type Format）信息读取套接字结构的 `skc_family` 字段。这是 CO-RE（Compile Once, Run Everywhere，一次编译到处运行）的一部分：编译后的 BPF 程序包含重定位记录，加载器会根据运行中内核的 BTF 数据进行修补。在一个内核版本上编译的程序可以在其他版本上运行，即使结构体字段偏移不同。 -**字节序处理**：目标地址以网络字节序（大端）存储，与 `inet_pton` 产生的结果一致，所以直接比较。端口也以网络字节序存储，所以比较前用 `bpf_htons()` 把主机字节序的目标端口转换一下。 +**字节序处理**：远端和本地地址与 `inet_pton` 产生的网络字节序一致。远端 `skc_dport` 是网络字节序，要与 `bpf_htons(target_port)` 比较；本地 `skc_num` 已经是主机字节序。 -**三级过滤**：回调采用逐级收紧的过滤器。首先计数所有套接字（`scanned`），然后筛选出 IPv4 且 ESTABLISHED 状态的（`established`），最后筛选与目标地址精确匹配的（`matched`）。这种漏斗结构让你一眼就能看出工具是否在扫描正确的群体。 +**逐级过滤**：回调先计数所有 socket（`scanned`），再筛选 IPv4 ESTABLISHED socket（`established`），最后匹配远端。dry-run 用 `MATCH local=... remote=...` 打印每个候选；apply 模式在增加 `matched` 或执行销毁前还会重新检查选中的本地端点。工具故意不提供销毁所有远端匹配项的选项。 ### 用户态加载器 @@ -169,12 +187,18 @@ int quarantine_tcp(struct bpf_iter__tcp *ctx) #include "tcp_quarantine.skel.h" static struct env { - const char *destination; - unsigned int port; + const char *remote_argument; + const char *local_argument; bool apply; bool verbose; } env; +struct endpoint { + struct in_addr address; + unsigned int port; + char text[INET_ADDRSTRLEN + 7]; +}; + static int libbpf_print_fn(enum libbpf_print_level level, const char *format, va_list args) { @@ -186,15 +210,14 @@ static int libbpf_print_fn(enum libbpf_print_level level, const char *format, static void usage(const char *program) { fprintf(stderr, - "Usage: %s --destination IPv4 --port PORT [--apply] [--verbose]\n" + "Usage: %s [--apply LOCAL_IPV4:PORT] REMOTE_IPV4:PORT [--verbose]\n" "\n" - "Find established TCP client connections to an exact destination.\n" - "The default is a safe dry run; --apply destroys matching sockets.\n" + "List established TCP clients to REMOTE_IPV4:PORT.\n" + "The default is a dry run. Copy one listed local endpoint into\n" + "--apply to destroy only that exact IPv4 4-tuple.\n" "\n" "Options:\n" - " -d, --destination IPv4 exact remote IPv4 address\n" - " -p, --port PORT exact remote TCP port (1-65535)\n" - " -a, --apply destroy matching sockets\n" + " -a, --apply IPv4:PORT local endpoint selected from dry-run output\n" " -v, --verbose print libbpf diagnostics\n" " -h, --help show this help\n", program); @@ -213,31 +236,42 @@ static int parse_port(const char *value, unsigned int *port) return 0; } +static int parse_endpoint(const char *value, struct endpoint *endpoint) +{ + char address[INET_ADDRSTRLEN]; + const char *separator = strrchr(value, ':'); + size_t address_length; + + if (!separator || separator == value) + return -EINVAL; + address_length = separator - value; + if (address_length >= sizeof(address)) + return -EINVAL; + memcpy(address, value, address_length); + address[address_length] = '\0'; + if (inet_pton(AF_INET, address, &endpoint->address) != 1 || + parse_port(separator + 1, &endpoint->port)) + return -EINVAL; + snprintf(endpoint->text, sizeof(endpoint->text), "%s:%u", + address, endpoint->port); + return 0; +} + static int parse_args(int argc, char **argv) { static const struct option options[] = { - { "destination", required_argument, NULL, 'd' }, - { "port", required_argument, NULL, 'p' }, - { "apply", no_argument, NULL, 'a' }, + { "apply", required_argument, NULL, 'a' }, { "verbose", no_argument, NULL, 'v' }, { "help", no_argument, NULL, 'h' }, {}, }; int option; - while ((option = getopt_long(argc, argv, "d:p:avh", options, NULL)) != -1) { + while ((option = getopt_long(argc, argv, "a:vh", options, NULL)) != -1) { switch (option) { - case 'd': - env.destination = optarg; - break; - case 'p': - if (parse_port(optarg, &env.port)) { - fprintf(stderr, "invalid TCP port: %s\n", optarg); - return -EINVAL; - } - break; case 'a': env.apply = true; + env.local_argument = optarg; break; case 'v': env.verbose = true; @@ -250,15 +284,16 @@ static int parse_args(int argc, char **argv) } } - if (!env.destination || !env.port || optind != argc) + if (argc - optind != 1) return -EINVAL; + env.remote_argument = argv[optind]; return 0; } static int run_iterator(struct bpf_program *program) { struct bpf_link *link; - char buffer[256]; + char buffer[4096]; int iter_fd, length, err; link = bpf_program__attach_iter(program, NULL); @@ -275,8 +310,13 @@ static int run_iterator(struct bpf_program *program) goto cleanup; } - while ((length = read(iter_fd, buffer, sizeof(buffer))) > 0) - ; + while ((length = read(iter_fd, buffer, sizeof(buffer))) > 0) { + if (fwrite(buffer, 1, length, stdout) != (size_t)length) { + err = -EIO; + fprintf(stderr, "failed to print iterator output\n"); + break; + } + } if (length < 0) { err = -errno; fprintf(stderr, "failed while scanning TCP sockets: %s\n", strerror(errno)); @@ -291,7 +331,8 @@ cleanup: int main(int argc, char **argv) { struct tcp_quarantine_bpf *skel = NULL; - struct in_addr destination; + struct endpoint remote = {}; + struct endpoint local = {}; int err; err = parse_args(argc, argv); @@ -299,8 +340,14 @@ int main(int argc, char **argv) usage(argv[0]); return 1; } - if (inet_pton(AF_INET, env.destination, &destination) != 1) { - fprintf(stderr, "invalid IPv4 destination: %s\n", env.destination); + if (parse_endpoint(env.remote_argument, &remote)) { + fprintf(stderr, "invalid remote IPv4 endpoint: %s\n", + env.remote_argument); + return 1; + } + if (env.apply && parse_endpoint(env.local_argument, &local)) { + fprintf(stderr, "invalid local IPv4 endpoint: %s\n", + env.local_argument); return 1; } @@ -311,8 +358,10 @@ int main(int argc, char **argv) return 1; } - skel->rodata->target_addr = destination.s_addr; - skel->rodata->target_port = env.port; + skel->rodata->target_addr = remote.address.s_addr; + skel->rodata->target_port = remote.port; + skel->rodata->target_local_addr = local.address.s_addr; + skel->rodata->target_local_port = local.port; skel->rodata->apply = env.apply; err = tcp_quarantine_bpf__load(skel); @@ -329,9 +378,10 @@ int main(int argc, char **argv) if (err) goto cleanup; - printf("mode=%s destination=%s:%u scanned=%llu established=%llu " + printf("SUMMARY mode=%s remote=%s%s%s scanned=%llu established=%llu " "matched=%llu destroyed=%llu failed=%llu\n", - env.apply ? "apply" : "dry-run", env.destination, env.port, + env.apply ? "apply" : "dry-run", remote.text, + env.apply ? " local=" : "", env.apply ? local.text : "", skel->bss->stats.scanned, skel->bss->stats.established, skel->bss->stats.matched, skel->bss->stats.destroyed, skel->bss->stats.failed); @@ -348,7 +398,7 @@ cleanup: 1. **打开 skeleton**：`tcp_quarantine_bpf__open()` 解析嵌入的 BPF 对象但不加载。 -2. **配置参数**：我们写入 `skel->rodata->*` 来设置目标地址、端口和 apply 标志。这必须在 `load()` 之前完成。 +2. **配置参数**：在 `load()` 前把解析后的远端端点、可选本地端点和 apply 标志写入 `skel->rodata->*`。 3. **加载程序**：`tcp_quarantine_bpf__load()` 将 BPF 程序加载到内核。验证器运行，BTF 重定位被应用，kfunc 被解析。 @@ -356,7 +406,7 @@ cleanup: 5. **读取结果**：迭代完成后，我们从 `skel->bss->stats` 读取统计数据。BSS 段是自动内存映射的，所以这些读取只是访问共享内存。 -迭代器的 `read()` 循环丢弃缓冲区内容，因为这个工具不通过迭代器接口产生输出，它只是利用遍历来驱动 BPF 回调。其他迭代器程序可能会向迭代器写入数据（例如格式化的套接字信息），但我们的工具完全通过 BSS 统计数据进行通信。 +迭代器回调用 `BPF_SEQ_PRINTF` 写出每个候选项，`read()` 循环把这些内容复制到 stdout，使 dry-run 可以直接用于下一步选择；汇总结果仍来自 BSS 统计。 ## 网络命名空间边界 @@ -366,7 +416,7 @@ TCP BPF 迭代器扫描的是触发它的进程所在网络命名空间中的套 ```bash sudo nsenter --net=/proc//ns/net -- ./tcp_quarantine \ - --destination 203.0.113.99 --port 443 --apply + 203.0.113.99:443 ``` 这里 `` 是目标网络命名空间中任意进程的 PID。`--net` 标志只改变网络命名空间而保持 mount 命名空间不变，所以工具二进制文件仍然可访问。 @@ -379,7 +429,7 @@ sudo nsenter --net=/proc//ns/net -- ./tcp_quarantine \ - 写操作可能产生 `EPIPE` 并伴随 `SIGPIPE` 信号。 - 具体行为因内核版本和操作时机而异。 -本教程附带的测试接受以上任何一种作为连接被拆除的有效指示。 +这些结果都可能表示连接已经被拆除；请结合你的应用行为和内核版本确认预期现象。 ## 编译与运行 @@ -394,45 +444,39 @@ make -j2 **Dry-run 模式**（默认）扫描匹配项但不销毁： ```bash -sudo ./tcp_quarantine --destination 127.0.0.1 --port 42063 +sudo ./tcp_quarantine 127.0.0.1:42063 ``` -**Apply 模式**销毁匹配的套接字： +从一行 `MATCH` 中复制本地端点交给 apply 模式；它只销毁这条完整 4-tuple： ```bash -sudo ./tcp_quarantine --destination 127.0.0.1 --port 42063 --apply +sudo ./tcp_quarantine --apply 127.0.0.1:55490 127.0.0.1:42063 ``` 命令行参考： ```text -Usage: ./tcp_quarantine --destination IPv4 --port PORT [--apply] [--verbose] +Usage: ./tcp_quarantine [--apply LOCAL_IPV4:PORT] REMOTE_IPV4:PORT [--verbose] Options: - -d, --destination IPv4 精确的远程 IPv4 地址 - -p, --port PORT 精确的远程 TCP 端口 (1-65535) - -a, --apply 销毁匹配的套接字 + -a, --apply IPv4:PORT 从 dry-run 输出选中的本地端点 -v, --verbose 打印 libbpf 诊断信息 -h, --help 显示帮助 ``` ### 示例会话 -以下输出在运行内核 `7.0.0-rc2+` 的 x86_64 环境上采集。自动化测试创建两条回环 TCP 连接：一个目标（将被销毁）和一个控制（应该存活）。它在运行工具前验证两条连接上的流量都正常： +假设一个服务与同一个远端 listener 建立了两条连接。正常的 dry-run 会列出两个候选；把其中一个本地端点复制到 apply 命令后，只会选中这一条连接： ```console -mode=dry-run destination=127.0.0.1:42063 scanned=6 established=4 matched=1 destroyed=0 failed=0 -mode=apply destination=127.0.0.1:42063 scanned=6 established=4 matched=1 destroyed=1 failed=0 -PASS: dry-run preserved both connections; apply destroyed only the target +MATCH local=127.0.0.1:55490 remote=127.0.0.1:42063 +MATCH local=127.0.0.1:55494 remote=127.0.0.1:42063 +SUMMARY mode=dry-run remote=127.0.0.1:42063 scanned=8 established=6 matched=2 destroyed=0 failed=0 +MATCH local=127.0.0.1:55490 remote=127.0.0.1:42063 +SUMMARY mode=apply remote=127.0.0.1:42063 local=127.0.0.1:55490 scanned=7 established=5 matched=1 destroyed=1 failed=0 ``` -套接字计数和端口号每次运行都会变化。重要的是：恰好一个套接字匹配，dry-run 让它保持完好，apply 成功销毁了它，控制连接继续正常工作。 - -运行自动化测试： - -```bash -sudo make test -``` +socket 计数和端口每次都会变化。重要的是：dry-run 列出两个远端匹配且不改变它们；apply 对复制的 4-tuple 报告 `matched=1 destroyed=1`，另一条 socket 保持可用。 ### 环境要求 @@ -449,14 +493,14 @@ sudo make test ## 总结 -本教程展示了如何使用 BPF 迭代器和 `bpf_sock_destroy` kfunc 精确终止特定 TCP 连接。与杀进程、防火墙规则或用户态 RST 注入相比，这种方法： +本教程展示了如何使用 BPF 迭代器和 `bpf_sock_destroy` kfunc 精确终止一条选定的 TCP 连接。与杀进程、防火墙规则或用户态 RST 注入相比，这种方法： - 原子操作，检查和动作之间不存在竞态窗口 - 不需要猜测 TCP 序列号 - 不影响同一进程上的其他连接 - 不受加密或特殊网络配置的影响 -工具目前实现的是单次遍历、精确匹配一个 IPv4 目标地址。可能的扩展包括 IPv6 支持、通配符匹配、持续监控模式、进程/cgroup 归属、多命名空间编排，以及与威胁情报源集成。 +工具把完整 IPv4 4-tuple 作为最小安全销毁选择器。可能的扩展包括 IPv6、进程/cgroup 归属、多命名空间编排和威胁情报集成；宽泛的通配销毁故意不属于本教程。 > 如果你想深入了解 eBPF，请访问我们的教程代码仓库 或网站 。 diff --git a/src/51-tcp-quarantine/tcp_quarantine.bpf.c b/src/51-tcp-quarantine/tcp_quarantine.bpf.c index 6eb02dad..2cc00990 100644 --- a/src/51-tcp-quarantine/tcp_quarantine.bpf.c +++ b/src/51-tcp-quarantine/tcp_quarantine.bpf.c @@ -12,6 +12,8 @@ char LICENSE[] SEC("license") = "GPL"; const volatile __u32 target_addr; const volatile __u16 target_port; +const volatile __u32 target_local_addr; +const volatile __u16 target_local_port; const volatile bool apply; struct quarantine_stats stats; @@ -22,8 +24,11 @@ SEC("iter/tcp") int quarantine_tcp(struct bpf_iter__tcp *ctx) { struct sock_common *sk = ctx->sk_common; - __u32 dst_addr; + struct seq_file *seq = ctx->meta->seq; + __u32 dst_addr, local_addr; + __u32 dst_host, local_host; __u16 dst_port; + __u16 local_port; __u16 family; __u8 state; int err; @@ -42,8 +47,21 @@ int quarantine_tcp(struct bpf_iter__tcp *ctx) dst_port = BPF_CORE_READ(sk, skc_dport); if (dst_addr != target_addr || dst_port != bpf_htons(target_port)) return 0; + local_addr = BPF_CORE_READ(sk, skc_rcv_saddr); + local_port = BPF_CORE_READ(sk, skc_num); + if (apply && (local_addr != target_local_addr || + local_port != target_local_port)) + return 0; stats.matched++; + local_host = bpf_ntohl(local_addr); + dst_host = bpf_ntohl(dst_addr); + BPF_SEQ_PRINTF(seq, + "MATCH local=%u.%u.%u.%u:%u remote=%u.%u.%u.%u:%u\n", + local_host >> 24, (local_host >> 16) & 0xff, + (local_host >> 8) & 0xff, local_host & 0xff, local_port, + dst_host >> 24, (dst_host >> 16) & 0xff, + (dst_host >> 8) & 0xff, dst_host & 0xff, target_port); if (!apply) return 0; diff --git a/src/51-tcp-quarantine/tcp_quarantine.c b/src/51-tcp-quarantine/tcp_quarantine.c index 72b5f1bb..946d29fd 100644 --- a/src/51-tcp-quarantine/tcp_quarantine.c +++ b/src/51-tcp-quarantine/tcp_quarantine.c @@ -13,12 +13,18 @@ #include "tcp_quarantine.skel.h" static struct env { - const char *destination; - unsigned int port; + const char *remote_argument; + const char *local_argument; bool apply; bool verbose; } env; +struct endpoint { + struct in_addr address; + unsigned int port; + char text[INET_ADDRSTRLEN + 7]; +}; + static int libbpf_print_fn(enum libbpf_print_level level, const char *format, va_list args) { @@ -30,15 +36,14 @@ static int libbpf_print_fn(enum libbpf_print_level level, const char *format, static void usage(const char *program) { fprintf(stderr, - "Usage: %s --destination IPv4 --port PORT [--apply] [--verbose]\n" + "Usage: %s [--apply LOCAL_IPV4:PORT] REMOTE_IPV4:PORT [--verbose]\n" "\n" - "Find established TCP client connections to an exact destination.\n" - "The default is a safe dry run; --apply destroys matching sockets.\n" + "List established TCP clients to REMOTE_IPV4:PORT.\n" + "The default is a dry run. Copy one listed local endpoint into\n" + "--apply to destroy only that exact IPv4 4-tuple.\n" "\n" "Options:\n" - " -d, --destination IPv4 exact remote IPv4 address\n" - " -p, --port PORT exact remote TCP port (1-65535)\n" - " -a, --apply destroy matching sockets\n" + " -a, --apply IPv4:PORT local endpoint selected from dry-run output\n" " -v, --verbose print libbpf diagnostics\n" " -h, --help show this help\n", program); @@ -57,31 +62,42 @@ static int parse_port(const char *value, unsigned int *port) return 0; } +static int parse_endpoint(const char *value, struct endpoint *endpoint) +{ + char address[INET_ADDRSTRLEN]; + const char *separator = strrchr(value, ':'); + size_t address_length; + + if (!separator || separator == value) + return -EINVAL; + address_length = separator - value; + if (address_length >= sizeof(address)) + return -EINVAL; + memcpy(address, value, address_length); + address[address_length] = '\0'; + if (inet_pton(AF_INET, address, &endpoint->address) != 1 || + parse_port(separator + 1, &endpoint->port)) + return -EINVAL; + snprintf(endpoint->text, sizeof(endpoint->text), "%s:%u", + address, endpoint->port); + return 0; +} + static int parse_args(int argc, char **argv) { static const struct option options[] = { - { "destination", required_argument, NULL, 'd' }, - { "port", required_argument, NULL, 'p' }, - { "apply", no_argument, NULL, 'a' }, + { "apply", required_argument, NULL, 'a' }, { "verbose", no_argument, NULL, 'v' }, { "help", no_argument, NULL, 'h' }, {}, }; int option; - while ((option = getopt_long(argc, argv, "d:p:avh", options, NULL)) != -1) { + while ((option = getopt_long(argc, argv, "a:vh", options, NULL)) != -1) { switch (option) { - case 'd': - env.destination = optarg; - break; - case 'p': - if (parse_port(optarg, &env.port)) { - fprintf(stderr, "invalid TCP port: %s\n", optarg); - return -EINVAL; - } - break; case 'a': env.apply = true; + env.local_argument = optarg; break; case 'v': env.verbose = true; @@ -94,15 +110,16 @@ static int parse_args(int argc, char **argv) } } - if (!env.destination || !env.port || optind != argc) + if (argc - optind != 1) return -EINVAL; + env.remote_argument = argv[optind]; return 0; } static int run_iterator(struct bpf_program *program) { struct bpf_link *link; - char buffer[256]; + char buffer[4096]; int iter_fd, length, err; link = bpf_program__attach_iter(program, NULL); @@ -119,8 +136,13 @@ static int run_iterator(struct bpf_program *program) goto cleanup; } - while ((length = read(iter_fd, buffer, sizeof(buffer))) > 0) - ; + while ((length = read(iter_fd, buffer, sizeof(buffer))) > 0) { + if (fwrite(buffer, 1, length, stdout) != (size_t)length) { + err = -EIO; + fprintf(stderr, "failed to print iterator output\n"); + break; + } + } if (length < 0) { err = -errno; fprintf(stderr, "failed while scanning TCP sockets: %s\n", strerror(errno)); @@ -135,7 +157,8 @@ static int run_iterator(struct bpf_program *program) int main(int argc, char **argv) { struct tcp_quarantine_bpf *skel = NULL; - struct in_addr destination; + struct endpoint remote = {}; + struct endpoint local = {}; int err; err = parse_args(argc, argv); @@ -143,8 +166,14 @@ int main(int argc, char **argv) usage(argv[0]); return 1; } - if (inet_pton(AF_INET, env.destination, &destination) != 1) { - fprintf(stderr, "invalid IPv4 destination: %s\n", env.destination); + if (parse_endpoint(env.remote_argument, &remote)) { + fprintf(stderr, "invalid remote IPv4 endpoint: %s\n", + env.remote_argument); + return 1; + } + if (env.apply && parse_endpoint(env.local_argument, &local)) { + fprintf(stderr, "invalid local IPv4 endpoint: %s\n", + env.local_argument); return 1; } @@ -155,8 +184,10 @@ int main(int argc, char **argv) return 1; } - skel->rodata->target_addr = destination.s_addr; - skel->rodata->target_port = env.port; + skel->rodata->target_addr = remote.address.s_addr; + skel->rodata->target_port = remote.port; + skel->rodata->target_local_addr = local.address.s_addr; + skel->rodata->target_local_port = local.port; skel->rodata->apply = env.apply; err = tcp_quarantine_bpf__load(skel); @@ -173,9 +204,10 @@ int main(int argc, char **argv) if (err) goto cleanup; - printf("mode=%s destination=%s:%u scanned=%llu established=%llu " + printf("SUMMARY mode=%s remote=%s%s%s scanned=%llu established=%llu " "matched=%llu destroyed=%llu failed=%llu\n", - env.apply ? "apply" : "dry-run", env.destination, env.port, + env.apply ? "apply" : "dry-run", remote.text, + env.apply ? " local=" : "", env.apply ? local.text : "", skel->bss->stats.scanned, skel->bss->stats.established, skel->bss->stats.matched, skel->bss->stats.destroyed, skel->bss->stats.failed); diff --git a/src/51-tcp-quarantine/tests/test_tcp_quarantine.py b/src/51-tcp-quarantine/tests/test_tcp_quarantine.py index de762174..085046c9 100644 --- a/src/51-tcp-quarantine/tests/test_tcp_quarantine.py +++ b/src/51-tcp-quarantine/tests/test_tcp_quarantine.py @@ -1,10 +1,11 @@ #!/usr/bin/env python3 -"""End-to-end smoke test for the TCP quarantine tool.""" +"""End-to-end safety checks for exact TCP 4-tuple quarantine.""" from __future__ import annotations import errno import pathlib +import re import socket import subprocess import sys @@ -12,98 +13,129 @@ BROKEN_ERRORS = {errno.ECONNABORTED, errno.ECONNRESET, errno.EPIPE} +MATCH = re.compile(r"^MATCH local=([^ ]+) remote=([^ ]+)$", re.MULTILINE) -def open_connection() -> tuple[socket.socket, socket.socket, socket.socket, int]: - listener = socket.socket(socket.AF_INET, socket.SOCK_STREAM) - listener.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) - listener.bind(("127.0.0.1", 0)) - listener.listen(1) - client = socket.create_connection(listener.getsockname(), timeout=2) - server, _ = listener.accept() - client.settimeout(1) - server.settimeout(1) - return listener, client, server, listener.getsockname()[1] +def open_listener() -> socket.socket: + listener = socket.socket(socket.AF_INET, socket.SOCK_STREAM) + listener.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) + listener.bind(("127.0.0.1", 0)) + listener.listen(4) + return listener + + +def connect(listener: socket.socket) -> tuple[socket.socket, socket.socket]: + client = socket.create_connection(listener.getsockname(), timeout=2) + server, _ = listener.accept() + client.settimeout(1) + server.settimeout(1) + return client, server + + +def endpoint(address: tuple[str, int]) -> str: + return f"{address[0]}:{address[1]}" def round_trip(client: socket.socket, server: socket.socket, payload: bytes) -> None: - client.sendall(payload) - assert server.recv(len(payload)) == payload - server.sendall(payload) - assert client.recv(len(payload)) == payload + client.sendall(payload) + assert server.recv(len(payload)) == payload + server.sendall(payload) + assert client.recv(len(payload)) == payload -def run_tool(binary: pathlib.Path, port: int, apply: bool) -> str: - command = [str(binary), "--destination", "127.0.0.1", "--port", str(port)] - if apply: - command.append("--apply") - result = subprocess.run(command, check=True, text=True, capture_output=True) - print(result.stdout, end="") - return result.stdout +def run_tool(binary: pathlib.Path, remote: str, local: str | None = None) -> str: + command = [str(binary), remote] + if local is not None: + command.extend(("--apply", local)) + result = subprocess.run(command, check=True, text=True, capture_output=True) + print(result.stdout, end="") + return result.stdout def wait_for_destroy(client: socket.socket) -> None: - deadline = time.monotonic() + 2 - while time.monotonic() < deadline: - try: - client.send(b"x") - except OSError as error: - if error.errno in BROKEN_ERRORS: - return - raise - error = client.getsockopt(socket.SOL_SOCKET, socket.SO_ERROR) - if error in BROKEN_ERRORS: - return - time.sleep(0.05) - raise AssertionError("quarantined TCP socket remained usable") + deadline = time.monotonic() + 2 + while time.monotonic() < deadline: + try: + client.send(b"x") + except OSError as error: + if error.errno in BROKEN_ERRORS: + return + raise + error = client.getsockopt(socket.SOL_SOCKET, socket.SO_ERROR) + if error in BROKEN_ERRORS: + return + time.sleep(0.05) + raise AssertionError("selected TCP socket remained usable") def main() -> int: - if len(sys.argv) != 2: - raise SystemExit(f"usage: {sys.argv[0]} /path/to/tcp_quarantine") - if not hasattr(socket, "SO_ERROR"): - raise SystemExit("SO_ERROR is unavailable") - - binary = pathlib.Path(sys.argv[1]).resolve() - if not binary.is_file(): - raise SystemExit(f"missing binary: {binary}") - - invalid = subprocess.run( - [str(binary), "--destination", "not-an-ip", "--port", "1"], - text=True, - capture_output=True, - ) - assert invalid.returncode != 0 - assert "invalid IPv4 destination" in invalid.stderr - - target = open_connection() - control = open_connection() - all_sockets = target[:3] + control[:3] - try: - round_trip(target[1], target[2], b"before-target") - round_trip(control[1], control[2], b"before-control") - - dry_run = run_tool(binary, target[3], apply=False) - assert "mode=dry-run" in dry_run - assert "matched=1" in dry_run - assert "destroyed=0" in dry_run - round_trip(target[1], target[2], b"after-dry-run") - round_trip(control[1], control[2], b"control-after-dry-run") - - applied = run_tool(binary, target[3], apply=True) - assert "mode=apply" in applied - assert "matched=1" in applied - assert "destroyed=1" in applied - assert "failed=0" in applied - wait_for_destroy(target[1]) - round_trip(control[1], control[2], b"control-survives") - finally: - for sock in all_sockets: - sock.close() - - print("PASS: dry-run preserved both connections; apply destroyed only the target") - return 0 + if len(sys.argv) != 2: + raise SystemExit(f"usage: {sys.argv[0]} /path/to/tcp_quarantine") + + binary = pathlib.Path(sys.argv[1]).resolve() + if not binary.is_file(): + raise SystemExit(f"missing binary: {binary}") + + invalid = subprocess.run( + [str(binary), "not-an-endpoint"], + text=True, + capture_output=True, + ) + assert invalid.returncode != 0 + assert "invalid remote IPv4 endpoint" in invalid.stderr + + target_listener = open_listener() + control_listener = open_listener() + selected_client, selected_server = connect(target_listener) + sibling_client, sibling_server = connect(target_listener) + control_client, control_server = connect(control_listener) + sockets = ( + target_listener, + control_listener, + selected_client, + selected_server, + sibling_client, + sibling_server, + control_client, + control_server, + ) + try: + remote = endpoint(target_listener.getsockname()) + selected_local = endpoint(selected_client.getsockname()) + sibling_local = endpoint(sibling_client.getsockname()) + round_trip(selected_client, selected_server, b"before-selected") + round_trip(sibling_client, sibling_server, b"before-sibling") + round_trip(control_client, control_server, b"before-control") + + dry_run = run_tool(binary, remote) + matches = set(MATCH.findall(dry_run)) + assert matches == { + (selected_local, remote), + (sibling_local, remote), + }, dry_run + assert "SUMMARY mode=dry-run" in dry_run + assert "matched=2" in dry_run + assert "destroyed=0" in dry_run + round_trip(selected_client, selected_server, b"after-dry-run") + round_trip(sibling_client, sibling_server, b"sibling-after-dry-run") + + applied = run_tool(binary, remote, selected_local) + assert f"MATCH local={selected_local} remote={remote}" in applied + assert sibling_local not in applied + assert "SUMMARY mode=apply" in applied + assert "matched=1" in applied + assert "destroyed=1" in applied + assert "failed=0" in applied + wait_for_destroy(selected_client) + round_trip(sibling_client, sibling_server, b"sibling-survives") + round_trip(control_client, control_server, b"control-survives") + finally: + for sock in sockets: + sock.close() + + print("PASS: dry-run listed two sockets; exact 4-tuple apply destroyed only one") + return 0 if __name__ == "__main__": - raise SystemExit(main()) + raise SystemExit(main()) diff --git a/src/52-fsession-latency/README.md b/src/52-fsession-latency/README.md index c3460c7f..ad966cd0 100644 --- a/src/52-fsession-latency/README.md +++ b/src/52-fsession-latency/README.md @@ -1,8 +1,10 @@ # eBPF Tutorial: Tracing Slow vfs_read Calls with fsession -When a file-backed service shows read-latency spikes, application-level timing tells you that requests slowed down, but it cannot distinguish whether the kernel blocked on I/O or user-space logic took too long. The useful questions are: which thread issued the read, how many bytes did it request, what did the call return, and how long did that single `vfs_read` invocation take? +When a file-backed service shows read-latency spikes, application-level timing tells you that requests slowed down, but it cannot distinguish whether the kernel spent time inside `vfs_read` or user-space logic took too long. The useful questions are: which thread issued the read, which VFS object was involved, how many bytes did it request, what did the call return, and how long did that single `vfs_read` invocation take? -This tutorial demonstrates how to measure `vfs_read` call latency using the **fsession** mechanism introduced in Linux 7.0. fsession is a new eBPF program type that runs once at function entry and once at return, with built-in per-invocation storage for correlating the two phases. The tool we build timestamps function entry, computes latency at return, filters by process and threshold, and reports slow-read events through a ring buffer. +This tutorial demonstrates how to measure `vfs_read` call latency using the **fsession** mechanism introduced in Linux 7.0. fsession is a new eBPF program type that runs once at function entry and once at return, with built-in per-invocation storage for correlating the two phases. The tool timestamps function entry, computes latency at return, filters by process and threshold, and reports the object type plus a `major:minor:inode` identity through a ring buffer. + +`vfs_read` is not limited to regular files: pipes, character devices, and other file-backed objects also pass through it. A slow event therefore identifies a slow `vfs_read` call, not necessarily slow storage. Use the reported `type` together with the `major:minor:inode` identity to distinguish regular-file reads from FIFOs and other VFS objects. > Complete source: @@ -110,6 +112,10 @@ struct latency_event { unsigned long long requested; long long result; unsigned long long latency_ns; + unsigned int device_major; + unsigned int device_minor; + unsigned long long inode; + unsigned int mode; char comm[FSESSION_COMM_LEN]; }; @@ -131,9 +137,105 @@ The four counters track: `FSESSION_COMM_LEN` is 16 to match the kernel's `TASK_COMM_LEN`. +Each event also carries `device_major`, `device_minor`, `inode`, and `mode`. The BPF program splits the kernel's raw `s_dev` encoding into its 12-bit major and 20-bit minor fields. Combined with `i_ino`, this gives a stable VFS object identity; `i_mode` lets user space print `regular`, `fifo`, `character`, and other object types. + ### BPF Program -`fsession_latency.bpf.c` is the core of the tool. Let's walk through it section by section. +`fsession_latency.bpf.c` is the core of the tool. Here is the complete kernel-side program: + +```c +// SPDX-License-Identifier: GPL-2.0 +#define bpf_session_is_return bpf_session_is_return_vmlinux_snapshot +#define bpf_session_cookie bpf_session_cookie_vmlinux_snapshot +#include "vmlinux.h" +#undef bpf_session_is_return +#undef bpf_session_cookie +#include +#include +#include +#include "fsession_latency.h" + +#define KERNEL_MINOR_BITS 20 +#define KERNEL_MINOR_MASK ((1U << KERNEL_MINOR_BITS) - 1) + +char LICENSE[] SEC("license") = "GPL"; + +const volatile __u64 threshold_ns; +const volatile __u32 target_tgid; + +struct latency_stats stats; + +struct { + __uint(type, BPF_MAP_TYPE_RINGBUF); + __uint(max_entries, 256 * 1024); +} events SEC(".maps"); + +/* + * The repository vmlinux.h snapshot predates the ctx argument on these + * kfunc prototypes. Rename those stale declarations while including the + * snapshot, then provide the Linux 7.0 signatures below. + */ +extern bool bpf_session_is_return(void *ctx) __ksym; +extern __u64 *bpf_session_cookie(void *ctx) __ksym; + +SEC("fsession/vfs_read") +int BPF_PROG(measure_vfs_read, struct file *file, char *buf, size_t count, + loff_t *pos, ssize_t ret) +{ + __u64 pid_tgid = bpf_get_current_pid_tgid(); + __u64 *started = bpf_session_cookie(ctx); + struct latency_event *event; + struct inode *inode; + __u32 device; + __u64 latency; + + if (!bpf_session_is_return(ctx)) { + if (target_tgid && pid_tgid >> 32 != target_tgid) { + *started = 0; + return 0; + } + *started = bpf_ktime_get_ns(); + return 0; + } + + if (!*started) + return 0; + + latency = bpf_ktime_get_ns() - *started; + __sync_fetch_and_add(&stats.calls, 1); + if (ret < 0) + __sync_fetch_and_add(&stats.errors, 1); + if (latency < threshold_ns) + return 0; + + __sync_fetch_and_add(&stats.slow, 1); + event = bpf_ringbuf_reserve(&events, sizeof(*event), 0); + if (!event) { + __sync_fetch_and_add(&stats.dropped, 1); + return 0; + } + + __builtin_memset(event, 0, sizeof(*event)); + event->pid = (__u32)pid_tgid; + event->tgid = pid_tgid >> 32; + event->requested = count; + event->result = ret; + event->latency_ns = latency; + inode = BPF_CORE_READ(file, f_inode); + if (inode) { + device = BPF_CORE_READ(inode, i_sb, s_dev); + event->device_major = device >> KERNEL_MINOR_BITS; + event->device_minor = device & KERNEL_MINOR_MASK; + event->inode = BPF_CORE_READ(inode, i_ino); + event->mode = BPF_CORE_READ(inode, i_mode); + } + bpf_get_current_comm(event->comm, sizeof(event->comm)); + bpf_ringbuf_submit(event, 0); + return 0; +} +``` + +The walkthrough below explains the same program section by section. ```c // SPDX-License-Identifier: GPL-2.0 @@ -142,10 +244,14 @@ The four counters track: #include "vmlinux.h" #undef bpf_session_is_return #undef bpf_session_cookie +#include #include #include #include "fsession_latency.h" +#define KERNEL_MINOR_BITS 20 +#define KERNEL_MINOR_MASK ((1U << KERNEL_MINOR_BITS) - 1) + char LICENSE[] SEC("license") = "GPL"; ``` @@ -189,6 +295,8 @@ int BPF_PROG(measure_vfs_read, struct file *file, char *buf, size_t count, __u64 pid_tgid = bpf_get_current_pid_tgid(); __u64 *started = bpf_session_cookie(ctx); struct latency_event *event; + struct inode *inode; + __u32 device; __u64 latency; ``` @@ -235,20 +343,29 @@ If the latency is below the threshold, we're done - the call is counted but no e return 0; } + __builtin_memset(event, 0, sizeof(*event)); event->pid = (__u32)pid_tgid; event->tgid = pid_tgid >> 32; event->requested = count; event->result = ret; event->latency_ns = latency; + inode = BPF_CORE_READ(file, f_inode); + if (inode) { + device = BPF_CORE_READ(inode, i_sb, s_dev); + event->device_major = device >> KERNEL_MINOR_BITS; + event->device_minor = device & KERNEL_MINOR_MASK; + event->inode = BPF_CORE_READ(inode, i_ino); + event->mode = BPF_CORE_READ(inode, i_mode); + } bpf_get_current_comm(event->comm, sizeof(event->comm)); bpf_ringbuf_submit(event, 0); return 0; } ``` -For slow calls, increment the `slow` counter and try to reserve space in the ring buffer. If the reservation fails (buffer full), increment `dropped` so the user knows events were lost. On success, fill in the event fields and submit. +For slow calls, increment the `slow` counter and try to reserve space in the ring buffer. If the reservation fails (buffer full), increment `dropped` so the user knows events were lost. On success, zero the event, copy call fields, and read object identity from `file->f_inode` before submitting it. -Note that `count` (the requested byte count) is available directly - no need to have stored it at entry. This is the fsession advantage: function arguments persist to the return phase. +Note that both `count` and `file` are available directly - no need to store either at entry. This is the fsession advantage: function arguments persist to the return phase. The tool intentionally does not resolve a path; paths can be renamed or have multiple aliases. For a regular file, use the device to identify the mount and search it by inode, for example `find /mount -xdev -inum INODE -print`. ### User-Space Loader @@ -268,16 +385,18 @@ static int handle_event(void *context, void *data, size_t size) { const struct latency_event *event = data; - printf("EVENT comm=%-16s tgid=%u pid=%u requested=%llu result=%lld " - "latency_us=%llu\n", - event->comm, event->tgid, event->pid, event->requested, - event->result, event->latency_ns / 1000); + printf("EVENT comm=%-16s tgid=%u pid=%u object=%u:%u:%llu type=%s " + "requested=%llu result=%lld latency_us=%llu\n", + event->comm, event->tgid, event->pid, + event->device_major, event->device_minor, event->inode, + file_type(event->mode), event->requested, event->result, + event->latency_ns / 1000); events_printed++; return 0; } ``` -Each event is printed with the process name, IDs, requested bytes, return value, and latency in microseconds. +Each event is printed with the process name, IDs, VFS identity and type, requested bytes, return value, and latency in microseconds. **Clean shutdown sequence**: ```c @@ -332,14 +451,12 @@ Options: ### Example Output -The following output was captured on x86_64 with kernel `7.0.0-rc2+`. The integration test performs 64 ordinary file reads plus one pipe read delayed by 50 ms. The pipe read exceeds the 10 ms threshold and appears in the event stream: +For example, a Python service waiting on a FIFO can produce this output when the writer responds after 50 ms: ```console Tracing vfs_read for 1 seconds; threshold=10000 us; pid=selected -EVENT comm=python3 tgid=1257 pid=1257 requested=1 result=1 latency_us=50160 +EVENT comm=python3 tgid=1245 pid=1245 object=0:16:784 type=fifo requested=1 result=1 latency_us=50246 SUMMARY calls=66 slow=1 errors=0 dropped=0 events=1 -TEST-SUMMARY miss_calls=0 high_threshold_calls=66 high_threshold_slow=0 threshold_calls=66 threshold_slow=1 events=1 dropped=0 -PASS: PID filtering, threshold miss, and slow-read reporting behaved as expected ``` The `SUMMARY` line shows: @@ -349,18 +466,6 @@ The `SUMMARY` line shows: - 0 events were dropped - 1 event was printed -### Running the Tests - -```bash -sudo make test -``` - -The integration test validates: -- **PID filtering**: Targeting a nonexistent TGID produces zero calls -- **Threshold filtering**: A very high threshold (10 seconds) produces zero slow events from fast reads -- **Event reporting**: A deliberately slow pipe read (50ms) triggers an event under a 10ms threshold -- **Accounting invariant**: With threshold=0, `slow == events + dropped` (every threshold-matching call is either delivered or counted as dropped) - ## Environment Requirements | Requirement | Details | @@ -372,7 +477,7 @@ The integration test validates: | Architecture | Tested on x86_64 | | Privilege | root | -Runtime behavior was tested on x86_64 with kernel `7.0.0-rc2+`. The upstream merge commit is `f17b474e36647c23801ef8fdaf2255ab66dd2973`. +The upstream merge commit is `f17b474e36647c23801ef8fdaf2255ab66dd2973`. ## Extending This Tool @@ -395,6 +500,8 @@ This tutorial demonstrated how to measure kernel function latency using the fses 3. **No state leaks**: The kernel manages cookie lifecycle - no cleanup required, no leaks possible 4. **Arguments available at return**: Function parameters persist to the return phase without explicit storage +For this `vfs_read` example, that last property also lets the return phase report the VFS object's device, inode, and type. Treat the result as generic VFS-call latency; use the object identity for follow-up rather than assuming every slow event is a disk problem. + The pattern is simple: check `bpf_session_is_return()`, use `bpf_session_cookie()` for per-invocation state, and access arguments/return value directly. This applies wherever you need to correlate function entry and exit. > To learn more about eBPF, check out our tutorial repository at or visit . diff --git a/src/52-fsession-latency/README.zh.md b/src/52-fsession-latency/README.zh.md index 8d0ed803..fad6fdaa 100644 --- a/src/52-fsession-latency/README.zh.md +++ b/src/52-fsession-latency/README.zh.md @@ -1,8 +1,10 @@ # eBPF 教程：使用 fsession 追踪慢速 vfs_read 调用 -当一个文件服务出现读延迟尖峰时，应用层计时只能告诉你请求变慢了，却无法区分是内核在 I/O 上阻塞还是用户态逻辑耗时。真正有用的问题是：哪个线程发起了读操作、请求了多少字节、调用返回了什么、这一次 `vfs_read` 花了多长时间？ +当一个文件服务出现读延迟尖峰时，应用层计时只能告诉你请求变慢了，却无法区分是内核在 `vfs_read` 中耗时还是用户态逻辑耗时。真正有用的问题是：哪个线程发起了读操作、涉及哪个 VFS 对象、请求了多少字节、调用返回了什么、这一次 `vfs_read` 花了多长时间？ -本教程展示如何使用 Linux 7.0 引入的 **fsession** 机制测量 `vfs_read` 调用延迟。fsession 是一种新的 eBPF 程序类型，它在函数进入时执行一次、返回时再执行一次，并提供内置的调用级存储来关联这两个阶段。我们构建的工具会在函数进入时记录时间戳、返回时计算延迟，按进程和阈值过滤后，通过 ring buffer 上报慢读事件。 +本教程展示如何使用 Linux 7.0 引入的 **fsession** 机制测量 `vfs_read` 调用延迟。fsession 是一种新的 eBPF 程序类型，它在函数进入时执行一次、返回时再执行一次，并提供内置的调用级存储来关联这两个阶段。工具会在函数进入时记录时间戳、返回时计算延迟，按进程和阈值过滤后，通过 ring buffer 上报对象类型和 `major:minor:inode` 身份。 + +`vfs_read` 不只处理普通文件，pipe、字符设备等文件对象也会经过它。因此慢事件表示某次 `vfs_read` 较慢，并不必然说明存储较慢。应结合输出中的 `type` 和 `major:minor:inode` 身份，区分普通文件读、FIFO 以及其他 VFS 对象。 > 完整源码： @@ -110,6 +112,10 @@ struct latency_event { unsigned long long requested; long long result; unsigned long long latency_ns; + unsigned int device_major; + unsigned int device_minor; + unsigned long long inode; + unsigned int mode; char comm[FSESSION_COMM_LEN]; }; @@ -131,9 +137,105 @@ struct latency_stats { `FSESSION_COMM_LEN` 设为 16，与内核的 `TASK_COMM_LEN` 一致。 +每个事件还携带 `device_major`、`device_minor`、`inode` 和 `mode`。BPF 程序把内核原始 `s_dev` 编码拆成 12 位主设备号和 20 位次设备号，再结合 `i_ino` 得到稳定的 VFS 对象身份；用户态根据 `i_mode` 输出 `regular`、`fifo`、`character` 等对象类型。 + ### BPF 程序 -`fsession_latency.bpf.c` 是工具的核心。我们逐段分析。 +`fsession_latency.bpf.c` 是工具的核心。下面是完整的内核态程序： + +```c +// SPDX-License-Identifier: GPL-2.0 +#define bpf_session_is_return bpf_session_is_return_vmlinux_snapshot +#define bpf_session_cookie bpf_session_cookie_vmlinux_snapshot +#include "vmlinux.h" +#undef bpf_session_is_return +#undef bpf_session_cookie +#include +#include +#include +#include "fsession_latency.h" + +#define KERNEL_MINOR_BITS 20 +#define KERNEL_MINOR_MASK ((1U << KERNEL_MINOR_BITS) - 1) + +char LICENSE[] SEC("license") = "GPL"; + +const volatile __u64 threshold_ns; +const volatile __u32 target_tgid; + +struct latency_stats stats; + +struct { + __uint(type, BPF_MAP_TYPE_RINGBUF); + __uint(max_entries, 256 * 1024); +} events SEC(".maps"); + +/* + * The repository vmlinux.h snapshot predates the ctx argument on these + * kfunc prototypes. Rename those stale declarations while including the + * snapshot, then provide the Linux 7.0 signatures below. + */ +extern bool bpf_session_is_return(void *ctx) __ksym; +extern __u64 *bpf_session_cookie(void *ctx) __ksym; + +SEC("fsession/vfs_read") +int BPF_PROG(measure_vfs_read, struct file *file, char *buf, size_t count, + loff_t *pos, ssize_t ret) +{ + __u64 pid_tgid = bpf_get_current_pid_tgid(); + __u64 *started = bpf_session_cookie(ctx); + struct latency_event *event; + struct inode *inode; + __u32 device; + __u64 latency; + + if (!bpf_session_is_return(ctx)) { + if (target_tgid && pid_tgid >> 32 != target_tgid) { + *started = 0; + return 0; + } + *started = bpf_ktime_get_ns(); + return 0; + } + + if (!*started) + return 0; + + latency = bpf_ktime_get_ns() - *started; + __sync_fetch_and_add(&stats.calls, 1); + if (ret < 0) + __sync_fetch_and_add(&stats.errors, 1); + if (latency < threshold_ns) + return 0; + + __sync_fetch_and_add(&stats.slow, 1); + event = bpf_ringbuf_reserve(&events, sizeof(*event), 0); + if (!event) { + __sync_fetch_and_add(&stats.dropped, 1); + return 0; + } + + __builtin_memset(event, 0, sizeof(*event)); + event->pid = (__u32)pid_tgid; + event->tgid = pid_tgid >> 32; + event->requested = count; + event->result = ret; + event->latency_ns = latency; + inode = BPF_CORE_READ(file, f_inode); + if (inode) { + device = BPF_CORE_READ(inode, i_sb, s_dev); + event->device_major = device >> KERNEL_MINOR_BITS; + event->device_minor = device & KERNEL_MINOR_MASK; + event->inode = BPF_CORE_READ(inode, i_ino); + event->mode = BPF_CORE_READ(inode, i_mode); + } + bpf_get_current_comm(event->comm, sizeof(event->comm)); + bpf_ringbuf_submit(event, 0); + return 0; +} +``` + +下面继续逐段解释同一个程序。 ```c // SPDX-License-Identifier: GPL-2.0 @@ -142,10 +244,14 @@ struct latency_stats { #include "vmlinux.h" #undef bpf_session_is_return #undef bpf_session_cookie +#include #include #include #include "fsession_latency.h" +#define KERNEL_MINOR_BITS 20 +#define KERNEL_MINOR_MASK ((1U << KERNEL_MINOR_BITS) - 1) + char LICENSE[] SEC("license") = "GPL"; ``` @@ -189,6 +295,8 @@ int BPF_PROG(measure_vfs_read, struct file *file, char *buf, size_t count, __u64 pid_tgid = bpf_get_current_pid_tgid(); __u64 *started = bpf_session_cookie(ctx); struct latency_event *event; + struct inode *inode; + __u32 device; __u64 latency; ``` @@ -235,20 +343,29 @@ TGID 在 `bpf_get_current_pid_tgid()` 返回值的高 32 位；低 32 位是线 return 0; } + __builtin_memset(event, 0, sizeof(*event)); event->pid = (__u32)pid_tgid; event->tgid = pid_tgid >> 32; event->requested = count; event->result = ret; event->latency_ns = latency; + inode = BPF_CORE_READ(file, f_inode); + if (inode) { + device = BPF_CORE_READ(inode, i_sb, s_dev); + event->device_major = device >> KERNEL_MINOR_BITS; + event->device_minor = device & KERNEL_MINOR_MASK; + event->inode = BPF_CORE_READ(inode, i_ino); + event->mode = BPF_CORE_READ(inode, i_mode); + } bpf_get_current_comm(event->comm, sizeof(event->comm)); bpf_ringbuf_submit(event, 0); return 0; } ``` -对于慢调用，递增 `slow` 计数器并尝试在 ring buffer 中预留空间。如果预留失败（缓冲区满），递增 `dropped` 让用户知道有事件丢失。成功后填入各字段并提交。 +对于慢调用，递增 `slow` 计数器并尝试在 ring buffer 中预留空间。如果预留失败（缓冲区满），递增 `dropped` 让用户知道有事件丢失。成功后先清零事件，复制调用字段，再从 `file->f_inode` 读取对象身份并提交。 -注意 `count`（请求的字节数）可以直接访问，不需要在进入时存储。这就是 fsession 的优势：函数参数在返回阶段仍然可用。 +注意 `count` 和 `file` 都可以直接访问，不需要在进入时存储。这就是 fsession 的优势：函数参数在返回阶段仍然可用。工具故意不解析路径，因为路径可能改名或有多个别名。对于普通文件，可先用设备号确定挂载点，再按 inode 搜索，例如 `find /mount -xdev -inum INODE -print`。 ### 用户空间加载器 @@ -268,16 +385,18 @@ static int handle_event(void *context, void *data, size_t size) { const struct latency_event *event = data; - printf("EVENT comm=%-16s tgid=%u pid=%u requested=%llu result=%lld " - "latency_us=%llu\n", - event->comm, event->tgid, event->pid, event->requested, - event->result, event->latency_ns / 1000); + printf("EVENT comm=%-16s tgid=%u pid=%u object=%u:%u:%llu type=%s " + "requested=%llu result=%lld latency_us=%llu\n", + event->comm, event->tgid, event->pid, + event->device_major, event->device_minor, event->inode, + file_type(event->mode), event->requested, event->result, + event->latency_ns / 1000); events_printed++; return 0; } ``` -每个事件打印进程名、ID、请求字节数、返回值和延迟（微秒）。 +每个事件打印进程名、ID、VFS 身份和类型、请求字节数、返回值和延迟（微秒）。 **干净关闭序列**： ```c @@ -332,14 +451,12 @@ Usage: ./fsession_latency [--threshold-us USEC] [--duration SEC] [--pid TGID] [- ### 输出示例 -以下输出在运行内核 `7.0.0-rc2+` 的 x86_64 环境中采集。集成测试执行 64 次普通文件读取加一次延迟 50 ms 的管道读取。管道读超过 10 ms 阈值后出现在事件流中： +例如，一个 Python 服务等待 FIFO，而写入端在 50 ms 后响应时，会产生下面的输出： ```console Tracing vfs_read for 1 seconds; threshold=10000 us; pid=selected -EVENT comm=python3 tgid=1257 pid=1257 requested=1 result=1 latency_us=50160 +EVENT comm=python3 tgid=1245 pid=1245 object=0:16:784 type=fifo requested=1 result=1 latency_us=50246 SUMMARY calls=66 slow=1 errors=0 dropped=0 events=1 -TEST-SUMMARY miss_calls=0 high_threshold_calls=66 high_threshold_slow=0 threshold_calls=66 threshold_slow=1 events=1 dropped=0 -PASS: PID filtering, threshold miss, and slow-read reporting behaved as expected ``` `SUMMARY` 行显示： @@ -349,18 +466,6 @@ PASS: PID filtering, threshold miss, and slow-read reporting behaved as expected - 0 个事件被丢弃 - 1 个事件被打印 -### 运行测试 - -```bash -sudo make test -``` - -集成测试验证： -- **PID 过滤**：针对不存在的 TGID 产生零调用 -- **阈值过滤**：很高的阈值（10 秒）使快速读取产生零慢事件 -- **事件上报**：故意延迟的管道读取（50ms）在 10ms 阈值下触发事件 -- **计数不变式**：阈值为 0 时，`slow == events + dropped`（每个达到阈值的调用要么被投递要么被计入丢弃） - ## 环境要求 | 要求 | 详情 | @@ -372,7 +477,7 @@ sudo make test | 架构 | 已在 x86_64 上测试 | | 权限 | root | -运行时行为在 x86_64 内核 `7.0.0-rc2+` 上测试。上游合并提交为 `f17b474e36647c23801ef8fdaf2255ab66dd2973`。 +上游合并提交为 `f17b474e36647c23801ef8fdaf2255ab66dd2973`。 ## 扩展方向 @@ -395,6 +500,8 @@ sudo make test 3. **无状态泄漏**：内核管理 cookie 生命周期，不需要清理，不可能泄漏 4. **返回时可访问参数**：函数参数在返回阶段仍然可用，无需显式存储 +在这个 `vfs_read` 示例中，最后一点还让返回阶段可以报告 VFS 对象的设备号、inode 和类型。应把结果理解为通用 VFS 调用延迟，并用对象身份继续定位，而不是假设每个慢事件都是磁盘问题。 + 模式很简单：检查 `bpf_session_is_return()`，用 `bpf_session_cookie()` 管理调用级状态，直接访问参数和返回值。这个模式适用于任何需要关联函数进入和退出的场景。 > 如果你想深入了解 eBPF，请查看我们的教程代码仓库 或访问我们的网站 。 diff --git a/src/52-fsession-latency/fsession_latency.bpf.c b/src/52-fsession-latency/fsession_latency.bpf.c index db903ee4..89dc95b3 100644 --- a/src/52-fsession-latency/fsession_latency.bpf.c +++ b/src/52-fsession-latency/fsession_latency.bpf.c @@ -4,10 +4,14 @@ #include "vmlinux.h" #undef bpf_session_is_return #undef bpf_session_cookie +#include #include #include #include "fsession_latency.h" +#define KERNEL_MINOR_BITS 20 +#define KERNEL_MINOR_MASK ((1U << KERNEL_MINOR_BITS) - 1) + char LICENSE[] SEC("license") = "GPL"; const volatile __u64 threshold_ns; @@ -35,6 +39,8 @@ int BPF_PROG(measure_vfs_read, struct file *file, char *buf, size_t count, __u64 pid_tgid = bpf_get_current_pid_tgid(); __u64 *started = bpf_session_cookie(ctx); struct latency_event *event; + struct inode *inode; + __u32 device; __u64 latency; if (!bpf_session_is_return(ctx)) { @@ -63,11 +69,20 @@ int BPF_PROG(measure_vfs_read, struct file *file, char *buf, size_t count, return 0; } + __builtin_memset(event, 0, sizeof(*event)); event->pid = (__u32)pid_tgid; event->tgid = pid_tgid >> 32; event->requested = count; event->result = ret; event->latency_ns = latency; + inode = BPF_CORE_READ(file, f_inode); + if (inode) { + device = BPF_CORE_READ(inode, i_sb, s_dev); + event->device_major = device >> KERNEL_MINOR_BITS; + event->device_minor = device & KERNEL_MINOR_MASK; + event->inode = BPF_CORE_READ(inode, i_ino); + event->mode = BPF_CORE_READ(inode, i_mode); + } bpf_get_current_comm(event->comm, sizeof(event->comm)); bpf_ringbuf_submit(event, 0); return 0; diff --git a/src/52-fsession-latency/fsession_latency.c b/src/52-fsession-latency/fsession_latency.c index cdc91a84..e03fff04 100644 --- a/src/52-fsession-latency/fsession_latency.c +++ b/src/52-fsession-latency/fsession_latency.c @@ -7,6 +7,7 @@ #include #include #include +#include #include #include #include "fsession_latency.h" @@ -26,6 +27,25 @@ static struct env { static unsigned long long events_printed; +static const char *file_type(unsigned int mode) +{ + if (S_ISREG(mode)) + return "regular"; + if (S_ISDIR(mode)) + return "directory"; + if (S_ISCHR(mode)) + return "character"; + if (S_ISBLK(mode)) + return "block"; + if (S_ISFIFO(mode)) + return "fifo"; + if (S_ISLNK(mode)) + return "symlink"; + if (S_ISSOCK(mode)) + return "socket"; + return "unknown"; +} + static void handle_signal(int signal) { (void)signal; @@ -137,10 +157,12 @@ static int handle_event(void *context, void *data, size_t size) if (size < sizeof(*event)) return 0; - printf("EVENT comm=%-16s tgid=%u pid=%u requested=%llu result=%lld " - "latency_us=%llu\n", - event->comm, event->tgid, event->pid, event->requested, - event->result, event->latency_ns / 1000); + printf("EVENT comm=%-16s tgid=%u pid=%u object=%u:%u:%llu type=%s " + "requested=%llu result=%lld latency_us=%llu\n", + event->comm, event->tgid, event->pid, + event->device_major, event->device_minor, event->inode, + file_type(event->mode), event->requested, event->result, + event->latency_ns / 1000); events_printed++; return 0; } diff --git a/src/52-fsession-latency/fsession_latency.h b/src/52-fsession-latency/fsession_latency.h index 0bc3cb6e..27be4ba5 100644 --- a/src/52-fsession-latency/fsession_latency.h +++ b/src/52-fsession-latency/fsession_latency.h @@ -10,6 +10,10 @@ struct latency_event { unsigned long long requested; long long result; unsigned long long latency_ns; + unsigned int device_major; + unsigned int device_minor; + unsigned long long inode; + unsigned int mode; char comm[FSESSION_COMM_LEN]; }; diff --git a/src/52-fsession-latency/tests/test_fsession_latency.py b/src/52-fsession-latency/tests/test_fsession_latency.py index c86e46ee..9b2445f2 100644 --- a/src/52-fsession-latency/tests/test_fsession_latency.py +++ b/src/52-fsession-latency/tests/test_fsession_latency.py @@ -17,8 +17,8 @@ r"SUMMARY calls=(\d+) slow=(\d+) errors=(\d+) dropped=(\d+) events=(\d+)" ) EVENT = re.compile( - r"^EVENT comm=.{16} tgid=(\d+) pid=(\d+) requested=(\d+) " - r"result=(-?\d+) latency_us=(\d+)$", + r"^EVENT comm=.{16} tgid=(\d+) pid=(\d+) object=(\d+):(\d+):(\d+) " + r"type=(\w+) requested=(\d+) result=(-?\d+) latency_us=(\d+)$", re.MULTILINE, ) @@ -174,17 +174,19 @@ def main() -> int: assert slow["calls"] >= 65, slow assert 1 <= slow["slow"] < slow["calls"], slow assert slow["events"] > 0, slow - event_records = [tuple(map(int, match.groups())) for match in EVENT.finditer(slow_output)] + event_records = [match.groups() for match in EVENT.finditer(slow_output)] assert any( - tgid == os.getpid() - and thread_pid == os.getpid() - and requested == 1 - and result == 1 - and latency_us >= 10000 - for tgid, thread_pid, requested, result, latency_us in event_records + int(record[0]) == os.getpid() + and int(record[1]) == os.getpid() + and int(record[4]) > 0 + and record[5] == "fifo" + and int(record[6]) == 1 + and int(record[7]) == 1 + and int(record[8]) >= 10000 + for record in event_records ), slow_output - deadline, _ = run_trace( + deadline, deadline_output = run_trace( binary, "--duration", "1", "--threshold-us", "0", @@ -195,6 +197,14 @@ def main() -> int: assert deadline["calls"] > 0, deadline assert deadline["events"] > 0, deadline assert deadline["slow"] == deadline["events"] + deadline["dropped"], deadline + source_stat = pathlib.Path(__file__).stat() + assert any( + int(match.group(3)) == os.major(source_stat.st_dev) + and int(match.group(4)) == os.minor(source_stat.st_dev) + and int(match.group(5)) == source_stat.st_ino + and match.group(6) == "regular" + for match in EVENT.finditer(deadline_output) + ), deadline_output tracing_line = next(line for line in slow_output.splitlines() if line.startswith("Tracing ")) first_event = next(line for line in slow_output.splitlines() if line.startswith("EVENT ")) @@ -209,7 +219,10 @@ def main() -> int: f"threshold_calls={slow['calls']} threshold_slow={slow['slow']} " f"events={slow['events']} dropped={slow['dropped']}" ) - print("PASS: PID filtering, threshold miss, and slow-read reporting behaved as expected") + print( + "PASS: PID filtering, threshold miss, and regular/fifo object identity " + "behaved as expected" + ) return 0 diff --git a/src/53-egress-pacer/README.md b/src/53-egress-pacer/README.md index 1ef6edf3..f275223a 100644 --- a/src/53-egress-pacer/README.md +++ b/src/53-egress-pacer/README.md @@ -384,17 +384,21 @@ The `SEC(".struct_ops")` block at the end registers the callbacks as a `Qdisc_op #include #include #include +#include #include #include #include #include #include +#include #include #include #include #include "egress_pacer.h" #include "egress_pacer.skel.h" +extern char **environ; + static volatile sig_atomic_t exiting; static struct env { @@ -575,10 +579,145 @@ static int wait_for_duration(void) return 0; } +static const char *find_tc_binary(void) +{ + static const char *const candidates[] = { + "/usr/sbin/tc", + "/sbin/tc", + }; + size_t index; + + for (index = 0; index < sizeof(candidates) / sizeof(candidates[0]); index++) { + if (!access(candidates[index], X_OK)) + return candidates[index]; + } + return NULL; +} + +static void print_qdisc_state(bool all_interfaces) +{ + const char *tc_binary = find_tc_binary(); + char *const all_arguments[] = { + (char *)"tc", (char *)"qdisc", (char *)"show", NULL, + }; + char *const interface_arguments[] = { + (char *)"tc", (char *)"qdisc", (char *)"show", (char *)"dev", + (char *)env.interface, NULL, + }; + char *const *arguments = all_interfaces ? all_arguments : interface_arguments; + posix_spawn_file_actions_t actions; + pid_t child; + pid_t waited; + int error; + int status; + + fprintf(stderr, "%s:\n", + all_interfaces ? "qdisc state across interfaces" : "current qdisc state"); + if (!tc_binary) { + fprintf(stderr, "tc was not found in /usr/sbin or /sbin\n"); + goto manual; + } + + error = posix_spawn_file_actions_init(&actions); + if (error) + goto spawn_failed; + error = posix_spawn_file_actions_adddup2(&actions, STDERR_FILENO, + STDOUT_FILENO); + if (!error) + error = posix_spawn(&child, tc_binary, &actions, NULL, arguments, + environ); + posix_spawn_file_actions_destroy(&actions); + if (error) + goto spawn_failed; + + do { + waited = waitpid(child, &status, 0); + } while (waited < 0 && errno == EINTR); + if (waited >= 0 && WIFEXITED(status) && !WEXITSTATUS(status)) + return; + if (waited < 0) + error = errno; + else + error = EIO; + +spawn_failed: + fprintf(stderr, "failed to run tc: %s\n", strerror(error)); +manual: + if (all_interfaces) + fprintf(stderr, "run manually: tc qdisc show\n"); + else + fprintf(stderr, "run manually: tc qdisc show dev %s\n", + env.interface); +} + +static int install_pacer(struct egress_pacer_bpf *skel, + struct bpf_tc_hook *hook) +{ + int error; + + error = egress_pacer_bpf__attach(skel); + if (error) { + if (error == -EEXIST) { + fprintf(stderr, + "bpf_pacer is already registered globally; another interface may own it\n"); + print_qdisc_state(true); + fprintf(stderr, + "inspect the bpf_pacer owner above before removing any root qdisc\n"); + } else { + fprintf(stderr, "failed to register bpf_pacer qdisc: %s\n", + strerror(-error)); + } + return error; + } + + error = bpf_tc_hook_create(hook); + if (!error) + return 0; + if (error == -EEXIST) { + fprintf(stderr, "refusing to replace the existing root qdisc on %s\n", + env.interface); + print_qdisc_state(false); + fprintf(stderr, + "if it is stale, recover with: sudo tc qdisc del dev %s root\n", + env.interface); + } else { + fprintf(stderr, "failed to attach bpf_pacer to %s: %s\n", + env.interface, strerror(-error)); + } + return error; +} + +static int cleanup_pacer(struct egress_pacer_bpf *skel, + struct bpf_tc_hook *hook, bool qdisc_created, int error) +{ + struct pacer_stats final_stats = {}; + int cleanup_error; + + if (qdisc_created) { + cleanup_error = bpf_tc_hook_destroy(hook); + if (cleanup_error) { + fprintf(stderr, "failed to remove bpf_pacer from %s: %s\n", + env.interface, strerror(-cleanup_error)); + if (!error) + error = cleanup_error; + } + } + if (skel && skel->bss) + final_stats = skel->bss->stats; + if (qdisc_created) { + printf("SUMMARY enqueued=%llu dequeued=%llu policy_dropped=%llu " + "cleanup_dropped=%llu bytes_dequeued=%llu max_qlen=%llu\n", + final_stats.enqueued, final_stats.dequeued, + final_stats.policy_dropped, final_stats.cleanup_dropped, + final_stats.bytes_dequeued, final_stats.max_qlen); + } + egress_pacer_bpf__destroy(skel); + return error != 0; +} + int main(int argc, char **argv) { struct egress_pacer_bpf *skel = NULL; - struct pacer_stats final_stats = {}; struct bpf_tc_hook hook = { .sz = sizeof(hook), .attach_point = BPF_TC_QDISC, @@ -588,7 +727,6 @@ int main(int argc, char **argv) }; bool qdisc_created = false; unsigned int ifindex; - int cleanup_err; int err; err = parse_args(argc, argv); @@ -626,24 +764,9 @@ int main(int argc, char **argv) goto cleanup; } - err = egress_pacer_bpf__attach(skel); - if (err) { - fprintf(stderr, "failed to register bpf_pacer qdisc: %s\n", - strerror(-err)); - goto cleanup; - } - - err = bpf_tc_hook_create(&hook); - if (err) { - if (err == -EEXIST) - fprintf(stderr, - "refusing to replace the existing root qdisc on %s\n", - env.interface); - else - fprintf(stderr, "failed to attach bpf_pacer to %s: %s\n", - env.interface, strerror(-err)); + err = install_pacer(skel, &hook); + if (err) goto cleanup; - } qdisc_created = true; printf("READY interface=%s rate_kbps=%llu queue_limit=%u duration=%u\n", @@ -653,26 +776,7 @@ int main(int argc, char **argv) err = wait_for_duration(); cleanup: - if (qdisc_created) { - cleanup_err = bpf_tc_hook_destroy(&hook); - if (cleanup_err) { - fprintf(stderr, "failed to remove bpf_pacer from %s: %s\n", - env.interface, strerror(-cleanup_err)); - if (!err) - err = cleanup_err; - } - } - if (skel && skel->bss) - final_stats = skel->bss->stats; - if (qdisc_created) { - printf("SUMMARY enqueued=%llu dequeued=%llu policy_dropped=%llu " - "cleanup_dropped=%llu bytes_dequeued=%llu max_qlen=%llu\n", - final_stats.enqueued, final_stats.dequeued, - final_stats.policy_dropped, final_stats.cleanup_dropped, - final_stats.bytes_dequeued, final_stats.max_qlen); - } - egress_pacer_bpf__destroy(skel); - return err != 0; + return cleanup_pacer(skel, &hook, qdisc_created, err); } ``` @@ -684,7 +788,7 @@ The loader follows a standard libbpf pattern: 4. Attach the `struct_ops` implementation, making `bpf_pacer` available as a qdisc type. 5. Create the root qdisc on the target interface with `bpf_tc_hook_create`. -The `bpf_tc_hook_create` call has exclusive semantics: if the interface already has a root qdisc, the kernel returns `-EEXIST` and the program exits without changing anything. This safety measure means the tool is best suited for dedicated test interfaces where you control the configuration. +The `bpf_tc_hook_create` call has exclusive semantics: if the interface already has a root qdisc, the kernel returns `-EEXIST` and the program exits without changing anything. For a root-qdisc conflict on the requested interface, the loader prints `tc qdisc show dev IFACE` and the recovery command `sudo tc qdisc del dev IFACE root`. A `struct_ops` name conflict is global rather than interface-local, so the loader instead prints `tc qdisc show` for every interface and does not guess which interface to modify. It never runs a destructive command automatically; remove a root qdisc only after confirming that the displayed `bpf_pacer` entry is stale. This safety measure means the tool is best suited for dedicated interfaces where you control the configuration. After successful attachment, the program prints `READY`, then polls every 100ms until duration expires or a signal arrives. Cleanup order matters: first `bpf_tc_hook_destroy` (which triggers reset and frees queued packets), then read final statistics, then destroy the skeleton. @@ -698,40 +802,27 @@ make clean make -j2 ``` -The easiest way to test is with the included integration test, which creates a temporary veth pair and cleans it up afterward. Root is required for loading BPF and modifying qdiscs: +Pick an interface you control and inspect its current root qdisc before attaching the pacer: ```bash -sudo python3 tests/test_egress_pacer.py ./egress_pacer -``` - -Here is output from a successful run on x86_64 with kernel `7.0.0-rc2+` (commit `a03114efd0720dff230388f7e160e427e54ea31b`): - -```console -READY interface=epac_tx rate_kbps=64 queue_limit=8 duration=2 -SUMMARY enqueued=9 dequeued=9 policy_dropped=31 cleanup_dropped=0 bytes_dequeued=9216 max_qlen=8 -TEST-SUMMARY attempts=40 received=9 send_errors=31 span_ms=1024 observed_kbps=64.0 -PASS: conflict refusal, bounded drops, pacing, accounting, normal/signal cleanup, and SIGKILL recovery succeeded +tc qdisc show dev veth-service ``` -What happened: the test sent 40 raw Ethernet frames (1024 bytes each, EtherType `0x88B5`) into a 64 Kbit/s queue limited to 8 packets. Nine packets made it through the queue, spaced over 1024ms, matching the expected rate. The other 31 sends hit the queue limit and went to `policy_dropped`. Exact numbers vary by system timing. - -The test also exercises edge cases: -- Pre-installs a `pfifo` to confirm the loader refuses to replace existing qdiscs. -- Sends SIGTERM with packets queued, verifying `cleanup_dropped > 0` and that `enqueued = dequeued + cleanup_dropped`. -- Sends SIGKILL (which skips cleanup), then uses `tc` to manually remove the residual qdisc and confirm the interface recovers. - -For manual testing, pick an interface you control and check its current qdisc: +If the interface is available for this experiment, start the pacer and let normal application traffic pass through it: ```bash -tc qdisc show dev veth-service +sudo ./egress_pacer --interface veth-service --rate-kbps 64000 --queue-limit 256 --duration 30 ``` -If it shows the default `noqueue` or `pfifo_fast`, you can attach the pacer: +The tool prints `READY` only after the qdisc is active. When the duration expires, its output has this shape: -```bash -sudo ./egress_pacer --interface veth-service --rate-kbps 64000 --queue-limit 256 --duration 30 +```console +READY interface=veth-service rate_kbps=64000 queue_limit=256 duration=30 +SUMMARY enqueued=... dequeued=... policy_dropped=... cleanup_dropped=0 bytes_dequeued=... max_qlen=... ``` +`enqueued` counts packets accepted by the qdisc, `dequeued` counts packets released to the device, and `policy_dropped` counts packets rejected when the queue was full. If the process is killed before normal cleanup and leaves `bpf_pacer` behind, the next run prints qdisc state across all interfaces. Find the interface whose line contains `bpf_pacer`, confirm that it is the stale instance, and then remove that actual root qdisc with `sudo tc qdisc del dev ACTUAL_IFACE root`. + Command-line options: ```text @@ -764,7 +855,7 @@ BPF qdisc requires Linux 6.16 or later. Your kernel must have BTF and BPF JIT en | libbpf | 1.6.0+ | | Privileges | root | -The repository includes libbpf and bpftool in `src/third_party`. The integration test additionally requires Python 3, raw socket capability, and `ip`/`tc` from iproute2. A veth pair is sufficient for testing; verified on x86_64. +The repository includes libbpf and bpftool in `src/third_party`. Use `tc` from iproute2 to inspect the target interface and recover a stale root qdisc if necessary. ## Summary diff --git a/src/53-egress-pacer/README.zh.md b/src/53-egress-pacer/README.zh.md index 3a4f5684..70bfa288 100644 --- a/src/53-egress-pacer/README.zh.md +++ b/src/53-egress-pacer/README.zh.md @@ -382,17 +382,21 @@ watchdog 机制（`bpf_qdisc_watchdog_schedule`）是 BPF qdisc 实现发送时 #include #include #include +#include #include #include #include #include #include +#include #include #include #include #include "egress_pacer.h" #include "egress_pacer.skel.h" +extern char **environ; + static volatile sig_atomic_t exiting; static struct env { @@ -573,10 +577,145 @@ static int wait_for_duration(void) return 0; } +static const char *find_tc_binary(void) +{ + static const char *const candidates[] = { + "/usr/sbin/tc", + "/sbin/tc", + }; + size_t index; + + for (index = 0; index < sizeof(candidates) / sizeof(candidates[0]); index++) { + if (!access(candidates[index], X_OK)) + return candidates[index]; + } + return NULL; +} + +static void print_qdisc_state(bool all_interfaces) +{ + const char *tc_binary = find_tc_binary(); + char *const all_arguments[] = { + (char *)"tc", (char *)"qdisc", (char *)"show", NULL, + }; + char *const interface_arguments[] = { + (char *)"tc", (char *)"qdisc", (char *)"show", (char *)"dev", + (char *)env.interface, NULL, + }; + char *const *arguments = all_interfaces ? all_arguments : interface_arguments; + posix_spawn_file_actions_t actions; + pid_t child; + pid_t waited; + int error; + int status; + + fprintf(stderr, "%s:\n", + all_interfaces ? "qdisc state across interfaces" : "current qdisc state"); + if (!tc_binary) { + fprintf(stderr, "tc was not found in /usr/sbin or /sbin\n"); + goto manual; + } + + error = posix_spawn_file_actions_init(&actions); + if (error) + goto spawn_failed; + error = posix_spawn_file_actions_adddup2(&actions, STDERR_FILENO, + STDOUT_FILENO); + if (!error) + error = posix_spawn(&child, tc_binary, &actions, NULL, arguments, + environ); + posix_spawn_file_actions_destroy(&actions); + if (error) + goto spawn_failed; + + do { + waited = waitpid(child, &status, 0); + } while (waited < 0 && errno == EINTR); + if (waited >= 0 && WIFEXITED(status) && !WEXITSTATUS(status)) + return; + if (waited < 0) + error = errno; + else + error = EIO; + +spawn_failed: + fprintf(stderr, "failed to run tc: %s\n", strerror(error)); +manual: + if (all_interfaces) + fprintf(stderr, "run manually: tc qdisc show\n"); + else + fprintf(stderr, "run manually: tc qdisc show dev %s\n", + env.interface); +} + +static int install_pacer(struct egress_pacer_bpf *skel, + struct bpf_tc_hook *hook) +{ + int error; + + error = egress_pacer_bpf__attach(skel); + if (error) { + if (error == -EEXIST) { + fprintf(stderr, + "bpf_pacer is already registered globally; another interface may own it\n"); + print_qdisc_state(true); + fprintf(stderr, + "inspect the bpf_pacer owner above before removing any root qdisc\n"); + } else { + fprintf(stderr, "failed to register bpf_pacer qdisc: %s\n", + strerror(-error)); + } + return error; + } + + error = bpf_tc_hook_create(hook); + if (!error) + return 0; + if (error == -EEXIST) { + fprintf(stderr, "refusing to replace the existing root qdisc on %s\n", + env.interface); + print_qdisc_state(false); + fprintf(stderr, + "if it is stale, recover with: sudo tc qdisc del dev %s root\n", + env.interface); + } else { + fprintf(stderr, "failed to attach bpf_pacer to %s: %s\n", + env.interface, strerror(-error)); + } + return error; +} + +static int cleanup_pacer(struct egress_pacer_bpf *skel, + struct bpf_tc_hook *hook, bool qdisc_created, int error) +{ + struct pacer_stats final_stats = {}; + int cleanup_error; + + if (qdisc_created) { + cleanup_error = bpf_tc_hook_destroy(hook); + if (cleanup_error) { + fprintf(stderr, "failed to remove bpf_pacer from %s: %s\n", + env.interface, strerror(-cleanup_error)); + if (!error) + error = cleanup_error; + } + } + if (skel && skel->bss) + final_stats = skel->bss->stats; + if (qdisc_created) { + printf("SUMMARY enqueued=%llu dequeued=%llu policy_dropped=%llu " + "cleanup_dropped=%llu bytes_dequeued=%llu max_qlen=%llu\n", + final_stats.enqueued, final_stats.dequeued, + final_stats.policy_dropped, final_stats.cleanup_dropped, + final_stats.bytes_dequeued, final_stats.max_qlen); + } + egress_pacer_bpf__destroy(skel); + return error != 0; +} + int main(int argc, char **argv) { struct egress_pacer_bpf *skel = NULL; - struct pacer_stats final_stats = {}; struct bpf_tc_hook hook = { .sz = sizeof(hook), .attach_point = BPF_TC_QDISC, @@ -586,7 +725,6 @@ int main(int argc, char **argv) }; bool qdisc_created = false; unsigned int ifindex; - int cleanup_err; int err; err = parse_args(argc, argv); @@ -624,24 +762,9 @@ int main(int argc, char **argv) goto cleanup; } - err = egress_pacer_bpf__attach(skel); - if (err) { - fprintf(stderr, "failed to register bpf_pacer qdisc: %s\n", - strerror(-err)); - goto cleanup; - } - - err = bpf_tc_hook_create(&hook); - if (err) { - if (err == -EEXIST) - fprintf(stderr, - "refusing to replace the existing root qdisc on %s\n", - env.interface); - else - fprintf(stderr, "failed to attach bpf_pacer to %s: %s\n", - env.interface, strerror(-err)); + err = install_pacer(skel, &hook); + if (err) goto cleanup; - } qdisc_created = true; printf("READY interface=%s rate_kbps=%llu queue_limit=%u duration=%u\n", @@ -651,26 +774,7 @@ int main(int argc, char **argv) err = wait_for_duration(); cleanup: - if (qdisc_created) { - cleanup_err = bpf_tc_hook_destroy(&hook); - if (cleanup_err) { - fprintf(stderr, "failed to remove bpf_pacer from %s: %s\n", - env.interface, strerror(-cleanup_err)); - if (!err) - err = cleanup_err; - } - } - if (skel && skel->bss) - final_stats = skel->bss->stats; - if (qdisc_created) { - printf("SUMMARY enqueued=%llu dequeued=%llu policy_dropped=%llu " - "cleanup_dropped=%llu bytes_dequeued=%llu max_qlen=%llu\n", - final_stats.enqueued, final_stats.dequeued, - final_stats.policy_dropped, final_stats.cleanup_dropped, - final_stats.bytes_dequeued, final_stats.max_qlen); - } - egress_pacer_bpf__destroy(skel); - return err != 0; + return cleanup_pacer(skel, &hook, qdisc_created, err); } ``` @@ -682,7 +786,7 @@ cleanup: 4. 挂载 `struct_ops` 实现，使 `bpf_pacer` 成为可用的 qdisc 类型。 5. 用 `bpf_tc_hook_create` 在目标接口上创建 root qdisc。 -`bpf_tc_hook_create` 调用具有独占语义：如果接口已经有 root qdisc，内核返回 `-EEXIST`，程序直接退出而不做任何修改。这一安全措施意味着该工具最适合用于你完全控制配置的专用测试接口。 +`bpf_tc_hook_create` 调用具有独占语义：如果接口已经有 root qdisc，内核返回 `-EEXIST`，程序直接退出而不做任何修改。若请求接口的 root qdisc 冲突，加载器会打印 `tc qdisc show dev IFACE` 和恢复命令 `sudo tc qdisc del dev IFACE root`。`struct_ops` 名称冲突是全局的，而不是某个接口局部的，因此加载器会改为打印所有接口的 `tc qdisc show`，不会猜测应该修改哪个接口。程序绝不会自动执行破坏性命令；只有确认显示的 `bpf_pacer` 项确实是残留状态后，才能删除对应的 root qdisc。这一安全措施意味着该工具最适合用于你完全控制配置的专用接口。 挂载成功后，程序打印 `READY`，然后每隔 100ms 检查一次，直到时长到期或收到信号。清理顺序很重要：先调用 `bpf_tc_hook_destroy`（触发 reset 并释放排队的报文），然后读取最终统计，最后销毁 skeleton。 @@ -696,40 +800,27 @@ make clean make -j2 ``` -最简单的测试方式是使用附带的集成测试，它会创建一个临时的 veth 对并在测试后清理。加载 BPF 和修改 qdisc 需要 root 权限： +选择一个你能够控制的接口，并在挂载 pacer 前检查它当前的 root qdisc： ```bash -sudo python3 tests/test_egress_pacer.py ./egress_pacer -``` - -下面是一次成功运行的输出，环境为 x86_64，内核版本 `7.0.0-rc2+`（commit `a03114efd0720dff230388f7e160e427e54ea31b`）： - -```console -READY interface=epac_tx rate_kbps=64 queue_limit=8 duration=2 -SUMMARY enqueued=9 dequeued=9 policy_dropped=31 cleanup_dropped=0 bytes_dequeued=9216 max_qlen=8 -TEST-SUMMARY attempts=40 received=9 send_errors=31 span_ms=1024 observed_kbps=64.0 -PASS: conflict refusal, bounded drops, pacing, accounting, normal/signal cleanup, and SIGKILL recovery succeeded +tc qdisc show dev veth-service ``` -发生了什么：测试向一个 64 Kbit/s、最多容纳 8 个报文的队列发送了 40 个原始以太网帧（每个 1024 字节，EtherType `0x88B5`）。9 个报文成功通过队列，分布在 1024ms 内，与预期速率吻合。其他 31 次发送触发了队列上限，计入 `policy_dropped`。具体数字因系统时序而异。 - -测试还覆盖了边界情况： -- 预先安装一个 `pfifo`，确认加载器拒绝替换现有的 qdisc。 -- 在队列中有报文时发送 SIGTERM，验证 `cleanup_dropped > 0` 且 `enqueued = dequeued + cleanup_dropped`。 -- 发送 SIGKILL（跳过清理），然后用 `tc` 手动删除残留的 qdisc，确认接口恢复正常。 - -手动测试时，选择一个你控制的接口，先查看其当前 qdisc： +确认该接口可用于这次实验后，启动 pacer，让正常应用流量经过它： ```bash -tc qdisc show dev veth-service +sudo ./egress_pacer --interface veth-service --rate-kbps 64000 --queue-limit 256 --duration 30 ``` -如果显示的是默认的 `noqueue` 或 `pfifo_fast`，就可以挂载 pacer： +qdisc 生效后工具才会打印 `READY`。运行时长结束后，输出形式如下： -```bash -sudo ./egress_pacer --interface veth-service --rate-kbps 64000 --queue-limit 256 --duration 30 +```console +READY interface=veth-service rate_kbps=64000 queue_limit=256 duration=30 +SUMMARY enqueued=... dequeued=... policy_dropped=... cleanup_dropped=0 bytes_dequeued=... max_qlen=... ``` +`enqueued` 表示 qdisc 接收的报文数，`dequeued` 表示已经交给设备的报文数，`policy_dropped` 表示队列满时拒绝的报文数。如果进程在正常清理前被强制终止并留下 `bpf_pacer`，下次启动会打印所有接口的 qdisc 状态。找到包含 `bpf_pacer` 的那一行，确认它确实是残留实例，然后再用 `sudo tc qdisc del dev ACTUAL_IFACE root` 删除实际所属接口的 root qdisc。 + 命令行选项： ```text @@ -762,7 +853,7 @@ BPF qdisc 需要 Linux 6.16 或更高版本。内核必须启用 BTF 和 BPF JIT | libbpf | 1.6.0+ | | 权限 | root | -仓库已在 `src/third_party` 中包含 libbpf 和 bpftool。集成测试另外依赖 Python 3、原始套接字能力，以及 iproute2 中的 `ip` 和 `tc`。一对 veth 接口即可满足测试需求；已在 x86_64 上验证。 +仓库已在 `src/third_party` 中包含 libbpf 和 bpftool。使用 iproute2 中的 `tc` 检查目标接口，并在必要时清理残留的 root qdisc。 ## 总结 diff --git a/src/53-egress-pacer/egress_pacer.c b/src/53-egress-pacer/egress_pacer.c index b739fb1d..32f39598 100644 --- a/src/53-egress-pacer/egress_pacer.c +++ b/src/53-egress-pacer/egress_pacer.c @@ -4,17 +4,21 @@ #include #include #include +#include #include #include #include #include #include +#include #include #include #include #include "egress_pacer.h" #include "egress_pacer.skel.h" +extern char **environ; + static volatile sig_atomic_t exiting; static struct env { @@ -195,10 +199,145 @@ static int wait_for_duration(void) return 0; } +static const char *find_tc_binary(void) +{ + static const char *const candidates[] = { + "/usr/sbin/tc", + "/sbin/tc", + }; + size_t index; + + for (index = 0; index < sizeof(candidates) / sizeof(candidates[0]); index++) { + if (!access(candidates[index], X_OK)) + return candidates[index]; + } + return NULL; +} + +static void print_qdisc_state(bool all_interfaces) +{ + const char *tc_binary = find_tc_binary(); + char *const all_arguments[] = { + (char *)"tc", (char *)"qdisc", (char *)"show", NULL, + }; + char *const interface_arguments[] = { + (char *)"tc", (char *)"qdisc", (char *)"show", (char *)"dev", + (char *)env.interface, NULL, + }; + char *const *arguments = all_interfaces ? all_arguments : interface_arguments; + posix_spawn_file_actions_t actions; + pid_t child; + pid_t waited; + int error; + int status; + + fprintf(stderr, "%s:\n", + all_interfaces ? "qdisc state across interfaces" : "current qdisc state"); + if (!tc_binary) { + fprintf(stderr, "tc was not found in /usr/sbin or /sbin\n"); + goto manual; + } + + error = posix_spawn_file_actions_init(&actions); + if (error) + goto spawn_failed; + error = posix_spawn_file_actions_adddup2(&actions, STDERR_FILENO, + STDOUT_FILENO); + if (!error) + error = posix_spawn(&child, tc_binary, &actions, NULL, arguments, + environ); + posix_spawn_file_actions_destroy(&actions); + if (error) + goto spawn_failed; + + do { + waited = waitpid(child, &status, 0); + } while (waited < 0 && errno == EINTR); + if (waited >= 0 && WIFEXITED(status) && !WEXITSTATUS(status)) + return; + if (waited < 0) + error = errno; + else + error = EIO; + +spawn_failed: + fprintf(stderr, "failed to run tc: %s\n", strerror(error)); +manual: + if (all_interfaces) + fprintf(stderr, "run manually: tc qdisc show\n"); + else + fprintf(stderr, "run manually: tc qdisc show dev %s\n", + env.interface); +} + +static int install_pacer(struct egress_pacer_bpf *skel, + struct bpf_tc_hook *hook) +{ + int error; + + error = egress_pacer_bpf__attach(skel); + if (error) { + if (error == -EEXIST) { + fprintf(stderr, + "bpf_pacer is already registered globally; another interface may own it\n"); + print_qdisc_state(true); + fprintf(stderr, + "inspect the bpf_pacer owner above before removing any root qdisc\n"); + } else { + fprintf(stderr, "failed to register bpf_pacer qdisc: %s\n", + strerror(-error)); + } + return error; + } + + error = bpf_tc_hook_create(hook); + if (!error) + return 0; + if (error == -EEXIST) { + fprintf(stderr, "refusing to replace the existing root qdisc on %s\n", + env.interface); + print_qdisc_state(false); + fprintf(stderr, + "if it is stale, recover with: sudo tc qdisc del dev %s root\n", + env.interface); + } else { + fprintf(stderr, "failed to attach bpf_pacer to %s: %s\n", + env.interface, strerror(-error)); + } + return error; +} + +static int cleanup_pacer(struct egress_pacer_bpf *skel, + struct bpf_tc_hook *hook, bool qdisc_created, int error) +{ + struct pacer_stats final_stats = {}; + int cleanup_error; + + if (qdisc_created) { + cleanup_error = bpf_tc_hook_destroy(hook); + if (cleanup_error) { + fprintf(stderr, "failed to remove bpf_pacer from %s: %s\n", + env.interface, strerror(-cleanup_error)); + if (!error) + error = cleanup_error; + } + } + if (skel && skel->bss) + final_stats = skel->bss->stats; + if (qdisc_created) { + printf("SUMMARY enqueued=%llu dequeued=%llu policy_dropped=%llu " + "cleanup_dropped=%llu bytes_dequeued=%llu max_qlen=%llu\n", + final_stats.enqueued, final_stats.dequeued, + final_stats.policy_dropped, final_stats.cleanup_dropped, + final_stats.bytes_dequeued, final_stats.max_qlen); + } + egress_pacer_bpf__destroy(skel); + return error != 0; +} + int main(int argc, char **argv) { struct egress_pacer_bpf *skel = NULL; - struct pacer_stats final_stats = {}; struct bpf_tc_hook hook = { .sz = sizeof(hook), .attach_point = BPF_TC_QDISC, @@ -208,7 +347,6 @@ int main(int argc, char **argv) }; bool qdisc_created = false; unsigned int ifindex; - int cleanup_err; int err; err = parse_args(argc, argv); @@ -246,24 +384,9 @@ int main(int argc, char **argv) goto cleanup; } - err = egress_pacer_bpf__attach(skel); - if (err) { - fprintf(stderr, "failed to register bpf_pacer qdisc: %s\n", - strerror(-err)); - goto cleanup; - } - - err = bpf_tc_hook_create(&hook); - if (err) { - if (err == -EEXIST) - fprintf(stderr, - "refusing to replace the existing root qdisc on %s\n", - env.interface); - else - fprintf(stderr, "failed to attach bpf_pacer to %s: %s\n", - env.interface, strerror(-err)); + err = install_pacer(skel, &hook); + if (err) goto cleanup; - } qdisc_created = true; printf("READY interface=%s rate_kbps=%llu queue_limit=%u duration=%u\n", @@ -273,24 +396,5 @@ int main(int argc, char **argv) err = wait_for_duration(); cleanup: - if (qdisc_created) { - cleanup_err = bpf_tc_hook_destroy(&hook); - if (cleanup_err) { - fprintf(stderr, "failed to remove bpf_pacer from %s: %s\n", - env.interface, strerror(-cleanup_err)); - if (!err) - err = cleanup_err; - } - } - if (skel && skel->bss) - final_stats = skel->bss->stats; - if (qdisc_created) { - printf("SUMMARY enqueued=%llu dequeued=%llu policy_dropped=%llu " - "cleanup_dropped=%llu bytes_dequeued=%llu max_qlen=%llu\n", - final_stats.enqueued, final_stats.dequeued, - final_stats.policy_dropped, final_stats.cleanup_dropped, - final_stats.bytes_dequeued, final_stats.max_qlen); - } - egress_pacer_bpf__destroy(skel); - return err != 0; + return cleanup_pacer(skel, &hook, qdisc_created, err); } diff --git a/src/53-egress-pacer/tests/test_egress_pacer.py b/src/53-egress-pacer/tests/test_egress_pacer.py index 955be8a0..984c974b 100644 --- a/src/53-egress-pacer/tests/test_egress_pacer.py +++ b/src/53-egress-pacer/tests/test_egress_pacer.py @@ -18,6 +18,8 @@ TX_INTERFACE = "epac_tx" RX_INTERFACE = "epac_rx" +ALT_TX_INTERFACE = "epac_alt_tx" +ALT_RX_INTERFACE = "epac_alt_rx" ETHERTYPE = 0x88B5 MAGIC = b"EPAC" FRAME_SIZE = 1024 @@ -41,17 +43,20 @@ def run(*command: str, check: bool = True) -> subprocess.CompletedProcess[str]: ) -def delete_test_link() -> None: - run("ip", "link", "del", TX_INTERFACE, check=False) +def delete_test_link(tx_interface: str = TX_INTERFACE) -> None: + run("ip", "link", "del", tx_interface, check=False) -def create_test_link() -> None: - delete_test_link() - run("ip", "link", "add", TX_INTERFACE, "type", "veth", "peer", "name", RX_INTERFACE) - run("ip", "link", "set", "dev", TX_INTERFACE, "addrgenmode", "none") - run("ip", "link", "set", "dev", RX_INTERFACE, "addrgenmode", "none") - run("ip", "link", "set", "dev", TX_INTERFACE, "up") - run("ip", "link", "set", "dev", RX_INTERFACE, "up") +def create_test_link( + tx_interface: str = TX_INTERFACE, + rx_interface: str = RX_INTERFACE, +) -> None: + delete_test_link(tx_interface) + run("ip", "link", "add", tx_interface, "type", "veth", "peer", "name", rx_interface) + run("ip", "link", "set", "dev", tx_interface, "addrgenmode", "none") + run("ip", "link", "set", "dev", rx_interface, "addrgenmode", "none") + run("ip", "link", "set", "dev", tx_interface, "up") + run("ip", "link", "set", "dev", rx_interface, "up") def wait_until_ready(process: subprocess.Popen[str]) -> list[str]: @@ -162,6 +167,11 @@ def main() -> int: ) assert conflict.returncode != 0, conflict.stdout assert "refusing to replace" in conflict.stdout, conflict.stdout + assert "current qdisc state:" in conflict.stdout, conflict.stdout + assert "pfifo" in conflict.stdout, conflict.stdout + assert ( + f"sudo tc qdisc del dev {TX_INTERFACE} root" in conflict.stdout + ), conflict.stdout run("tc", "qdisc", "del", "dev", TX_INTERFACE, "root") receiver_ready = threading.Event() @@ -263,6 +273,39 @@ def main() -> int: assert process.returncode == -signal.SIGKILL, process.returncode qdisc_after_kill = run("tc", "qdisc", "show", "dev", TX_INTERFACE).stdout assert "bpf_pacer" in qdisc_after_kill, qdisc_after_kill + + create_test_link(ALT_TX_INTERFACE, ALT_RX_INTERFACE) + cross_interface_conflict = run( + binary, + "--interface", ALT_TX_INTERFACE, + "--duration", "1", + check=False, + ) + assert cross_interface_conflict.returncode != 0, cross_interface_conflict.stdout + assert "already registered globally" in cross_interface_conflict.stdout, ( + cross_interface_conflict.stdout + ) + assert "qdisc state across interfaces:" in cross_interface_conflict.stdout, ( + cross_interface_conflict.stdout + ) + assert "bpf_pacer" in cross_interface_conflict.stdout, cross_interface_conflict.stdout + assert TX_INTERFACE in cross_interface_conflict.stdout, cross_interface_conflict.stdout + assert ( + f"sudo tc qdisc del dev {ALT_TX_INTERFACE} root" + not in cross_interface_conflict.stdout + ), cross_interface_conflict.stdout + + crash_conflict = run( + binary, + "--interface", TX_INTERFACE, + "--duration", "1", + check=False, + ) + assert crash_conflict.returncode != 0, crash_conflict.stdout + assert "bpf_pacer" in crash_conflict.stdout, crash_conflict.stdout + assert ( + f"sudo tc qdisc del dev {TX_INTERFACE} root" not in crash_conflict.stdout + ), crash_conflict.stdout run("tc", "qdisc", "del", "dev", TX_INTERFACE, "root") qdisc_after_recovery = run("tc", "qdisc", "show", "dev", TX_INTERFACE).stdout assert "bpf_pacer" not in qdisc_after_recovery, qdisc_after_recovery @@ -291,6 +334,7 @@ def main() -> int: stop.set() if receiver is not None: receiver.join(timeout=2) + delete_test_link(ALT_TX_INTERFACE) delete_test_link() return 0