diff --git a/AGENTS.md b/AGENTS.md index 1c52f7a2..e2c489d5 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -102,7 +102,7 @@ Reviewers should **block on boundary violations** before nitpicking style: wrong ### TorchProbe overhead (do not regress) -Changing overhead **formulas**, **`_close_step_wall` hook order**, or **deferred drain defaults** requires reading **`docs/src/design/overhead-invariants.zh.md`** and updating the tests listed there. +Changing overhead **formulas**, **`_close_step_wall` hook order**, or **deferred drain defaults** requires reading **`docs/src/design/overhead.zh.md#change-invariants`** and updating the tests listed there. | Invariant | Do not | |-----------|--------| diff --git a/docs/Makefile b/docs/Makefile index 0ee5ae2d..15c68e02 100644 --- a/docs/Makefile +++ b/docs/Makefile @@ -24,8 +24,8 @@ install: # Live preview server serve: @echo "Starting live preview server..." - @echo "Open http://127.0.0.1:8000 in your browser" - mkdocs serve --dev-addr 0.0.0.0:8000 + @echo "Open http://127.0.0.1:8001 in your browser" + $(PYTHON) -m mkdocs serve --dev-addr 127.0.0.1:8001 # Build static site build: diff --git a/docs/archive/REVIEW_COMPARISON.md b/docs/archive/REVIEW_COMPARISON.md index 04b5ff2f..f3e7e49e 100644 --- a/docs/archive/REVIEW_COMPARISON.md +++ b/docs/archive/REVIEW_COMPARISON.md @@ -230,7 +230,7 @@ Python 侧最精巧的实现（1,554 行），三个关键设计： #### 1.3.3 文档质量 — 评分：5.0/5.0 - `modularity.md` 以表格追踪架构边界和技术债务，达到工程级别 -- `overhead-invariants.zh.md` 精确定义了开销不变性公式和测试要求 +- `overhead.zh.md#change-invariants` 精确定义了开销不变性公式和测试要求 - `AGENTS.md` 为 AI 协作设计的编码规范 - 13 个 skill 各有 `SKILL.md` 人类文档 - API.md / CHANGELOG.md 维护良好 diff --git a/docs/mkdocs.yml b/docs/mkdocs.yml index 3ad35571..7346236a 100644 --- a/docs/mkdocs.yml +++ b/docs/mkdocs.yml @@ -92,25 +92,19 @@ plugins: Live Debugging: 现场调试 Troubleshooting: 常见问题 Architecture: 架构 - Platform: 平台核心 + Foundations: 基础架构 Collectors & Profiling: 采集与 Profiling - Distributed: 分布式 + Distributed Query & Analysis: 分布式查询与分析 Modularity & Boundaries: 模块化与边界 + Activation & Runtime Control: 启用、注入与运行时控制 Data Layer: 数据层 Extensibility: 扩展机制 - CLI Command Tree: CLI 命令树 - Profiling: 性能分析 - Torch Profiler SQL: Torch Profiler SQL（Kineto 虚拟表） + Profiling & Tracing: 性能分析与 Tracing Overhead: 开销测量 NCCL Profiler: NCCL Profiler - Debugging Engine: 调试引擎 - Training Phases: 训练阶段 - Overview: 概览 - Torchrun Cluster Heartbeat: torchrun 集群心跳 + Distributed Membership: 分布式成员与控制面 Federated Query Engine: 联邦查询引擎 - Hierarchical Fan-out: 分层集群查询 - Cluster with Pulsing: 基于 Pulsing 的集群 - System Architecture (legacy): 系统架构（旧版） + Distributed Profiler: 分布式 Profiler 查询与可视化 Examples: 示例 Training Debugging: 训练调试 Memory Leak: 内存泄漏 @@ -122,6 +116,8 @@ plugins: Skill Format: Skill 格式规范 Versions: 版本兼容性 Contributing: 贡献指南 + Contributing Guide: 贡献指南 + Documentation Style: 文档写作规范 Development Setup: 开发环境 Welcome Contributors: 欢迎参与开发 Welcome: 欢迎参与 @@ -152,26 +148,19 @@ nav: - Troubleshooting: guide/troubleshooting.md - Architecture: - design/index.md - - Platform: + - Foundations: - Modularity & Boundaries: design/modularity.md + - Activation & Runtime Control: design/activation-injection.md - Data Layer: design/data-layer.md - Extensibility: design/extensibility.md - - CLI Command Tree: design/cli.md - Collectors & Profiling: - - Profiling: design/profiling.md - - Torch Profiler SQL: design/torch-profiler-sql.md - - Overhead: design/overhead.md - - Overhead invariants: design/overhead-invariants.md + - Profiling & Tracing: design/profiling.md - NCCL Profiler: design/nccl-profiler.md - - Debugging Engine: design/debugging.md - - Training Phases: design/training-phase.md - - Distributed: - - Overview: design/distributed.md - - Torchrun Cluster Heartbeat: design/torchrun-cluster.md + - Overhead: design/overhead.md + - Distributed Query & Analysis: + - Distributed Membership: design/distributed.md - Federated Query Engine: design/federation.md - - Hierarchical Fan-out: design/hierarchical-fanout.md - - Cluster with Pulsing: design/cluster-pulsing.md - - System Architecture (legacy): design/architecture.md + - Distributed Profiler: design/distributed-profiler.md - Examples: - examples/index.md - Training Debugging: examples/training-debugging.md @@ -185,7 +174,9 @@ nav: - Environment Variables: reference/env-vars.md - Skill Format: reference/skill-format.md - Versions: versions.md - - Contributing: contributing.md + - Contributing: + - Contributing Guide: contributing.md + - Documentation Style: writing.md extra: generator: false diff --git a/docs/src/api-reference.md b/docs/src/api-reference.md index 3cb0a6e9..652e0589 100644 --- a/docs/src/api-reference.md +++ b/docs/src/api-reference.md @@ -179,7 +179,7 @@ configure_backends(["memtable", "logger"]) # terminal + memtable configure_backends(["memtable", "my_sink"]) ``` -### @table (dataclass plugins) +### @table (dataclass plugins) {#table-dataclass-plugins} ```python from dataclasses import dataclass diff --git a/docs/src/api-reference.zh.md b/docs/src/api-reference.zh.md index 5c6a65e8..73e248dd 100644 --- a/docs/src/api-reference.zh.md +++ b/docs/src/api-reference.zh.md @@ -175,7 +175,7 @@ configure_backends(["memtable", "logger"]) # 终端 + memtable configure_backends(["memtable", "my_sink"]) ``` -### @table（dataclass 插件） +### @table（dataclass 插件） {#table-dataclass-plugins} ```python from dataclasses import dataclass diff --git a/docs/src/assets/architecture/probing-activation-paths.svg b/docs/src/assets/architecture/probing-activation-paths.svg new file mode 100644 index 00000000..79498b06 --- /dev/null +++ b/docs/src/assets/architecture/probing-activation-paths.svg @@ -0,0 +1,56 @@ + + + + + 两种执行入口，最终加载同一个 libprobing + .pth 是解释器启动钩子；inject 是已获 ptrace 权限后的动态加载；两者都不是权限提升 + + + 启动时启用：PROBING=1 / 2 + 适合可控制启动命令的训练；Linux / macOS / Windows 都可用 + + CPython site 初始化 + 扫描 site-packages/*.pth + + + probing.pth + 执行一行 import probing_hook + + + probing_hook -> site_hook.run_site_hook() + 跳过 torchrun supervisor / probing CLI / lightweight helper；_RAN 阻止重复执行 + 解析 followed / nested / regex / script / init:file，命中后 import probing + + + Python import probing._core -> 动态加载扩展模块 + 注册 Python 主线程、tracer、ExternalTable 与 Python API + + + 运行时注入：probing -t PID inject + 仅 Linux；目标已运行；调用方必须先通过内核 ptrace 权限检查 + + attach PID 与全部 TID + pete::Ptracer / PTRACE_ATTACH + + + 保存寄存器与可执行区原字节 + 为短跳板、原字节和寄存器恢复做准备 + + + 写入 shellcode，借目标 libc 调用 setenv / malloc / dlopen / free + x86_64：nop+nop+call r9+int3（6 B）；AArch64：nop+nop+blr x8+brk（16 B） + canonical libprobing.so 路径写入目标内存；dlopen(..., RTLD_LAZY) 触发构造函数 + + + 恢复原字节与寄存器 -> detach 全部 TID + 库继续留在目标进程；临时执行跳板不再存在 + + + + + libprobing.so / probing._core + #[ctor] -> start_local -> initialize_engine -> Unix socket；按环境再启动 TCP、采样器与集群心跳 + 之后 CLI 通过 HTTP 查询；训练回调开始把事件写到本机 MEMT + + 选择原则：能控制启动时优先 .pth；只有错过启动窗口且具备 ptrace 权限时才使用 inject。 + diff --git a/docs/src/assets/architecture/probing-cli-control-surface.svg b/docs/src/assets/architecture/probing-cli-control-surface.svg new file mode 100644 index 00000000..e492c3ef --- /dev/null +++ b/docs/src/assets/architecture/probing-cli-control-surface.svg @@ -0,0 +1,55 @@ + + + + + + + + + CLI 是公开控制协议的客户端，不拥有第二套运行时 + 命令只表达意图；状态、查询与诊断能力由目标进程中的 server 和 extension 提供 + + + probing CLI + + + 进程入口inject / launch / list + + 查询入口query / tables / cluster + + 现场诊断eval / repl / backtrace + + 运行时状态memory / config / flamegraph + + Agent 工作流skill / mcp + + + + + 公开传输与契约 + + 本机 Unix socket + + HTTP / WebSocket + 认证、超时、DTO、readiness + + + + 目标进程 + + probing-server + 路由、鉴权、生命周期 + + + Query Engine + SQL、catalog、federation + + + Extensions + Torch / Stack / NCCL / HCCL / System + 每个采集器只拥有自己的状态 + + + 边界：CLI、Web 与 Agent 可以失败或重试，但不能进入采集器内部，也不能阻塞训练回调 + + diff --git a/docs/src/assets/architecture/probing-cluster-heartbeat.svg b/docs/src/assets/architecture/probing-cluster-heartbeat.svg new file mode 100644 index 00000000..cbaaf3c1 --- /dev/null +++ b/docs/src/assets/architecture/probing-cluster-heartbeat.svg @@ -0,0 +1,29 @@ + + + + + Heartbeat 分层收敛：连接数按节点规模增长 + leaf 只连接本机 local0，local0 再向 global rank 0 汇报合并后的成员状态 + + + 节点内 leaf + rank 1 / 2 / 3 ... + PUT /apis/nodes仅自身 endpoint 与健康 + + + + local0 + 按 rank 合并本机 heartbeat保留 dead / stale 状态 + 生成节点成员快照自身也执行本地 SQL + + + + global rank 0 + 合并各节点快照形成 membership epoch + 提供查询路由不集中保存训练数据 + + + 收敛前保持短间隔；稳定后退避；一个 TTL 标 dead，第二个 TTL 移除 + 网络失败只改变成员质量与 coverage，不阻塞训练，也不修改 torch rendezvous + + diff --git a/docs/src/assets/architecture/probing-cluster-membership.svg b/docs/src/assets/architecture/probing-cluster-membership.svg new file mode 100644 index 00000000..bc29005f --- /dev/null +++ b/docs/src/assets/architecture/probing-cluster-membership.svg @@ -0,0 +1,56 @@ + + + + + + + + 分布式成员控制面：分层传播身份，采集数据始终留在 Rank + torchrun 坐标决定成员身份；heartbeat 建立查询路由，不搬运训练遥测 + + + torchrun / elastic environment + WORLD_SIZE / RANK / GROUP_RANK / LOCAL_RANK / role + + + + 节点 A + + local0 / Rank 0 + 本机代理 + 本地 SQL + + leaf ranks + 各自写本地表 + + + 本机成员快照 + endpoint / rank / local rank / role / health + + + 节点 B ... 节点 N + + local0 / proxy + 节点级 heartbeat + + leaf ranks + 各自写本地表 + + + 本机成员快照 + 只汇总身份和健康，不汇总采集数据 + + + + + global rank 0 + job snapshot + + + + CLI / Web / MCP / global.* + 查询发生时才沿成员快照 fan-out + + + 控制面只回答“谁存在、在哪里、是否可查询”；训练事实仍由每个 Rank 本地保存 + + diff --git a/docs/src/assets/architecture/probing-collector-landscape.svg b/docs/src/assets/architecture/probing-collector-landscape.svg new file mode 100644 index 00000000..9d6d5f8d --- /dev/null +++ b/docs/src/assets/architecture/probing-collector-landscape.svg @@ -0,0 +1,57 @@ + + + + + + 采集全景：不同数据源各自处理回调并写入本机表 + 训练、通信、堆栈与系统采样互不调用；step、rank、时间戳是查询阶段的关联坐标 + + + 数据从哪里来 + + PyTorch module / optimizer + NCCL profiler callbacks + HCCL MSProf callbacks + Python / native stacks + GPU · CPU · RDMA + Megatron parallel state + 短窗口 torch.profiler + + + + 每个采集器自己决定 + + 触发hook、callback、signal、timer 或 HTTP + 热路径预算采样、固定池、无阻塞或后台排水 + 事件语义模块、collective、stack 或设备快照 + 失败边界记录该数据源错误；不调用其他采集器 + 固定 schema 的行 + + + + + + + 共同字段与写入合同 + + RowWriter / push_row / Python @table + 训练回调只写当前进程的 mmap；不发网络请求 + + + 本机采集表 + python.torch_trace · nccl.* · hccl.* · stack folded lines + gpu.* · cpu.* · rdma.* · python.profile_capture / profile_hotspot + + + 查询时再关联 + 同一步：local_step / global_step　同一进程：rank / role + 同一时间窗：epoch ns　同一通信：comm / peer / seq + 跨 rank 通过 global.* 和 fan-out 形成一张逻辑表 + + 扩展新数据源：实现公开表合同，注册到组装根 + 不修改已有采集器，也不增加跨采集器回调 + + + 采集器写入事件和指标；跨表关系在查询阶段用 step、rank、时间与角色字段建立。 + + diff --git a/docs/src/assets/architecture/probing-data-query-contract.svg b/docs/src/assets/architecture/probing-data-query-contract.svg new file mode 100644 index 00000000..7c0dac50 --- /dev/null +++ b/docs/src/assets/architecture/probing-data-query-contract.svg @@ -0,0 +1,36 @@ + + + + + + 数据写入与查询读取通过表契约汇合 + 训练热路径只写本地表；CLI、Web 与 Skill 只从服务端读取 + + + 写入路径 + CollectorsTorch / GPU / NCCL + + RowWriterpush_row / mmap + + + 本地表：MEMT / MEMC + 采集器只拥有自己的 schema 与写入状态 + 无网络请求 · 无跨采集器 callback + + + 读取路径 + CLI / Web / SkillPOST /query + + Serverpublic contract + + + Engine / TableProvider + SQL 解析、表扫描、联邦合并 + 返回 DataFrame + QueryQuality + + + TableProvider scan + + 新增指标使用新表；跨信号分析使用 SQL 或 Skill；保留策略由数据层配置 + + diff --git a/docs/src/assets/architecture/probing-distributed-stacks.svg b/docs/src/assets/architecture/probing-distributed-stacks.svg new file mode 100644 index 00000000..8ae09a42 --- /dev/null +++ b/docs/src/assets/architecture/probing-distributed-stacks.svg @@ -0,0 +1,72 @@ + + + + + + Python / native 堆栈：先在每个 rank 折叠，再合并成分布式火焰图 + 同一调用路径跨 rank 累加样本；已返回样本和失败节点分别写入结果与 fan-out 元数据 + + + 单进程采集：两套帧进入同一快照 + + 持续采样 + SIGPROF -> native return address + Python code pointer + 固定 ring 聚合；信号路径只写 POD snapshot + + 按需快照 + SIGUSR2 / sync walk / PYSTACKS + 未启持续采样时，至少可拿主线程 Python 栈 + + 混合栈拼接 + Python：outer -> inner　native：leaf -> root + 以 CPython eval frame 为锚点替换解释器帧 + 去掉解释器启动噪声，最终统一为 root -> leaf + + folded line + [py] train;[c++] nccl;kernel 37 + + + + 集群 fan-out：并发拉取各 rank folded lines + + rank 0local + rank 1HTTP folded/json + rank Nbounded fan-out + + + + 归一化与合并 + 去掉 all / thread-* 前缀，规范符号与路径 + 相同 path：sum(count)，union(ranks) + mode=py 时最后过滤 native / C 帧 + + 部分结果策略 + 默认 deadline 15s；超时即中止剩余任务 + 返回 completed ranks + nodesFailed + partial 标记 + + 重复 rank 按 rank id 去重，避免贡献数膨胀 + + + + 分布式火焰图：宽度是样本，节点带 rank + + all · ranks 0–7 + [py] train_step · 0–7 + loader + [py] forward + sync + [c++] gemm + wait + 点击下钻 · 搜索函数 · hover 查看 count / ranks + + 它回答的不是“哪个栈最多” + 而是：同一热点覆盖全部 rank，还是只在少数 + rank 出现？Python 路径下方是否卡在 native 通信？ + rank 归属随 frame 一起返回给前端 + + + + 两种展示模式 + mixed 保留 Python + C/C++/Rust 调用链，用于定位跨语言阻塞；py 只保留 Python frame，用于比较训练代码路径。两者共享同一份 folded 合并结果。 + + diff --git a/docs/src/assets/architecture/probing-fanout-internals.svg b/docs/src/assets/architecture/probing-fanout-internals.svg new file mode 100644 index 00000000..68a48f0e --- /dev/null +++ b/docs/src/assets/architecture/probing-fanout-internals.svg @@ -0,0 +1,77 @@ + + + + + 分层查询：发起方只连接每台机器的代理 rank + 连接数从 O(rank 数) 变成 O(机器数)；代理 rank 再查询本机其他训练进程 + + + 发起方 / coordinator + + 解析 global.* SQL + 选择先汇总 / 扫描 / 分别执行 + + + 读取 cluster.nodes 元数据 + group_rank / local_rank / addr + 缺失分组信息时返回 503，不静默退化 + + + 并行请求每机 local_rank=0 + peer SQL 一律查询 probe.* + 默认总超时 30 s；有连接与行数限制 + + + 合并 / 再汇总 / 全局 LIMIT + 补来源列并生成 FanoutMeta + + + 训练集群 + + + + + 机器 A · local proxy rank 0 + + rank 0 server + 接收 node scope + + + 本机 fan-out + rank 1 / 2 / 3 + + 合并本机批次与失败信息，再回传 coordinator + local0 自身也执行 probe.*，不会遗漏 rank 0 + + + 机器 B · local proxy rank 4 + + rank 4 server + 接收 node scope + + + 本机 fan-out + rank 5 / 6 / 7 + + 本机结果与 FanoutMeta 一起向上归并 + + + 每一行都在发起方补充固定来源列 + _host · _addr · _rank + _node_rank · _local_rank · _role + 来源列不写入采集表，避免每行重复存储节点元数据。 + peer 返回 DataFrame；发起方负责列对齐、合并与最终关系运算。 + 复杂查询的“分别执行”不是跨 rank JOIN。 + + + 失败语义也分层归并 + 普通模式 + 保留成功 rank 的数据；partial=true；记录失败地址与原因；HTTP 503。 + 严格模式 + 任一 rank 失败即整次查询失败，不返回不完整表。 + 代理 rank 与发起方分别 absorb，将两处失败记录合并进 FanoutMeta。 + + + coordinator scope -> node scope -> local scope；每一层只负责连接下一层、合并数据、传播完整性元数据。 + 自动模式需要完整的分组元数据；这样规模优化是可验证的配置，而不是不可见的运行时猜测。 + diff --git a/docs/src/assets/architecture/probing-feature-placement.svg b/docs/src/assets/architecture/probing-feature-placement.svg new file mode 100644 index 00000000..4833ac02 --- /dev/null +++ b/docs/src/assets/architecture/probing-feature-placement.svg @@ -0,0 +1,50 @@ + + + + + 新能力放在哪里，由它新增的事实与状态决定 + 先判断是否缺少原始信号，再判断缺少分析、展示还是控制契约 + + + 需求是否产生新的原始事实？ + 新的事件、指标、堆栈或设备状态 + + + 是 + + Collector + Table contract + 系统 / GPU / NCCL / HCCL：Rust extension + 训练语义：Python hook + @table + 只发布 schema，不直接调用其他 collector + + + 否 + + 现有事实能否通过 SQL 表达？ + JOIN、聚合、阈值与解释规则 + + + + Diagnostic Skill + steps.yaml + SQL + 多端共享执行语义 + + Query view + 虚拟表 / SQL function + 可归并状态放查询层 + + + + 只缺新的交互？ + Web 使用现有 HTTP / SQL + + 只缺新的控制入口？ + server + proto，CLI 仅作客户端 + + 只缺新的分析流程？ + Skill 组合已有表，不改 server + + + 禁止：collector 互调、Web 链接 core、server 为单个 skill 写分支、CLI 绕过公开协议 + + diff --git a/docs/src/assets/architecture/probing-federation-path-selection.svg b/docs/src/assets/architecture/probing-federation-path-selection.svg new file mode 100644 index 00000000..7ba36c04 --- /dev/null +++ b/docs/src/assets/architecture/probing-federation-path-selection.svg @@ -0,0 +1,38 @@ + + + + + + 联邦 SQL 路径选择 + 基于 AST 保守路由；无法证明可拆分时进入本地广播路径 + + 单条 SELECT？解析失败也按“否”处理 + + 是 + + 否 + + 单表且无 JOIN / CTE / UNION / 子查询？关系运算能否在单表分片上保持语义 + + 是 + + 否 + + 可分解 GROUP BY / 聚合？count / sum / min / max 等可归并状态 + + + 是否 + + 路径 A聚合下推 + 全局归并 + 路径 B联邦 Lazy Scan + 路径 C每个进程本地执行结果按分片拼接不声称完成跨 Rank JOIN + + + 正确性优先 + - avg / count distinct 需可证明归并 + - ORDER BY / LIMIT 在全局合并后执行 + - Filter 只下推精确可翻译谓词 + - Peer 永远执行 probe.* + - 不确定时不做激进优化 + + diff --git a/docs/src/assets/architecture/probing-federation-pipeline.svg b/docs/src/assets/architecture/probing-federation-pipeline.svg new file mode 100644 index 00000000..a3d018c6 --- /dev/null +++ b/docs/src/assets/architecture/probing-federation-pipeline.svg @@ -0,0 +1,31 @@ + + + + + + 联邦查询处理流水线 + cluster=false 直接本地执行；cluster=true 才进入分片计划 + + + 用户 SQLquery request + + Cluster?scope decision + + 路径选择A / B / C + + Catalog 改写global.* -> probe.* + + 分片执行Rank / Node + + 全局合并tags + quality + + + + 否：本地执行 + Local DataFusion / probe.*不进入 fan-out + + + 统一结果：QueryOutcome<DataFrame> + QueryQuality + 来源标签、覆盖率、nodes_failed、partial 与 strict 语义随结果一起返回 + + diff --git a/docs/src/assets/architecture/probing-hccl-collector.svg b/docs/src/assets/architecture/probing-hccl-collector.svg new file mode 100644 index 00000000..4bf6eb53 --- /dev/null +++ b/docs/src/assets/architecture/probing-hccl-collector.svg @@ -0,0 +1,64 @@ + + + + + + HCCL 采集：用 MSProf shim 同时完成旁路记录与原调用转发 + HCCL 仍按原方式 dlsym MSProf；Probing 的 libprofapi.so 只插在边界上解码事件并写表 + + + HCCL runtime + + dlopen + libprofapi.so + + MSProf 上报 + ReportApi + ReportCompactInfo + ReportAdditionalInfo + RegTypeInfo / GetHashId + 其他导出符号 + + + + Probing libprofapi.so shim + + 1. 导出 HCCL 要找的同名 Msprof* 符号 + 调用 ABI 不变；HCCL 无需链接 Probing 内部类型 + + 2. 分类并解码上报负载 + hash 映射 op / task 名；type id 选择 AdditionalInfo 类型 + Api 提供 host timing；CompactInfo 提供 collective 参数 + AdditionalInfo 提供 task / MC2 stream / context id + + 3. mmap 表按需打开并 push_row + 某张表打开失败只停用该表，并只记录一次警告 + + 4. 原参数转发给真实 CANN MSProf + 显式路径避免 shim 再次 dlopen 自己 + + + + 本机 HCCL 采集表 + hccl.host_ops　Host API 时间线与事件分类 + hccl.collectives　op 耗时 + count / group / algorithm + hccl.tasks　local/remote rank、plane、bytes、transport + hccl.mc2_streams　通信 stream 拓扑 + hccl.context_ids　FFTS context 范围 + 通过 global.hccl.* 跨 rank 查询 + + + 真实 CANN libprofapi.so + 解析顺序： + PROBING_HCCL_PROFAPI_REAL + shim 同目录 libprofapi.so.real + ASCEND_HOME / ASCEND_INSTALL_PATH + 真实 profiler 是否启用仍由 CANN / HCCL 配置决定； + shim 不伪造事件，也不改变原函数返回值。 + + + 部署与版本边界 + 把 shim 目录放在 LD_LIBRARY_PATH 前面，并保存匹配当前 CANN 版本的真实 libprofapi.so；MSProf 结构体布局需要按部署版本验证。 + 这一方案复用 HCCL 已有上报面，不修改训练代码，也不要求 HCCL 直接依赖 Probing。 + + diff --git a/docs/src/assets/architecture/probing-hot-cold-overview.svg b/docs/src/assets/architecture/probing-hot-cold-overview.svg new file mode 100644 index 00000000..867610ce --- /dev/null +++ b/docs/src/assets/architecture/probing-hot-cold-overview.svg @@ -0,0 +1,30 @@ + + + + + + 冷热数据层：单一写路径，统一 SQL 视图 + MEMT 提供有界实时窗口，MEMC 提供不可变压缩保留 + + + 写入与整理 + push_rowRowWriter + + MEMT 热层环形 Chunk · 固定容量generation + min/max timestamp + + Compactor / Roller转置 · 压缩 · exactly-once + + MEMC 冷层不可变 SegmentPage 索引 · TTL / 字节预算 + + + 查询 + SQL + 时间谓词projection / filter + + HotColdTable统一剪枝与扫描计划调用方不区分冷热 + + + Arrow Batches按 source/chunk/generation 去重后合并 + + 同一时间谓词同时裁剪热 Chunk 与冷 Segment；半行和损坏不会伪装成空数据 + + diff --git a/docs/src/assets/architecture/probing-megatron-integration.svg b/docs/src/assets/architecture/probing-megatron-integration.svg new file mode 100644 index 00000000..aad7f67c --- /dev/null +++ b/docs/src/assets/architecture/probing-megatron-integration.svg @@ -0,0 +1,66 @@ + + + + + + Megatron 集成：把并行角色与 iteration 对齐到 Probing 的公共坐标 + 集成不替代任何采集器；它只让 Torch、通信、堆栈和系统表知道“当前是哪个并行角色、哪一个训练步” + + + import hook 发现 Megatron + + 是否是 Megatron 作业 + 检测已导入模块或 TP / PP / DP 等环境变量 + PROBING_MEGATRON=auto，可显式开关 + + + megatron.core.parallel_state + 模块加载后调用 init_parallel_state + 并包装 initialize_model_parallel 的返回点 + + + megatron.training.training + 模块加载后调用 init_training + 按需包装 train_step；所有 hook 均 best-effort + + Megatron 不存在或 API 变化时静默降级 + + + + 读取 Megatron 状态，写入两个公共接口 + + probing.set_role(...) + tp = tensor model parallel rank + pp = pipeline model parallel rank + dp / ep / cp = data / expert / context rank + 规范化为 role 字符串，并更新节点注册信息 + + probing.step(...) + iteration 视为 optimizer local_step + 读取 get_num_microbatches；缺失时由 batch 参数估算 + micro_step = iteration x micro_batches + train_step 进入前强制同步一次，避免动态 batch 失配 + PROBING_MEGATRON_STEP_SYNC=auto，可单独关闭 + + + + + 适配器更新公共 step / role 状态 + + + TorchProbemodule / phase / duration + step + role + + Collective / NCCL / HCCL用 rank、时间窗与 step 字段在 SQL 中关联 + + 堆栈与分布式火焰图frame 节点显示贡献 rank 与并行角色 + + Torch Profiler / 系统指标capture、kernel、GPU/CPU 快照使用相同 step + + + 查询可按 tp / pp / dp / ep / cp 分组比较 + + + 为什么只做坐标集成 + Megatron 版本变化集中在少量 getter 和 hook 入口；采集器仍只依赖 probing.step / set_role 公共接口，避免把 Megatron 内部类型扩散到 Torch、通信和查询模块。 + + diff --git a/docs/src/assets/architecture/probing-membership-federation-boundary.svg b/docs/src/assets/architecture/probing-membership-federation-boundary.svg new file mode 100644 index 00000000..3c890c0c --- /dev/null +++ b/docs/src/assets/architecture/probing-membership-federation-boundary.svg @@ -0,0 +1,39 @@ + + + + + 成员发现与联邦查询通过一次不可变快照衔接 + 成员模块不解析 SQL；联邦引擎不发现 torchrun 拓扑 + + + Cluster membership + 持续维护的事实 + endpoint 与可达地址 + rank / node / local rank / role + alive / dead / stale 与 membership epoch + + 输出 cluster.nodes snapshot + 小而稳定的控制面契约 + + + + 查询开始 + 冻结一次 + peer snapshot + 不随 heartbeat 漂移 + + + + Federation + 基于快照执行的决策 + 选择 coordinator / local0 / leaf 路径 + 为每行注入固定来源标签 + 合并 partial 并执行全局关系运算 + + 输出 DataFrame + FanoutMeta + coverage / partial / failed nodes + + + 因果边界：heartbeat 改变下一次查询的成员集合，不改变正在执行查询的 peer 集合 + + diff --git a/docs/src/assets/architecture/probing-memc-format.svg b/docs/src/assets/architecture/probing-memc-format.svg new file mode 100644 index 00000000..98ec838c --- /dev/null +++ b/docs/src/assets/architecture/probing-memc-format.svg @@ -0,0 +1,73 @@ + + + + + MEMC：追加写 block，最后用 footer 把 segment 封口 + 所有 header、block 和 footer entry 都是 64 B；block 按 64 B 对齐；完整块带 xxh32 校验 + + + 物理文件布局 + + SegmentHeader · 64 B · magic MEMC + flags / writer_pid / writer_start / created_unix_ms / footer_off + ts_min / ts_max / page_count / header_xxh；seal 前 footer_off=0 + + MCTB table-definition block + BlockHeader 64 B + table payload：table name、column name、dtype + table_id 只在当前 segment 内有效 + + MCPG page block x N + BlockHeader：row/col count、ts_min/max、payload_len/xxh + 来源身份：source_instance + source_chunk + source_gen + payload：逐列编码；数值 Pco(level 8)，变长 RawVarLen + payload 后补零到下一个 64 B 边界，padding 不参与 checksum + + MCFT footer：一次性写 page directory + footer header：magic / count / entries_len / checksum + 每个 64 B entry 重复 page 的范围、offset、长度与来源身份 + 写完 footer 后回写 SegmentHeader：SEALED、footer_off、时间范围 + 封口后文件不可变；查询可直接用 footer 索引 page + + + 一页数据如何编码与读取 + + MEMT row cursor + 按行解码稳定 chunk + + + ColumnBuilder x cols + row-to-column transpose + + + encode_column + per-column sub-header + bytes + + 列编码 + + Pco = 1 + I8/I16/I32/I64/U*/F* 数值列 + 压缩 level 8；解码恢复 typed vector + 失败返回 InvalidData，不生成空列 + + RawFixed = 0 + 固定宽度原始 little-endian array + 用于无需/无法 Pco 的固定类型 + 长度必须与 row_count x width 一致 + + RawVarLen = 2 + Str / Bytes：[u32 len][bytes] 连续拼接 + 读取时逐个校验长度与 payload 边界 + UTF-8 只在 Str materialize 时验证 + + 读路径与崩溃恢复 + + SEALED segment + 校验 SegmentHeader、footer checksum、page count 和 block range；扫描 MCTB 后按 page directory 读取 + + UNSEALED segment + 从 offset 64 前向扫描 MCTB/MCPG；完整 block checksum 错误立即报错；只有越过 EOF 的未完成尾块可丢弃 + 恢复得到的 page 可读，但 segment 时间范围只有 seal 后才可作为整体剪枝索引 + + + block 先写长度、payload 与 checksum，footer 后写；尾部写入中断时，恢复逻辑扫描到最后一个通过校验的 block。 + diff --git a/docs/src/assets/architecture/probing-memc-lifecycle.svg b/docs/src/assets/architecture/probing-memc-lifecycle.svg new file mode 100644 index 00000000..c5fe1f39 --- /dev/null +++ b/docs/src/assets/architecture/probing-memc-lifecycle.svg @@ -0,0 +1,46 @@ + + + + + MEMC 整理：验证热块身份，按列编码，落盘后再推进水位 + compactor 只处理 sealed chunk；任何中途失败都保留旧水位，下一轮可以重试 + + + MEMT 热环 + Writing跳过 + Sealed chunkinstance · chunk · generationrows · min/max ts + drained_gen每个 source chunk 的最后成功 generation启动时从冷页恢复 + + + 一次整理事务 + 1. selectsealed && gen > watermark记录 generation_before + 2. transposerow cursor to column builders再次检查 generation_after + 3. encodenumeric to Pcovarlen to offsets + bytes + + + 4. append complete page + page header + source identity + encoded columns + payload checksum + 完整写入成功后：drained_gen[chunk] = generation + + + 失败 / generation 变化 + 丢弃临时列页，不更新 watermark + 下轮重新选择 + + segment roll + 达到大小或时间阈值 + footer directory then sealed header + + + MEMC segment + SegmentHeader + MCTB · table schema + MCPG · column pages + MCPG · column pages + Footer · page index + + + + “落盘成功后再推进水位”让整理具备可重试语义；source identity 让重启后仍能识别已处理的热块。 + 压缩、checksum 和 segment 封口都在后台线程完成，不进入训练回调。 + diff --git a/docs/src/assets/architecture/probing-memc-recovery-query.svg b/docs/src/assets/architecture/probing-memc-recovery-query.svg new file mode 100644 index 00000000..b65423bf --- /dev/null +++ b/docs/src/assets/architecture/probing-memc-recovery-query.svg @@ -0,0 +1,55 @@ + + + + + MEMC：恢复、保留和冷热查询共享同一份不可变段 + 恢复只容忍写入中断造成的尾部不完整；完整块损坏必须报错 + + + 打开 segment + Header 指向有效 footer？ + + 是否 + + Sealed + 验证 footer directory验证 range / checksum直接建立 page index + + Unsealed + 从 header 后向前扫描保留完整 MCTB/MCPG忽略不完整最后一块 + + 共同校验 + 完整 block 的 magic / length / checksum 错 -> corruption + 不把损坏解释成“没有数据” + 恢复 source identity -> drained_gen + + + 后台保留策略 + 候选文件只考虑结构可确认的 sealed segment按年龄与大小排序 + + + TTL / 空间预算 + 整段删除，不改写页永不删除 unsealed / unreadable始终保留最新 sealed segment + + + POSIX unlink + 目录不再发现旧段已 mmap 的 reader 继续读完 + + + HotColdTable 查询 + SQL：时间范围 + projection调用方不区分热行与冷页 + + + MEMT hot + chunk min/max 剪枝generation 复核行解码 -> Arrow + + MEMC cold + segment / page min/max只解码 projectioncolumn page -> Arrow + + + 合并 Arrow batches + 按 source_instance / chunk / generation 消除整理窗口交叠 + + + 恢复、淘汰和查询都以 segment 不可变为前提；任何完整损坏都显式暴露。 + 冷热合并隐藏在 TableProvider 内，SQL 只看到一张逻辑表。 + diff --git a/docs/src/assets/architecture/probing-memtable-internals.svg b/docs/src/assets/architecture/probing-memtable-internals.svg new file mode 100644 index 00000000..00aee87b --- /dev/null +++ b/docs/src/assets/architecture/probing-memtable-internals.svg @@ -0,0 +1,76 @@ + + + + + MEMT：固定容量环如何同时服务写者与多个读者 + 单写者发布已编码前缀；读者复核 generation；活跃 lease 阻止 chunk 被复用 + + + + 文件布局 + + 128-byte header · 文件级元数据 + 只读：magic / version / num_cols / num_chunks / offsets + 原子热区：write_chunk / refcount · 与只读区分离 cache line + + column descriptors + 列类型、定长/变长布局、时间戳列 + + reader lease array · 默认 16 slots + PID + process start time + local refcount，区分 PID 重用 + 固定大小 chunks，逻辑顺序为 oldest -> newest + + + chunk 7generation 31sealedmin/max tscolumn bytes + + chunk 0generation 32sealedmin/max tscolumn bytes + + chunk 1generation 33writingused / row_countcolumn bytes + + + + 写入与读取协议 + + 写者：直接编码到当前 chunk + RowWriter 流式写列，不为每行分配中间对象 + 写完 payload 后，以 Release 发布 used / row_count + + + 空间不足：封口并推进环 + 记录 min/max ts -> seal -> write_chunk 前移 + 回收旧 chunk 时 generation + 1；覆盖可计数 + 仍被活跃 reader lease 使用时，不覆盖该快照 + + + 读者：两次检查 generation + 1. 获取进程 lease，Acquire 读取 generation 与 used + 2. 按列读取稳定范围，物化 Arrow RecordBatch + 3. 再读 generation；若变化，丢弃本 chunk 结果 + 4. 查询结束释放 lease；异常进程由 liveness 回收 + 结果：宁可少读一个被回收的 chunk，也不返回 torn rows + + + 从 mmap 快照到 Arrow + + 目录定位 PID / schema / table + RingMmapTable 打开同一张 MEMT 文件 + + + 按时间范围选择 chunk + 用 min_ts / max_ts 剪枝 + 获取 lease，并记录 generation₀ + + + 解码投影列 + 只遍历 Release 已发布的稳定前缀 + 结束前复核 generation₁ + + + Arrow RecordBatch + generation 未变化才返回 + 变化时丢弃该 chunk 的临时结果 + + + 热路径使用预分配空间追加写入；读者通过发布顺序、lease 和 generation 检测并发复用。 + mmap 避免查询前复制整个文件，但实际 SQL 结果仍会物化为 Arrow 批次。 + diff --git a/docs/src/assets/architecture/probing-module-dependencies.svg b/docs/src/assets/architecture/probing-module-dependencies.svg new file mode 100644 index 00000000..f06159ed --- /dev/null +++ b/docs/src/assets/architecture/probing-module-dependencies.svg @@ -0,0 +1,46 @@ + + + + + + + + 模块依赖：体验与控制依赖平台契约，采集器彼此隔离 + 箭头表示允许的运行时或编译期依赖方向 + + + 使用体验 + Web / WASM UI + Diagnostic Skills + Python SDKhooks / integrations + + + 运行时控制 + probing-cli + probing-server + + + 数据采集 + + Python / Torch + CPU / RDMA + GPU + NCCL / HCCL + + + + 平台契约 + + SQL / Federation + MEMT / MEMC + Proto / DTO + Macros + + + + + + + 采集器之间不直连；跨信号关系通过 SQL JOIN，所有组装集中在 server/engine.rs + + diff --git a/docs/src/assets/architecture/probing-nccl-event-model.svg b/docs/src/assets/architecture/probing-nccl-event-model.svg new file mode 100644 index 00000000..1ae63e25 --- /dev/null +++ b/docs/src/assets/architecture/probing-nccl-event-model.svg @@ -0,0 +1,62 @@ + + + + + + NCCL 事件模型：父子引用维持生命周期，时间源按可信度逐级退化 + host enqueue 不是设备执行结束；collective 的真实窗口由 KernelCh 或 ProxyOp 子事件重建 + + + 事件层级与并发边界 + + Collective + comm_hash · seq · op · bytes · host start/stop + + + KernelCh events + GPU globaltimer / channel window + + ProxyOp events + peer · direction · proxy start/stop + + + ProxyStep events + wait reason · bytes · channel · peer + 固定 slot pool 避免回调时分配；句柄携带 shard + slot。 + 按 communicator hash 分片加锁；锁内生成行，锁外批量写 MEMT。 + + + 执行窗口重建与等待归因 + + + kernel_gpuGPU globaltimer + + kernel_chchannel active + + proxymin/max ProxyOp + + enqueuehost launch + + + 从左到右逐级退化；只在更可信时间源缺失时使用下一项 + + + 发送侧 + send_gpu_wait：等本地 GPU 产出 + send_peer_wait：等接收方 credit + + 网络与接收侧 + send_wait / recv_wait：等网络或对端 + recv_flush_wait：等接收刷新完成 + + + 输出表 + nccl.coll_perf：重建窗口、timing_source、算法与带宽 + nccl.proxy_ops：peer、方向、channel 与等待分解 + + + 解释边界 + 等待时间描述“在哪里等待”，不是自动根因。watchdog 使用 try_lock，不允许为了采集而等待通信线程；忙时记录 debug 并跳过本次快照。 + 需要训练 step 时，用统一的 UNIX epoch ns 与 python.comm_collective 时间窗口关联；host API duration 不能替代设备执行窗口。 + + diff --git a/docs/src/assets/architecture/probing-nccl-write-path.svg b/docs/src/assets/architecture/probing-nccl-write-path.svg new file mode 100644 index 00000000..86524974 --- /dev/null +++ b/docs/src/assets/architecture/probing-nccl-write-path.svg @@ -0,0 +1,63 @@ + + + + + + NCCL 回调链：固定状态池承接高频事件，锁外批量写表 + 回调更新固定状态池；父子事件结束后生成 collective、proxy 和等待记录 + + + NCCL plugin v3 / v4 + + Collective start / stop + ProxyOp start / stop + ProxyStep state change + KernelCh / globaltimer + NetPlugin / watchdog + + 回调可能来自通信线程、proxy线程或 watchdog，不能共享大锁。 + + + + 事件状态：按 communicator 分片 + + handle = shard + slot + generation + 固定 slot pool；start 不做堆分配 + stop 校验类型、索引和 generation，拒绝陈旧句柄 + + 父子关系留在同一 shard 内 + Collective 持有 KernelCh 与 ProxyOp 子事件引用 + ProxyOp 累加 ProxyStep 状态停留和字节进度 + 最后一个子事件关闭后，父行才具备完整窗口 + + 不同调用者使用不同锁策略 + 普通回调：只锁对应 shard + watchdog：try_lock；忙则跳过，不等待通信线程 + + 锁内只更新状态并生成 Row + 释放 shard 后再批量 push_row + + + + 生成三类关联表 + + nccl.coll_perf + comm · seq · op · bytes · algorithm · bandwidth + start/end/duration · timing_source + 时间源：kernel_gpu -> kernel_ch -> proxy -> enqueue + + nccl.proxy_ops + peer · direction · channel · protocol · transferred bytes + send_gpu / send_peer / send / recv / flush wait + 回答“等在哪里”，不直接宣布责任方 + + nccl.inflight_ops + profiler_counters + 只读快照；用于 hang 时查看尚未关闭的 collective + 忙、池满、句柄错误都显式计数 + + 与 step 的关联发生在 SQL 时间窗，而非 NCCL 回调里 + + + watchdog 使用 try_lock；状态池忙时跳过本次快照，不等待通信回调。 + + diff --git a/docs/src/assets/architecture/probing-overhead-timing-window.svg b/docs/src/assets/architecture/probing-overhead-timing-window.svg new file mode 100644 index 00000000..12f1e36d --- /dev/null +++ b/docs/src/assets/architecture/probing-overhead-timing-window.svg @@ -0,0 +1,26 @@ + + + + + + TorchProbe Step 计时边界 + 先记录墙钟终点，再回收延迟事件；回收成本不计入刚结束的 Step + + + 上一 Step 结束_mark_step_wall_start + forward / backward / optimizerstep_duration_sec 覆盖的训练计算 + 记录墙钟终点_record_step_timing() + 回收延迟事件_drain_deferred() + 推进周期advance + + + 当前 Step 的计时窗口 + + 计时窗口之外的整理工作 + + + 固定顺序 + _record_step_timing() -> _drain_deferred() -> _advance_step_cycle_for_next() -> _mark_step_wall_start() + 禁止在记录 timing 前 drain，否则前几步的 GPU Event 回收会污染本步墙钟。 + + diff --git a/docs/src/assets/architecture/probing-pth-bootstrap.svg b/docs/src/assets/architecture/probing-pth-bootstrap.svg new file mode 100644 index 00000000..330f3a14 --- /dev/null +++ b/docs/src/assets/architecture/probing-pth-bootstrap.svg @@ -0,0 +1,75 @@ + + + + + .pth 启动链：解释器导入钩子先于训练脚本执行 + wheel 与 develop 的路径略有不同，但都收敛到 probing_hook.py -> site_hook.py + + + CPython startup + + 初始化 site 模块 + 构建 sys.path + + + 扫描 *.pth + 路径行：加入 sys.path + import 行：作为 Python 执行 + + + 安装 wheel + probing.pth: + import probing_hook + + develop 模式 + probing.pth：repo/python 路径 + probing_hook.pth： + insert repo path; import hook + + + + probing_hook.py / site_hook.py + + 先识别不应启动 engine 的进程 + elastic supervisor · probing CLI · nccl/skills/dev_pth helper + + + 解析 PROBING，并保存 PROBING_ORIGINAL + 1/followed：仅当前进程；2/nested：对子进程继续保留 + regex / SCRIPT.py：按 argv 与 __main__.__file__ 匹配 + + + 命中后 import probing + 导入前删除当前 PROBING，避免 import 链再次触发 + nested / regex 在导入后按规则恢复给子进程 + + + 可选执行 init:file；安装 crash handler + 异常写 stderr，不让 site 初始化失败中止训练脚本 + + + + 加载 Rust 扩展并进入训练运行时 + + 动态加载 probing._core / libprobing + root #[ctor]：若 PROBING_CLI_MODE 未设置，立即 start_local() + python crate #[ctor]：panic/crash/signal handler；按 ORIGINAL 置 enabled + + + _core Python module 初始化 + 记录 Python 主线程与 OS TID；初始化栈追踪全局状态 + 注册 ExternalTable、query、tracer、CLI、skill bindings + + + probing/__init__.py 组装 Python API + 导出 query / span / step / @table；安装 import hooks + 按已加载 framework 触发 Torch / Megatron / vLLM 接入 + + + 随后才执行用户 train.py + 此时本机 server、表目录和框架 import hook 已就绪 + + + .pth 获得的是“解释器启动时的代码执行入口”，权限与训练进程完全相同；它不会绕过操作系统权限。 + 失败策略是 best-effort：hook 异常输出到 stderr，避免因为诊断组件无法加载而阻止业务脚本启动。 + diff --git a/docs/src/assets/architecture/probing-ptrace-injection.svg b/docs/src/assets/architecture/probing-ptrace-injection.svg new file mode 100644 index 00000000..f0c67ffb --- /dev/null +++ b/docs/src/assets/architecture/probing-ptrace-injection.svg @@ -0,0 +1,62 @@ + + + + + ptrace 注入：用一个可回滚的函数调用跳板执行 dlopen + tracer 控制寄存器和内存；shellcode 只负责间接 call 与 trap，不包含库加载逻辑 + + + 0. 前置检查与停止世界 + CLI 从 /proc/self/exe 定位同目录 libprobing.so；目标 maps 中需有 Python；ptrace attach 主 PID，再枚举 /proc/PID/task attach 全部 TID。 + 权限由内核决定：同 UID、Yama ptrace_scope、父子关系或 CAP_SYS_PTRACE；attach 失败即终止，不存在降级绕过。 + + + 1. 准备可回滚跳板 + + 选择写入地址 + 读取 /proc/PID/maps，取第一个带 EXECUTE 的 mapping 起始地址 + + + 保存现场 + read_memory 保存原代码字节；registers() 保存完整通用寄存器 + + + 写入架构相关 shellcode + x86_64 6 B / AArch64 16 B + + + 2. 解析并调用目标 libc + + ASLR 地址换算 + 在 injector 进程 dlsym malloc/free/setenv/dlopen + symbol - local libc base + target libc base；libdl 单独换基址 + + + 每次函数调用都由 tracer 设置 ABI 寄存器 + x86_64：RIP=stub+2，R9=fn，RDI/RSI/RDX=args + RSP 对齐 16 B；返回值从 RAX 读取；INT3 -> SIGTRAP + AArch64：X8=fn，X0/X1/X2=args；BLR X8；BRK trap + 调用序列：setenv* -> malloc(path) -> write_memory(path) + then dlopen(path, RTLD_LAZY) -> free(path) + + + 3. trap、校验与恢复 + + 等待预期 trap + restart Continue -> wait；只接受 SIGTRAP/BRK 对应 stop + 其他 signal、tracee 退出或 dlopen 返回 NULL -> 显式失败 + + + 恢复目标现场 + write_memory 原字节 -> set_registers 原寄存器 + Drop 兜底再次尝试 remove，避免错误路径留下跳板 + + + Injector Drop：逐个 PTRACE_DETACH 全部已附着 TID + + + 成立条件与风险边界 + • injector 与 target 的 libc/libdl 符号布局必须兼容；地址通过相同 symbol offset 换算。 • 所有线程在 attach 窗口内暂停，因此注入不应成为频繁操作。 + • 恢复的是被覆盖的代码和选中 tracee 寄存器；dlopen 的库与其构造函数副作用会保留。 • inject 仅 Linux，支持 x86_64/AArch64。 + + diff --git a/docs/src/assets/architecture/probing-query-federation.svg b/docs/src/assets/architecture/probing-query-federation.svg new file mode 100644 index 00000000..2cb957e4 --- /dev/null +++ b/docs/src/assets/architecture/probing-query-federation.svg @@ -0,0 +1,77 @@ + + + + + + 联邦查询：global.* 把各进程的 probe.* 合成一张逻辑表 + 查询按 AST 判断可下推范围；远端不再 fan-out，协调端负责来源标注、全局排序和失败语义 + + + 协调端：理解 SQL 并选择执行方式 + + 用户 SQL + python.t + cluster=true -> global.python.t + + + 按查询结构选择 + 单表可分解聚合 -> 每个节点先 GROUP BY + 单表明细 -> global table lazy scan + JOIN / CTE / UNION -> 每个进程本地执行后拼接 + + + 生成远端 probe.* SQL + 只下推能证明等价的投影、谓词和聚合 + 解析失败或不可合并时走保守执行 + + Peer 永远执行 probe.*，防止递归联邦 + + + + 分层 fan-out：协调端只连接每台机器的 local0 + + coordinator + global.* + + + host A local0node scope + + host B local0node scope + + + + rank 0 + rank 1 + rank 2 + rank 3 + + + 每层只做三件事 + 连接下一层 · 合并成功分片 · 传播 nodes_failed + 连接数随机器数增长，而不是由 coordinator 直连全部 rank + + + + 协调端合并：数据与来源一起返回 + + 为每个分片注入六列 + _host · _addr · _rank + _node_rank · _local_rank · _role + 标签描述 endpoint，不写回本地 MEMT + + 完成全局关系运算 + 合并 partial aggregate · 再 GROUP BY 标签 + ORDER BY / LIMIT 在 merge 后执行 + 协调端在合并后执行全局 top-K + + 显式报告完整性 + 默认：成功分片继续合并，partial=true + strict：任一节点失败则整查失败 + 集群 API 对 partial 返回 503 + 可用 dataframe + + 结果 = DataFrame + FanoutMeta + + + 语义边界 + 跨表 JOIN 必须先在同一进程的 probe.* 内完成，再把结果跨 rank 拼接；不支持把两张 global 表直接做跨节点 JOIN。 + + diff --git a/docs/src/assets/architecture/probing-runtime-control.svg b/docs/src/assets/architecture/probing-runtime-control.svg new file mode 100644 index 00000000..8f4a04c8 --- /dev/null +++ b/docs/src/assets/architecture/probing-runtime-control.svg @@ -0,0 +1,86 @@ + + + + + + + + + + 运行时控制：动态库构造函数先启动入口，再异步准备 Engine + .pth import 与 ptrace dlopen 在这里汇合；监听状态、查询就绪和训练采集分别记录 + + + + `.pth` import probing._core + Python module 额外注册 bindings + + ptrace dlopen libprobing.so + 仅触发动态库构造函数 + + + + root #[ctor] setup() + 若不是 CLI mode：初始化日志与 panic hook，调用 probing_server::start_local() + + + + + 1. 控制入口先建立 + + 本机 Unix socket + 按目标 PID 连接 · same UID + + 跨机 TCP + token · 网络隔离 · heartbeat + + listener 已存在，不等于查询已经 ready + /health 说明进程可达；/ready 读取 Engine 状态并给出原因 + 初始化失败时入口仍保留，便于外部观察和重试 + + + 2. Engine 在后台完成组装 + + EngineInitClaim + Claimed / InProgress + + + 注册查询能力 + 表 · 扩展 · peer transport + + + Ready + 查询可以进入 Engine + + 成功后才启动后台工作 + cold compactor · CPU/GPU sampler · cluster heartbeat + 构建错误进入 failure state，不伪造空 Engine + + + + + + 3. 外部控制请求只通过公开接口进入 + + CLI · Web · MCP + 选择 PID / endpoint + + + middleware + 鉴权 · 超时 · 大小 · 连接数 + + + /query + SELECT -> Engine · SET -> config + + /apis + 状态 · 扩展 · profiling + + /mcp + query · skills · cluster tools + + + 训练 hook -> MEMT 的写入路径不经过 listener 或远端；控制面失败、超时或断连都不能反向阻塞训练回调。 + 响应保留 not ready、partial metadata 与 cause chain；cleanup dtor 停止运行时并清理 socket。 + + diff --git a/docs/src/assets/architecture/probing-sampling-overhead.svg b/docs/src/assets/architecture/probing-sampling-overhead.svg new file mode 100644 index 00000000..3dcbd8f8 --- /dev/null +++ b/docs/src/assets/architecture/probing-sampling-overhead.svg @@ -0,0 +1,76 @@ + + + + + TorchProbe 使用采样、延后读取和 shadow 估算自身开销 + step 采样控制频率，layer 采样控制单次覆盖面，shadow step 提供同作业基线，deferred drain 移出计时窗口 + + + 默认 20-step 采样周期（rate=0.05）与 4:1 shadow 周期叠加 + + 0sampled + 1dispatch + 2dispatch + 3dispatch + 4shadow + 5dispatch + 6dispatch + 7dispatch + 8dispatch + 9shadow + ... + 19shadow + 20sampled + + sampled：module hook + event + dispatch：只经过快速 gate + shadow：module/optimizer hook 立即 return + + + step 与 layer 两级采样 + + step gate + period=max(1, round(1/rate)) + sampled = (cycle % period == 0) + + + layer gate + stable_hash("hook", cycle, module_name) < layer_rate + 不读取 host RNG；各 rank 相同；每层随 step 变化 + offset=0 anchor 必采；命中的 pre 必有 matching post + 首个完整 step 只发现模块，不写 trace；后续 step 按 gate 规则采样。 + + + GPU event 延后读取 + + 采样步：只 record start / end CUDA event + 不在 live stream 上 synchronize + + + 后续 step：settle >= 3 后非阻塞检查 + ready 才进入 drain；lag >= 16 时 force complete，限制内存 + 先记录本步 wall time，再 drain，避免污染估计窗口 + + + bounded queue 4096，daemon save；满则同步回退 + + + 同作业估计与可选自适应 + + rolling window = 80 steps + dispatch = non-shadow && !sampled + baseline = shadow；都用 median step_duration + shadow_n >= 5 且 dispatch_n >= 16 才认为稳定 + + + adaptive rate controller（默认关闭） + dispatch overhead > high(10%)：rate x 0.5 + 低于 target(5%) 且曾降级：rate x 1.25 + 范围：[rate_floor 1%, initial rate 5%] + shadow 仍包含 NCCL/CPU/GPU 采集器，因此只估算 TorchProbe hook 成本。 + + + 摊销模型 + dispatch_overhead = median(dispatch) / median(shadow) - 1；sampled_overhead 同理 + effective = (1 - rate) x dispatch_overhead + rate x sampled_overhead；不能用 mean(probed) / mean(shadow) 替代 + diff --git a/docs/src/assets/architecture/probing-skill-multiclient-runtime.svg b/docs/src/assets/architecture/probing-skill-multiclient-runtime.svg new file mode 100644 index 00000000..3ba04d30 --- /dev/null +++ b/docs/src/assets/architecture/probing-skill-multiclient-runtime.svg @@ -0,0 +1,84 @@ + + + + + + CLI、MCP 和 Web 复用 probing-skills，并保留各自入口适配 + CLI/MCP 调用 execute_skill；Web 复用 run_step 与解释函数，并增加会话取消和页面导航 + + 入口 + + CLI + probing skill run + 本机或 --global + + MCP + plan_skill / run_skill + 服务端进程内执行 + + Web + Investigate / 页面推荐 + runner 编译进 WASM + + Python + 只负责发现与计划 + 不执行步骤 + + + + + + + + probing-skills · Rust 共享运行时 + + 发现 / 加载 / 严格校验 + catalog + SKILL.md + steps.yaml + + + 路由 / 计划 / 参数绑定 + 根据页面、意图、表可用性选择 Skill + + + 顺序执行步骤 + 共享处理 when / platform / on_empty / partial + + + 规则解释与摘要 + outcomes -> findings -> summary / next steps + + 输出：outcomes、findings、覆盖率、错误与跳过原因 + + + + + SkillBackend：把环境差异收在适配器 + + CliBackend + 经 ProbeEndpoint 调 HTTP + local / cluster query + + ServerBackend + MCP 进程内调用查询与 fan-out + 避免 HTTP 回环 + + WebBackend + WASM 经 ApiClient 调 HTTP + UI 步骤可以导航页面 + + 共同接口 + query_local · cluster_query + get · peer_count + + + 共同的数据与控制能力 + 本地查询 · 联邦查询 · 扩展 GET · 节点数量 + 返回 DataFrame 与 cluster_meta + + 新增入口只实现 Backend，不复制 Skill 语义 + + + LLM 的位置 + Web Investigate 可用 LLM 选择 Skill、填写参数和总结结果；步骤执行、查询范围、失败语义和证据保留仍由共享 runner 决定。 + + diff --git a/docs/src/assets/architecture/probing-span-persistence.svg b/docs/src/assets/architecture/probing-span-persistence.svg new file mode 100644 index 00000000..5d90d391 --- /dev/null +++ b/docs/src/assets/architecture/probing-span-persistence.svg @@ -0,0 +1,38 @@ + + + + + Span 状态与持久化分离：栈拥有语义，Recorder 只负责输出 + 阶段状态不从存储反推；backend 失败也不能破坏线程内 span 栈 + + + 用户入口 + probing.span + probing.event + record_span + + + + 状态所有者 + + Rust thread-local Span stack + 嵌套、phase、parent 关系 + + _RecordedSpan 编排提交 + 无 event：退出时提交闭区间 + + + + 输出边界 + SpanRecorder + + MEMT + logger + OTEL + python.trace_event: start / end / event + + + 提交决策：没有 event 时只写一次闭区间；首次 event 才写 start，退出再写 end + 结果：降低热路径写放大；代价：进行中的无 event span 暂不可由 SQL 看见 + + diff --git a/docs/src/assets/architecture/probing-top-down-overview.svg b/docs/src/assets/architecture/probing-top-down-overview.svg new file mode 100644 index 00000000..7c30b4a1 --- /dev/null +++ b/docs/src/assets/architecture/probing-top-down-overview.svg @@ -0,0 +1,92 @@ + + + + + + + + 入口、服务端、查询引擎、采集与本地存储 + 训练回调追加本机表；服务端和跨机查询通过 HTTP/proto 与 SQL 读取 + + + + 使用入口 + 提问与展示 + CLI / Web + + + 服务端 + 接收与协调 + 路由 / 鉴权 + + + 查询引擎 + SQL 与汇总 + 本机 / 跨 rank + + + 数据采集 + 训练 / 系统 + 各自写表 + + + 本地存储 + mmap 文件 + + + + 调用入口：CLI、Web、Skills 和 MCP 使用公开接口 + + CLIinject / query / skill run + WebExplore / Investigate + SkillsSKILL.md + steps.yaml + MCPquery / plan / run_skill + Python hooks / @table扩展数据与训练侧接入 + + + + + 服务端：接收请求、检查权限、组织跨机查询 + + 连接与访问检查本地同用户 / 跨机 token / 资源限制 + 请求分发/query · /apis · /mcp · /config + 模块注册与就绪状态server/engine.rs + 跨机查询协调心跳 · 每机代理 · 分层查询 + + + + + 查询引擎：采集模块之间只通过 SQL 关联 + + SQL 引擎（DataFusion）解析 · 优化 · 执行 + 表目录probe.* 本机 / global.* 集群 + 选择跨 rank 执行方式先汇总 / 按行读取 / 分别执行 + 公共接口与返回格式数据表注册 · proto 结果 · 表说明 + + + + + 数据采集：各自决定采什么，不互相调用 + + Python / Torch + NCCL Profiler + GPU + CPU / RDMA / Process + User @table + + + + + 本地存储 + 本地环形存储（MEMT）mmap · 固定容量 · 单写者 + 可选冷数据（MEMC）压缩只读段 · 默认关闭整理 + 每 PID 本地文件训练退出不依赖远端 control plane + + + + + 请求 ↓ + + + 查询结果 ↑ + diff --git a/docs/src/assets/architecture/probing-torch-profiler-integration.svg b/docs/src/assets/architecture/probing-torch-profiler-integration.svg new file mode 100644 index 00000000..78ec5d8c --- /dev/null +++ b/docs/src/assets/architecture/probing-torch-profiler-integration.svg @@ -0,0 +1,75 @@ + + + + + + Torch Profiler：短窗口结束后生成 capture 与 hotspot 表 + Kineto timeline 保留给交互浏览；SQL 默认面对 capture 与热点聚合，而不是 traceEvents 的逐事件镜像 + + + 按需控制 + + HTTP / UI / skill + start(steps, trigger) + + + ProfilerController + 单实例锁防止重复 capture + 目标步数、trigger、状态 + stop 可提前结束 + + + optimizer post hook + 首个 step 进入 profiler + 每步 profiler.step() + 达到 N 步自动 finalize + + CPU + 可用时 CUDA + + + + torch.profiler / Kineto 窗口 + + 采集选项 + record_shapes · with_stack · with_flops + 短窗口内记录 op、kernel、runtime、memcpy + + + finalize + profiler.__exit__ then key_averages() + 失败时退化到 profiler.events() + 超出 MAX_EVENTS 截断并保留 truncated 标记 + + + Adaptor：从诊断问题反推 bucket + kernel · cpu_op · cuda_runtime · memcpy + collective · other；按 kind+name 聚合 self/wall/calls + 补 capture_id、step、rank、role 与数据质量 + + + + + 两张 SQL 表 + + python.profile_capture：一次窗口一行 + capture_id · step · rank · trigger · wall_us · status + truncated · event_count · error 是结论前必须检查的数据质量 + + python.profile_hotspot：每个时间桶一行 + bucket_kind · bucket_name · self_us · wall_us · calls + pct_of_capture · local/global step · rank + global.python.profile_hotspot 支持同 kernel 跨 rank 比较 + + + + Kineto timeline 通过 HTTP / Web 读取 + export_chrome_trace 生成 traceEvents JSON，交给时间线组件 + 用途：查看单次 capture 的细粒度顺序、重叠与调用栈 + SQL 用于跨 rank 排序、分组与 JOIN；timeline 用于单窗口下钻。 + + + + 与 TorchProbe 独立运行 + 不复用 TorchProbe 的 hook、采样率、shadow 基线或 mmap 表；两者同时开启时开销相加，只通过 local_step / global_step / capture_id 在查询层关联。 + + diff --git a/docs/src/assets/architecture/probing-torchprobe-state.svg b/docs/src/assets/architecture/probing-torchprobe-state.svg new file mode 100644 index 00000000..e32724d2 --- /dev/null +++ b/docs/src/assets/architecture/probing-torchprobe-state.svg @@ -0,0 +1,62 @@ + + + + + + TorchProbe 状态机：step 坐标驱动 hook，异步队列负责慢操作 + 主线程拥有 cycle 与 step snapshot；autograd worker 只消费快照，避免 backward 在线程局部状态中丢失 step + + + 训练主线程 + optimizer/forward 识别 step 边界 -> 保存 snapshot -> 关闭 step 时严格按固定顺序推进 cycle + + + + Discovery + 第一次完整 forward只登记模块，不写性能行 + + Dispatch + hook 进入快速 gate不创建 CUDA event + + Shadow + 模块/optimizer hook 立即返回仍记录 step wall time + + Sampled + step gate + layer gate 命中创建成对事件与 metadata + + Close step + record timingdrain -> cycle++ + + + + 下一 cycle 重新按 shadow 周期与 sample period 分类 + + + Autograd worker：只消费主线程快照 + + forward pre保存 sampled module + + forward post封口同一对事件 + + backward仅跟踪已采样 forward + + snapshot 包含 global_step、micro_step、并行角色和 cycle；动态控制流只产生实际执行模块的记录。 + offset=0 的首个 pre 是强制锚点；所有命中的 pre 必须有 matching post。 + + + Deferred queue：把 CUDA elapsed 读取移出采样步 + + enqueueevent + step/module + + settle / query3+ step 非阻塞检查 + + save MEMTready / lag 16+ + + 队列默认 4096；满时同步保存而不是丢样本；退出时 atexit flush。 + 关键顺序：record_step_timing -> drain_deferred -> advance_cycle -> mark_next_start。 + + + 状态归属 + cycle、采样计划与 rolling window 由 TorchProbe 拥有；CUDA event 只经 bounded queue 传给保存线程；训练 hook 永远不发网络请求。 + + diff --git a/docs/src/assets/architecture/probing-web-spa-architecture.svg b/docs/src/assets/architecture/probing-web-spa-architecture.svg new file mode 100644 index 00000000..79d10c2d --- /dev/null +++ b/docs/src/assets/architecture/probing-web-spa-architecture.svg @@ -0,0 +1,85 @@ + + + + + + + + + Rust 单页面应用：一次加载应用壳，路由只切换工作区 + Dioxus 组件编译为 WASM；浏览器通过公开 HTTP 契约访问 Probing，不链接查询内核 + + + Rust 源码 + + Dioxus 0.7 组件树 + NextShell · 页面组件 · 可视化 + 声明式 UI 与组件生命周期 + + dioxus-router + History 路由 · Outlet · 不整页刷新 + + GlobalSignal 与异步任务 + 调查上下文 · 页面证据 · overlays + 可取消任务 · 可见时轮询 + + ApiClient + 公共类型 + reqwest · probing-proto + query · cluster · skills · extension API + + cargo / dx bundle --release + + + 编译 + + + 浏览器中的单页面应用 + + 一个 HTML 应用壳 + WASM + JS glue + Tailwind CSS + + 侧栏 + 页面入口 + 当前状态 + 全局操作 + + 路由工作区 Outlet + + Dashboard / Training / Cluster + 切换路由，只替换这里 + + Stacks / Traces / Profiles + 复用同一上下文与组件 + + Investigate / Skills + 全页或右侧面板 + + 常驻能力 + URL 同步 · 命令面板 · overlays · task runtime + + + HTTP + + + Probing 服务端 + + 内嵌静态资源 + 预压缩 wasm / js / css 随服务端发布 + + 公开 HTTP 接口 + /query · /apis/cluster/query · /skills/* + + + 查询与诊断能力 + DataFusion 查询 · 联邦查询 + Extension API · Skill discovery + DataFrame + 覆盖率 + partial 元数据 + + 边界保持简单 + 前端只消费公开契约 + 升级 UI 不改变采集与查询写入路径 + + + 构建链：Rust / Dioxus -> dx bundle -> WASM + 静态资源 -> 嵌入 probing._core -> 浏览器一次加载并持续路由 + + diff --git a/docs/src/assets/architecture/probing-wheel-composition.svg b/docs/src/assets/architecture/probing-wheel-composition.svg new file mode 100644 index 00000000..eb11227c --- /dev/null +++ b/docs/src/assets/architecture/probing-wheel-composition.svg @@ -0,0 +1,32 @@ + + + + + Wheel 只有一个 Native 产物，CLI 入口在组装根连接 + 这是打包期组合，不是采集器在运行时反向依赖 CLI + + + pip install probing + maturin wheel + + + + probing._core.so + core + server + Python ext + CLI adaptercli_main only + + + + Python console script + probing.cli.__main__ + _core.cli_main() + + + 允许的耦合 + 根 src/lib.rs 负责组装并导出 cli_main；独立 Rust binary 仍可使用同一 CLI library。 + 禁止的扩散 + collector、server 与 Python extension 不得引用 CLI 内部命令实现。 + + diff --git a/docs/src/assets/profiler-cross-rank-visuals.svg b/docs/src/assets/profiler-cross-rank-visuals.svg new file mode 100644 index 00000000..5f3da254 --- /dev/null +++ b/docs/src/assets/profiler-cross-rank-visuals.svg @@ -0,0 +1,112 @@ + + Cross-rank profiler visualization system + Six coordinated views reveal time, rank, operation, collective, call-path, and topology patterns while sharing one time and rank selection context. + + + + + + + + + Cross-Rank Views Are Different Projections of One Query Context + Select a rank set or time window in any view; every other view recomputes against the same evidence. + + + A. Rank x Time Heatmap + Find when and where rank behavior diverges + + + + + + + + + + + + + + + + + + + + + + B. Quantile-Band Timeline + Expose P50 / P95 / max separation over time + + + + + + + MAXP95P50 + + + + C. Collective Waterfall + Separate late arrival from slow transfer + + + + + + entryslow transfer + + + + D. Behavior Groups + Compress 10K tracks into representative patterns + + 9,600main pattern + 320slow group + 80 + + + + + E. Differential Flamegraph + Show call paths added by the slow cohort + + + + + + + training_step + slow cohort +reference + + + + + F. Topology Projection + Reveal node, PCIe, NIC, and rail concentration + + + + node A / rail 0node B / rail 1 + + + + + + + Shared ViewContext + capture group · alignment · time window · rank-set token · subject/reference cohorts + Every click becomes a new bounded distributed query and a replayable evidence token. + + + + diff --git a/docs/src/assets/profiler-distributed-query.svg b/docs/src/assets/profiler-distributed-query.svg new file mode 100644 index 00000000..1d8465ea --- /dev/null +++ b/docs/src/assets/profiler-distributed-query.svg @@ -0,0 +1,98 @@ + + Distributed profiler query architecture + A typed query is compiled into coordinator, node, and rank plans. All ranks participate, while bounded multi-resolution results drive timeline, flamegraph, and analysis views. + + + + + + + + + 10K-Rank Profiler Query: Compute Everywhere, Return What Matters + All target ranks participate; data movement is bounded by the question, viewport, and query budget. + + + TimelineQuery + scope + alignment + rank selector + tracks + event kinds + reductions + viewport + max rows + output kind + + + Query Coordinator + validate global semantics + compile explicit 3-stage plan + apply budget, stream, and cancel + + + Bounded Result + tiles / sketches / top-k / trees + rank-set and evidence tokens + coverage + error + failed parts + + + typed request + + Arrow stream + + + Hierarchical distributed execution + + + CoordinatorPlan + merge node states + global quantiles and top-k + behavior groups and outliers + visual tiles and receipt + + global mergeable states + + + NodePlan x N nodes + merge local-rank partials + node / role distributions + GPU + NIC + topology join + compress bitmap and sketches + + connections scale with node count + + + RankPlan x 10,000 ranks + filter capture, step, track, event + merge time intervals locally + build tiles, stacks, summaries + read exact slices only on drill-down + + raw profiler events stay near the source + + + partial states + + node states + + + + + timeline.* + rank-time tiles, quantile bands, exact slices + same coordinates from overview to raw event + + + flamegraph.* + aggregate, differential, variance, coverage + path distribution across cohorts and ranks + + + analysis.* + comparison, breakdown, findings, evidence + answers retain rank and time drill-down tokens + diff --git a/docs/src/assets/profiler-timeline-execution.svg b/docs/src/assets/profiler-timeline-execution.svg new file mode 100644 index 00000000..e84edf8b --- /dev/null +++ b/docs/src/assets/profiler-timeline-execution.svg @@ -0,0 +1,97 @@ + + Distributed timeline query execution + An explicit three-stage plan pushes filtering and aggregation to ranks, merges partial state by node, then calculates global visualization data and a query receipt. + + + + + + + + + Explicit Execution Plan for Global Timeline Semantics + No implicit “run locally, then concatenate” fallback for aggregate, join, or alignment. + + + RankPlan + runs on every target trace partition + + 1. Predicate pushdown + capture / step / time / track / kind + + 2. Local interval algebra + union / overlap / exposed time + + 3. Build partial state + tiles / sketches / top-k / stacks + + 4. Exact data gate + only bounded ranks and time windows + + rows_scanned, coverage + partial_state, error + + + NodePlan + one local coordinator per machine + + Merge local-rank state + quantile sketch + top-k + bitmap union + interval occupancy by node / role + + Join node-local context + GPU / NIC / PCIe / NUMA counters + topology labels and clock quality + + Bound and compress output + rank bitmap token + bounded candidates + Arrow RecordBatch partial stream + + ranks_seen, node_state + bytes_scanned, partial + + + CoordinatorPlan + global semantics and result contract + + Merge global state + P50 / P95 / P99, global top-k + rank coverage and membership epoch + + Compute global structure + behavior groups, outliers, alignment + cohort comparison and evidence tokens + + Render-ready outputs + timeline tiles / flamegraph tree + waterfall / heatmap / findings + + exact, error_bound, failed_parts + elapsed_ms, query_id + + + rank states + + node states + + + Runtime contract + Streaming first result + Backpressure + Viewport cancellation + Deadline and budget + Partial-result policy + Replayable receipt + + + cancel propagates to unfinished partitions + diff --git a/docs/src/assets/profiler-timeline-pyramid.svg b/docs/src/assets/profiler-timeline-pyramid.svg new file mode 100644 index 00000000..b36325f8 --- /dev/null +++ b/docs/src/assets/profiler-timeline-pyramid.svg @@ -0,0 +1,78 @@ + + Multi-resolution timeline pyramid + The same distributed timeline can be explored from a ten-thousand-rank overview through node and behavior groups to selected ranks and exact events. + + + + + + + + + One Timeline, Four Resolutions + Resolution changes along both time and rank axes; selection and alignment remain stable. + + + 1. Job overview — all 10,000 ranks participate + Rows: all ranks / nodes Time: coarse buckets Values: P50 / P95 / max / outlier count + + + + + + + + 2. Node / role / behavior groups + Rows: selected groups Time: medium buckets Result identifies spatial and behavioral patterns + + + + + + + + + + + 3. Selected ranks — compare candidates with peers + Rows: a bounded rank set Time: fine tiles or slices Flows and counters become visible + + + + + + + + + + + + 4. Exact trace — a few ranks and a narrow window + Raw slices + flows + counters; export a selected Perfetto trace when needed + + + + + + + + + Zoom + narrow time + expand ranks + + + Stable context + alignment + rank-set token + time-window token + diff --git a/docs/src/design/activation-injection.md b/docs/src/design/activation-injection.md new file mode 100644 index 00000000..904adb5d --- /dev/null +++ b/docs/src/design/activation-injection.md @@ -0,0 +1,110 @@ +# Activation, Injection, and Runtime Control + +Probing enters a target process through one of two paths: a CPython `.pth` startup hook, or Linux +ptrace injection into an already-running process. Both paths load the same `libprobing` and converge +on the same server, query engine, and extension composition root. + +The activation path changes **when the library is loaded**. It does not create a second set of data +tables, collectors, or query APIs. + +## Two activation paths + +![Startup activation and runtime injection converge on one library](../assets/architecture/probing-activation-paths.svg) + +| Path | Use when | Trigger | Platform | +|------|----------|---------|----------| +| Startup hook | The training launch environment is controllable | `PROBING=1 python train.py` | Linux, macOS, Windows | +| Runtime injection | The job is already running and cannot restart | `probing -t inject` | Linux | + +Both run with the target process's existing user and permissions. Injection is not a privilege +bypass: same-UID rules, parent/child relationships, Yama `ptrace_scope`, and `CAP_SYS_PTRACE` remain +kernel policy. + +## Startup hook + +![CPython processes the pth file before loading Probing](../assets/architecture/probing-pth-bootstrap.svg) + +During interpreter startup, Python's `site` module processes the installed `probing.pth`, which +imports `probing_hook.py` and then `site_hook.py`: + +1. parse `PROBING`, preserve the original value, and apply process filters; +2. skip excluded subprocesses; +3. import `probing`, loading the Rust-backed `_core` module; +4. let the Rust library constructor prepare the server and extensions; +5. install optional Python-side Torch, crash, and framework integrations. + +The `.pth` file only obtains one normal import opportunity. Configuration and readiness still +determine whether collectors and network listeners start. + +## Runtime injection + +![ptrace injection uses a reversible trampoline to call dlopen](../assets/architecture/probing-ptrace-injection.svg) + +The injector does not encode dynamic-loading policy in shellcode. It writes a tiny +architecture-specific call trampoline, while the tracer controls registers and ABI arguments for +`setenv`, `malloc`, `dlopen`, and `free` in the target process. + +The sequence is: + +1. attach the main thread and enumerate `/proc//task`; +2. save the original instruction bytes and tracee registers; +3. translate libc/libdl symbol addresses from injector mappings to target mappings; +4. write an x86_64 `call` + `INT3` or AArch64 `BLR` + `BRK` trampoline; +5. accept only the expected trap and validate the `dlopen` result; +6. restore bytes and registers, then detach all attached threads. + +Error paths also attempt restoration. Restoring the trampoline does not unload a successfully +loaded library; injection is a one-time control operation, not a sampling mechanism. + +## Runtime readiness + +![The library constructor establishes control before the Engine becomes ready](../assets/architecture/probing-runtime-control.svg) + +Library load and query readiness are distinct states: + +1. the constructor claims a local Unix socket or configured TCP listener; +2. the engine registers catalogs, data sources, and extensions in the background; +3. readiness moves from claimed/in-progress to ready; +4. CLI, Web, and MCP use documented HTTP interfaces; +5. training hooks append only to local tables and never wait for fan-out queries. + +This lets clients distinguish “not injected,” “library loaded but engine initializing,” and “ready +for queries” instead of reporting every case as connection refusal. + +## Control entry points and CLI structure + +Once the library is ready, the CLI is a client of published protocols; it does not own a second +control implementation: + +![CLI uses public transport to reach Server, Engine, and Extensions in the target](../assets/architecture/probing-cli-control-surface.svg) + +Invocation stays flat: `probing [-v] [-t TARGET] ...`. Root help groups commands under +Processes, Analyze, Diagnose, Runtime, and Agent without changing their protocol boundaries. +Commands with their own actions, such as `skill` and `mcp`, may retain a second level. + +`cluster query` and `cluster nodes` still exist. The target convergence is `query --global` and a +top-level `nodes`; documentation must not present those targets as implemented. Command registration +and per-command text live in `probing/cli/src/cli/commands.rs`; grouped root help lives in `help.rs`. +New commands must continue to use public HTTP/proto contracts. + +## Invariants + +- Both activation paths converge on one composition root. +- Shellcode only provides a remote call opportunity; collection and query logic stay in the library. +- Engine initialization or query failure must not terminate the host training process. +- Training callbacks perform no network I/O. +- CLI, Web, and MCP use published contracts rather than collector internals. +- Injection failure is explicit and must never be represented as an empty successful result. + +## Implementation map + +| Concern | Location | +|---------|----------| +| `.pth` and startup filtering | `python/probing.pth`, `python/probing_hook.py`, `python/probing/site_hook.py` | +| ptrace injection | `probing/cli/src/inject/` | +| Rust/Python library entry | root `src/lib.rs` and `probing._core` | +| Server composition root | `probing/server/src/engine.rs` | +| Public contracts | `probing/server/API.md`, `probing/proto/` | + +See [Installation](../installation.md), [Core model](../guide/concepts.md), and +[Modularity & boundaries](modularity.md). diff --git a/docs/src/design/activation-injection.zh.md b/docs/src/design/activation-injection.zh.md new file mode 100644 index 00000000..36790a4e --- /dev/null +++ b/docs/src/design/activation-injection.zh.md @@ -0,0 +1,112 @@ +# 启用、注入与运行时控制 + +Probing 有两条进入目标进程的路径：在 CPython 启动时通过 `.pth` 自动加载，或者在 Linux +进程运行期间使用 ptrace 注入。两条路径最终加载同一个 `libprobing`，随后由同一套服务端、 +查询引擎和扩展注册流程接管。 + +这两种方式改变的是**何时加载动态库**，不会改变 Probing 的数据表、查询接口或采集器边界。 + +## 两条启用路径 + +![启动加载与运行时注入最终加载同一动态库](../assets/architecture/probing-activation-paths.svg) + +| 路径 | 适用场景 | 触发方式 | 平台 | +|------|----------|----------|------| +| 启动加载 | 能控制训练进程的启动环境 | `PROBING=1 python train.py` | Linux、macOS、Windows | +| 运行时注入 | 任务已经运行且不能重启 | `probing -t inject` | Linux | + +两条路径都在目标进程原有的用户和权限上下文中运行。`inject` 不提供权限绕过；是否允许 +ptrace 由同 UID、父子关系、Yama `ptrace_scope` 和 `CAP_SYS_PTRACE` 等内核策略决定。 + +## 启动路径：`.pth` 如何获得执行机会 + +![CPython 处理 pth 文件后加载 Probing](../assets/architecture/probing-pth-bootstrap.svg) + +Python 启动时由 `site` 模块处理安装目录中的 `.pth` 文件。Probing wheel 安装的 +`probing.pth` 导入 `probing_hook.py`，后者再进入 `site_hook.py`： + +1. 解析 `PROBING`，保存原值并检查当前进程是否需要启用； +2. 跳过不应启动服务的进程，例如用户配置排除的子进程； +3. 导入 `probing`，从而加载包含 Rust 运行时的 `_core` Python 模块； +4. Rust 动态库构造函数准备服务端和扩展； +5. Python 侧安装 torch、异常处理和可选框架集成。 + +`.pth` 只负责在解释器正常启动流程中获得一次导入机会。是否启动采集和网络监听仍由 +`PROBING` 配置、进程过滤和后续初始化状态决定。 + +## 注入路径：可回滚的远程函数调用 + +![ptrace 注入使用可回滚跳板执行 dlopen](../assets/architecture/probing-ptrace-injection.svg) + +注入器不会把完整的动态加载逻辑写成 shellcode。它只写入一个与目标架构匹配的极小调用 +跳板，用 tracer 控制寄存器和 ABI 参数，依次调用目标进程中的 `setenv`、`malloc`、 +`dlopen` 和 `free`。 + +关键步骤如下： + +1. **停止并取得控制。** attach 主线程并枚举 `/proc//task`，确保注入期间线程不会在 + 被覆盖的指令区域继续执行。 +2. **保存现场。** 保存目标地址的原始代码字节和被选线程的寄存器。 +3. **换算函数地址。** 根据 injector 与 target 的共享库基址换算目标 libc/libdl 中的符号地址， + 不能把本进程虚拟地址直接写给目标进程。 +4. **写入调用跳板。** x86_64 使用间接 `call` 和 `INT3`，AArch64 使用 `BLR` 和 `BRK`； + 参数、栈对齐和返回值遵守各自 ABI。 +5. **校验结果。** 只接受预期 trap，并检查 `dlopen` 返回值。目标进程退出、收到其他 signal + 或动态库加载失败都会显式报错。 +6. **恢复与 detach。** 恢复原始字节和寄存器，再逐个 detach 已附着线程。错误路径也必须 + 尽力恢复现场。 + +恢复现场不等于卸载动态库：`dlopen` 成功后，库及其构造函数产生的运行时状态会保留。 +因此注入是一次控制操作，不应作为频繁采样机制。 + +## 动态库加载后如何接受控制 + +![动态库构造函数、监听入口与 Engine 就绪过程](../assets/architecture/probing-runtime-control.svg) + +动态库加载成功和查询引擎可用是两个不同状态： + +1. 构造函数尽快创建本地 Unix socket 或配置的 TCP 监听入口； +2. Engine 在后台完成 catalog、扩展和数据源注册； +3. readiness 从 `claimed` / `in-progress` 进入 `ready`； +4. CLI、Web 和 MCP 通过公开 HTTP 接口查询或控制； +5. 训练 hook 只写本地表，不等待远端控制请求，也不参与 fan-out。 + +监听入口提前存在，可以让调用方区分“进程尚未注入”“动态库已经加载但 Engine 尚未就绪” +和“服务可以查询”三种状态，而不是把所有失败都表现为连接拒绝。 + +## 控制入口与 CLI 结构 + +动态库进入 `ready` 后，CLI 只是公开协议的客户端，不拥有另一套控制逻辑： + +![CLI 通过公开传输访问目标进程中的 Server、Engine 与 Extensions](../assets/architecture/probing-cli-control-surface.svg) + +命令保持单层调用：`probing [-v] [-t TARGET] ...`；根帮助按 Processes、Analyze、 +Diagnose、Runtime、Agent 分组，但分组不改变协议或模块边界。`skill`、`mcp` 等具有自身动作的 +命令可以保留二级子命令。 + +当前仍保留 `cluster query` 与 `cluster nodes`。目标收敛方向是分别并入 `query --global` 和 +顶层 `nodes`；在实现完成前，文档不能把目标命令写成现有接口。命令注册与单命令说明位于 +`probing/cli/src/cli/commands.rs`，根帮助分组位于 `help.rs`。新增命令必须先判断它属于进程 +控制、查询、诊断还是 Agent 入口，并继续通过公开 HTTP/proto 契约访问服务端。 + +## 关键不变量 + +- 启动加载和运行时注入最终进入同一个 composition root，不维护两套服务端实现。 +- 注入器只负责取得一次 `dlopen` 执行机会；采集和查询逻辑不放进 shellcode。 +- 目标进程的训练线程不能因 Engine 初始化或远端查询失败而退出。 +- 训练回调不执行网络请求；采集写路径与 HTTP 控制面解耦。 +- CLI、Web 和 MCP 只使用公开接口，不链接或调用采集器内部实现。 +- 注入失败必须可观察，不能通过伪造“空数据”表现为成功。 + +## 相关实现 + +| 关注点 | 位置 | +|--------|------| +| `.pth` 与启动过滤 | `python/probing.pth`、`python/probing_hook.py`、`python/probing/site_hook.py` | +| inject CLI 与 ptrace | `probing/cli/` 中的 inject/ctrl 实现 | +| Rust/Python 动态库入口 | 根 crate `src/lib.rs` 与 `probing._core` | +| 服务端 composition root | `probing/server/src/engine.rs` | +| 公开接口 | `probing/server/API.md`、`probing/proto/` | + +使用方式见[安装指南](../installation.zh.md)和[核心模型](../guide/concepts.zh.md)；服务端与模块 +边界见[模块化与边界](modularity.zh.md)。 diff --git a/docs/src/design/architecture.md b/docs/src/design/architecture.md deleted file mode 100644 index ef85fa9b..00000000 --- a/docs/src/design/architecture.md +++ /dev/null @@ -1,8 +0,0 @@ -# System Architecture - -!!! warning "Legacy document" - This two-layer overview is **superseded** by **[Modularity & Boundaries](modularity.md)** (four-layer model, crate map, dependency rules, and ownership). - -**New contributors:** start with [Modularity & Boundaries](modularity.md), then [Data Layer](data-layer.md) and [Distributed overview](distributed.md). - -Historical Mermaid diagrams from the pre–four-layer model were removed to avoid drift from the current crate map. diff --git a/docs/src/design/architecture.zh.md b/docs/src/design/architecture.zh.md deleted file mode 100644 index 56306ae1..00000000 --- a/docs/src/design/architecture.zh.md +++ /dev/null @@ -1,8 +0,0 @@ -# 系统架构 - -!!! warning "旧版文档" - 本页的两层概览已被 **[模块化与边界](modularity.zh.md)**（四层模型、crate 地图、依赖规则与归属）**取代**。 - -**新贡献者：** 从 [模块化与边界](modularity.zh.md) 入手，再读 [数据层](data-layer.zh.md) 与 [分布式概览](distributed.zh.md)。 - -为避免与当前 crate 地图漂移，旧版两层 Mermaid 示意图已移除。 diff --git a/docs/src/design/cli.md b/docs/src/design/cli.md deleted file mode 100644 index fa7c12c1..00000000 --- a/docs/src/design/cli.md +++ /dev/null @@ -1,75 +0,0 @@ -# CLI command tree - -**Status:** draft · **SSOT** for grouping and migration · Code: `probing/cli/src/cli/{commands,help,mod}.rs` - -**Legend:** `T` = `-t/--target` · `*` = needs T · `—` = no T · `L` = Linux only · `H` = hidden - ---- - -## Principle: flat invocation, grouped help - -| Dimension | Rule | -|-----------|------| -| **Invocation** | Single-level: `probing [-v] [-t T] [args…]` | -| **Help** | `probing --help` grouped under Processes / Analyze / Diagnose / Runtime / Agent | -| **Consolidation** | Merge `cluster query` → `query --global`; `cluster nodes` → top-level `nodes` (TBD) | -| **Exceptions** | `skill` keeps subcommands; `bench` / `store` hidden | - -### Help implementation (option B, adopted) - -clap 4.5–4.6 does **not** support multiple subcommand headings ([clap#1553](https://github.com/clap-rs/clap/issues/1553) still open). We use a **custom root help template**: - -1. Root `help_template` **omits `{subcommands}`**; keeps `{about}` / `{usage}` / `{options}`. -2. `{after-help}` injects grouped command tables from `help.rs`. -3. Section titles and blurbs in **`help.rs` → `SECTIONS`**; per-command one-liners from clap `about` on each subcommand (`commands.rs` remains per-command wording SSOT). -4. `probing --help` still uses default clap per-command help. - -```text -Cli::build_command() - → CommandFactory::command() - → help::apply_grouped_root_help() -``` - ---- - -## Help sections (rationale) - -| Section | Commands | Notes | -|---------|----------|-------| -| **Processes** | `inject`, `launch`, `list` | Establish or discover probing on a process; avoid “Attach” (ptrace jargon) | -| **Analyze** | `query`, `tables`, `cluster` | SQL and catalog; `cluster` until merged into `query --global` / `nodes` | -| **Diagnose** | `eval`, `repl`, `backtrace` | Interactive, immediate inspection | -| **Runtime** | `memory`, `config`, `flamegraph`, `rdma` | Runtime state and profiling | -| **Agent** | `skill`, `mcp` | Coding-agent integration: skills and MCP config | - ---- - -## Target invocation tree - -```text -probing [-v] [-t T] … - -inject(L*)* launch(L)— list— -query* tables* nodes* # TBD: merge cluster into query/nodes -eval* repl* backtrace* flamegraph* rdma* -memory* config* -skill list— | install— | update— | run* … -mcp url* | config* -bench(H)— store(H)— -``` - ---- - -## Maintenance - -When adding or moving a top-level subcommand: - -1. Register in `commands.rs` (with `about`) -2. Add name to `help.rs` → `SECTIONS` -3. Update this doc - ---- - -## Related - -[federation.md](federation.md) · [skills](../guide/skills.md) · [api-reference](../api-reference.md) diff --git a/docs/src/design/cli.zh.md b/docs/src/design/cli.zh.md deleted file mode 100644 index 3fae94b7..00000000 --- a/docs/src/design/cli.zh.md +++ /dev/null @@ -1,163 +0,0 @@ -# CLI 命令树 - -**状态：** 草案 · **SSOT** 命令分组与迁移 · 实现：`probing/cli/src/cli/{commands,help,mod}.rs` - -**约定：** `T` = `-t/--target` · `*` = 需要 T · `—` = 不需要 T · `L` = 仅 Linux · `H` = hidden - ---- - -## 设计原则：平铺调用，分组帮助 - -| 维度 | 规则 | -|------|------| -| **调用** | 单层子命令：`probing [-v] [-t T] [args…]` | -| **帮助** | `probing --help` 按 Processes / Analyze / Diagnose / Runtime / Agent **分组展示** | -| **收敛** | 合并 `cluster query` → `query --global`；`cluster nodes` → 顶层 `nodes`（待做） | -| **例外** | `skill` 保留二级子命令；`bench`/`store` 隐藏 | - -### Help 实现（方案 B，已采用） - -clap 4.5–4.6 **不支持**同级子命令多 heading（[clap#1553](https://github.com/clap-rs/clap/issues/1553) 未 merge）。采用 **自定义根 help 模板**： - -1. 根 `help_template` **去掉 `{subcommands}`**，只保留 `{about}` / `{usage}` / `{options}`。 -2. `{after-help}` 注入 `help.rs` 渲染的分组命令表。 -3. 分组标题与组说明在 **`help.rs` → `SECTIONS`**；各命令单行说明来自 subcommand 的 clap `about`（`commands.rs` 仍为 per-command 文案 SSOT）。 -4. `probing --help` 仍走 clap 默认 per-command help。 - -```text -Cli::build_command() - → CommandFactory::command() - → help::apply_grouped_root_help() # template + after_long_help -``` - ---- - -## Help 分组（命名 rationale） - -| 组 | 命令 | 说明 | -|----|------|------| -| **Processes** | `inject`, `launch`, `list` | 与目标进程建立/发现 probing 关系；不用「Attach」（用户不熟悉 ptrace 术语） | -| **Analyze** | `query`, `tables`, `cluster` | SQL 与表目录；cluster 暂保留至 `query --global` / `nodes` 落地 | -| **Diagnose** | `eval`, `repl`, `backtrace` | 交互式、即时检查 | -| **Runtime** | `memory`, `config`, `flamegraph`, `rdma` | 运行时状态与 profiling（资源、配置、采样、I/O） | -| **Agent** | `skill`, `mcp` | 与 coding agent 集成：诊断 skill 与 MCP 端点配置 | - ---- - -## 当前（v0.2.5） - -```text -probing [-v] [-t T] … - -inject(L*)* launch(L)— list— config* tables* memory* -query|q* cluster/query* cluster/nodes* -eval* repl* backtrace* flamegraph* rdma* -skill/{list—,install—,update—,run*} mcp/{url*,config*} -bench(H)— store(H)— -external→probing- -``` - ---- - -## 目标 — 平铺命令 + 分组 `--help` - -### 调用树 - -```text -probing [-v] [-t T] … - -inject(L*)* [-D define…] -launch(L)— [-r] -list— [--tree] [--verbose] - -query* [-f fmt] [--global|--local|--flat] # 待做：吸收 cluster query -tables* [--all] [-f fmt] -nodes* # 待做：吸收 cluster nodes - -memory* config* flamegraph* rdma* -skill list— | install— | update— | run* … -mcp url* | config* -bench(H)— store(H)— -``` - -### 帮助树（`probing --help` 输出） - -```text -Processes — Start probing on a process, wrap a new command, or list probed PIDs - inject Inject libprobing into a running process (Linux ptrace) - launch Launch a command with probing enabled (Linux) - list List processes that already have probing enabled - -Analyze — Run SQL, inspect table catalog, fan out across cluster nodes - query Query data from the target process - tables List queryable tables in the target process - cluster On-demand cluster SQL fan-out and node listing - -Diagnose — Interactive inspection — Python eval, REPL, stack traces - eval Evaluate Python code in the target process - repl Interactive Python REPL session - backtrace Show the backtrace of the target process or thread - -Runtime — Runtime state and profiling — memory, config, flamegraphs, RDMA flows - memory Show memory usage (host RSS and GPU memory) of the target process - config Display or modify the configuration - flamegraph Fetch a flamegraph (CPU/pprof or PyTorch) from the target process - rdma Get RDMA flow of the target process or thread - -Agent — Integrate coding agents — diagnostic skills and MCP server config - skill Run structured diagnostic skills (shared with Web Agent) - mcp MCP endpoint URL and agent config for the target probing server - -Run `probing --help` for command-specific options. -``` - ---- - -## 典型用法 - -```text -probing inject -t PID -probing query -t PID 'SELECT …' -probing skill run -t PID health_overview -probing mcp config -t host:8080 -probing backtrace -t PID -``` - ---- - -## 变更摘要（相对 v0.2.5） - -| 状态 | 项 | -|------|-----| -| ✅ | 分组根 `--help`（`help.rs`） | -| ✅ | Help 组名：Processes / Analyze / Diagnose / Runtime / Agent | -| ✅ | `probing mcp url|config` — MCP 端点与 agent 配置片段 | -| 待做 | `cluster query` → `query --global` | -| 待做 | `cluster nodes` → 顶层 `nodes` | -| ✅ | 取消隐式 inject（无子命令不再 inject） | -| ✅ | CLI 导入跳过 `#[ctor]` 引擎启动（`__init__.py` CLI 轻量路径） | - ---- - -## 待定 - -| # | 问题 | 默认 | -|---|------|------| -| 1 | positional `[T]` 作 `-t` 糖 | TBD | -| 2 | external 插件 | TBD | - ---- - -## 维护 - -新增/移动顶层子命令时： - -1. 在 `commands.rs` 注册 subcommand（含 `about`） -2. 在 `help.rs` → `SECTIONS` 里加入组与命令名 -3. 更新本文调用树 - ---- - -## 相关 - -[federation.zh.md](federation.zh.md) · [skills](../guide/skills.md) · [api-reference](../api-reference.md) diff --git a/docs/src/design/cluster-pulsing.md b/docs/src/design/cluster-pulsing.md deleted file mode 100644 index 4a3126fb..00000000 --- a/docs/src/design/cluster-pulsing.md +++ /dev/null @@ -1,52 +0,0 @@ -# Cluster Management with Pulsing - -Design note: reuse [Pulsing](https://github.com/DeepLink-org/Pulsing) gossip membership and -failure detection for probing cluster discovery, while keeping probing's training semantics -(`rank`, `role`, federation tags). - -!!! note "Language" - The full design narrative is maintained in Chinese: - **[中文版](/zh/design/cluster-pulsing/)**. This page is an English summary. - -## Current probing cluster model - -- **Store**: in-memory `probing_proto::Cluster`, keyed by `host:addr`, with `rank_index`. -- **Node fields**: `host`, `addr`, ranks, `role_name` (torchrun), **`role`** (probing parallel - key like `dp=2,pp=1,tp=0`), `status`, `timestamp`. -- **Write path**: rank 0 calls `update_node` locally; other ranks `PUT /apis/nodes` to the - report address (usually rank 0). -- **Read path**: `GET /apis/nodes`, Web cluster view, `cluster.nodes` SQL table. -- **Limitation**: no built-in failure detection; liveness depends on app heartbeats. - -CLI today: `probing -t cluster nodes` and `cluster query`. - -## What Pulsing adds - -- SWIM-style gossip (`GossipCluster`) with PFail → Fail detection. -- `alive_members()` / HTTP `GET /cluster/members` for membership without app polling. - -## Design goals - -1. **Discovery & liveness** from Pulsing; probing keeps rank/role/world_size semantics. -2. **No breaking API** — existing `PUT /apis/nodes`, `cluster.nodes`, `global.*` federation unchanged. -3. **Optional integration** — probing works without Pulsing; Pulsing enhances multi-node discovery. - -## Proposed integration (summary) - -```mermaid -flowchart LR - TRAIN[Training ranks] -->|PUT /apis/nodes| PROBE[Probing server] - PULS[Pulsing gossip] -->|alive members| PROBE - PROBE --> NODES[cluster.nodes + federation] - CLI[probing cluster nodes] --> PROBE -``` - -- Map Pulsing `MemberInfo` → probing `Node` skeleton; merge training-reported fields on heartbeat. -- Suspect/dead members surface in `cluster.nodes.status` for Web/CLI. -- Federation rewrite (`_rank`, `_role`, …) unchanged — see [Distributed](distributed.md). - -## Related - -- **[Distributed](distributed.md)** — `global.*`, `cluster query`, `role` -- **[Cluster with Pulsing (中文)](/zh/design/cluster-pulsing/)** — full design -- **[SQL Tables](../reference/sql-tables.md)** — `cluster.nodes` columns diff --git a/docs/src/design/cluster-pulsing.zh.md b/docs/src/design/cluster-pulsing.zh.md deleted file mode 100644 index 3c6ce24a..00000000 --- a/docs/src/design/cluster-pulsing.zh.md +++ /dev/null @@ -1,183 +0,0 @@ -# 基于 Pulsing 的集群管理 - -本文档描述如何让 probing 的集群管理复用 Pulsing 的分布式能力（gossip 成员、故障检测、命名发现），在保持现有 API 与业务语义的前提下，获得自动成员发现与存活检测。 - -## 现状 - -### probing 当前集群模型 - -- **数据源**：内存中的 `probing_proto::Cluster`，key 为 `host:addr`，带 `rank_index`（rank → 节点）。 -- **Node 字段**（proto）：`host`, `addr`, `local_rank`, `rank`, `world_size`, `group_rank`, `role_name`, `role_rank`, `role_world_size`, `role`, `status`, `timestamp`。其中 `role` 是并行角色 key（如 `dp=2,pp=1,tp=0`），由训练进程上报，联邦查询时作为 `_role` 标签附加到每行（区别于 torchrun 的 `role_name`）。 -- **写入路径**： - - **rank 0**：本机直接调用 `update_node(node)`（见 `report.rs`）。 - - **其他 rank**：通过 HTTP `PUT /apis/nodes` 向「中心」（report_addr，通常为 rank 0 或独立 server）上报。 -- **读取路径**：HTTP `GET /apis/nodes`、Web 集群页、extensions/cc 的 `nodes` 表（Arrow）。 -- **特点**：无内置故障检测；依赖应用侧定时上报；中心聚合视图。 - -### Pulsing 提供的集群能力 - -- **Gossip 成员**：SWIM 风格，`GossipCluster`，周期 gossip + 故障检测（PFail → Fail）。 -- **成员信息**：`MemberInfo`：`node_id`（u128）、`addr`（SocketAddr）、`status`（Alive/Suspect/Dead）、`incarnation` 等。 -- **API**： - - Rust：`cluster.all_members()` / `alive_members()`。 - - Python：`await system.members()` → `list[dict]`（`node_id`, `addr`, `status` 等）。 - - HTTP：`GET /cluster/members` 返回 JSON 成员列表。 -- **发现**：seed 加入、named actor 注册、跨节点解析。 - -## 设计目标 - -1. **成员发现与存活**：由 Pulsing 负责「谁在集群里、谁还活着」，probing 不再仅依赖应用上报来推断存活。 -2. **保留业务语义**：rank、world_size、role_name 等训练/作业语义仍由 probing 侧维护（上报或配置）。 -3. **接口兼容**：现有 `get_nodes` / `put_node`、`nodes` 表、Web 集群页行为保持可用；可演进为「Pulsing 成员 + 业务元数据合并」视图。 -4. **松耦合**：probing 自管、主动发现 Pulsing；不要求 Pulsing 或应用先「接好」probing。 - -## 松耦合原则 - -1. **probing 被注入后，自己管好自己** - 不依赖外部先启动或配置 Pulsing；probing 进程内行为自洽（上报、本地 CLUSTER、HTTP API 等照常工作）。 - -2. **cluster 模块主动发现 Pulsing** - probing 的 cluster 模块在适当时机（例如 server 启动或首次访问集群视图时）**尝试发现**当前环境里是否已有 Pulsing（例如检测全局 ActorSystem、环境变量、或指定 URL/端口）。 - -3. **发现 Pulsing 后的两种路径** - - **a. Pulsing 已初始化** - 若发现 Pulsing 的 ActorSystem 已经存在（例如用户代码已 `pul.init()`）： - - 在该 ActorSystem 上**注册一个专用于集群管理的 actor**（例如 named actor `"probing/cluster"`）； - - 节点发现通过**向该 actor 请求**或**该 actor 订阅/拉取 members()** 得到，再写回 probing 的 `CLUSTER`。 - - **b. Pulsing 未初始化，且配置了初始化方式** - 若未发现已初始化的 Pulsing，但配置中指定了如何初始化（例如 `pulsing_seeds`、standalone 等）： - - probing **尝试自己初始化** Pulsing（例如在 Python 侧调用 `pul.init(seeds=...)` 或等价逻辑）； - - 初始化成功后，再走路径 a：注册集群管理 actor，通过该 actor 做节点发现。 - -未发现 Pulsing 或未配置初始化时，cluster 模块**仅使用现有上报**构建视图，行为与当前一致。 - -## 集成形态（在松耦合下的表现） - -- **有 Pulsing（已初始化）**：cluster 模块发现现有 ActorSystem → 通过拉取 members() 与上报合并写入 CLUSTER。 -- **有 Pulsing（需由 probing 初始化）**：cluster 模块按配置初始化 Pulsing → 同上。 -- **无 Pulsing**：不注册 actor，仅靠 PUT /apis/nodes 与 rank 0 本地 update_node 构建视图。 - -### 组网由 Pulsing 后台负责，调用方只调 API 等待 - -Pulsing 提供 **bootstrap** 模块：在**后台线程**中自动尝试组网（先 Ray 再 torchrun），对外只暴露 `wait_ready(timeout)` / `await_ready(timeout)`。probing 或其他调用方**不**实现 init_in_ray / init_in_torchrun，只需在需要集群时调用 `pulsing.bootstrap.wait_ready(timeout)` 等待，返回 True 后即可 `get_system()` 并使用。组网逻辑（谁当 seed、如何广播）全部在 Pulsing 内部完成。 - -### 借助 Ray / torchrun 组网（由 bootstrap 内部调用） - -- **Ray**：`pulsing.integrations.ray.init_in_ray()`，首个进程通过 Ray KV 成为 seed，其余以 `seeds=[seed_addr]` 加入。 -- **torchrun**：`pulsing.integrations.torchrun.init_in_torchrun()`，rank0 广播地址，其他 rank 以 seed 加入。 - -bootstrap 后台会按顺序尝试上述两种方式；probing 只需在启动同步前调用 `pulsing.bootstrap.wait_ready(PROBING_PULSING_BOOTSTRAP_TIMEOUT)` 等待即可。 - -## 数据流（松耦合） - -```text -┌─────────────────────────────────────────────────────────────────┐ -│ probing 进程（被注入后自管） │ -│ • cluster 模块尝试发现 Pulsing │ -└────────────────────────────┬──────────────────────────────────┘ - │ - ┌───────────────────┼───────────────────┐ - ▼ ▼ ▼ - 未发现 Pulsing 发现已初始化 Pulsing 未初始化但配置了 init - │ │ │ - ▼ ▼ ▼ - 仅用上报构建视图 在 ActorSystem 上 probing 自己 init - (当前行为) 注册集群管理 actor Pulsing，再注册 actor - │ │ - └─────────┬─────────┘ - ▼ -┌─────────────────────────────────────────────────────────────────┐ -│ Pulsing ActorSystem │ -│ • 集群管理 actor（如 "probing/cluster"）拉取 members() │ -│ • 映射为 Node 基础信息，与 PUT 上报的 rank/role 合并 │ -└────────────────────────────┬──────────────────────────────────┘ - │ - ▼ -┌─────────────────────────────────────────────────────────────────┐ -│ probing_core::cluster::CLUSTER │ -│ Cluster { nodes: host:addr -> Node, rank_index } │ -│ Node = 基础(Pulsing) + 业务(rank, world_size, role, ...) │ -└────────────────────────────┬──────────────────────────────────┘ - │ - ┌───────────────┼───────────────┐ - ▼ ▼ ▼ - GET /apis/nodes Web 集群页 extensions/cc nodes 表 -``` - -- **合并规则**：以 `host:addr` 或 Pulsing `node_id` 为键；若某节点仅有 Pulsing 信息则只填基础字段，rank/role 为空或默认；若仅有上报则仅填业务字段，存活状态可标为「未知」或依赖超时剔除。 -- **故障检测**：从 Pulsing 来的节点若 `status != Alive`，可在 probing 视图中标记为 Unhealthy/Unreachable 或从列表中过滤/标注。 - -## 接口与兼容性 - -- **保持**：`GET /apis/nodes`、`PUT /apis/nodes`、`cluster::get_nodes()`、`cluster::update_node()`、extensions/cc `nodes` 表 schema（host, addr, rank, world_size, role_*, role, status, timestamp）。 -- **扩展**（可选）： - - Node 或 API 增加 `pulsing_node_id`、`pulsing_status`，便于调试与展示。 - - 配置项：发现方式（如 `pulsing_auto_discover`）、初始化方式（如 `pulsing_seeds`、`pulsing_init_mode`）。 -- **兼容**：未发现 Pulsing 且未配置初始化时，行为与当前一致，仅靠上报构建集群视图。 - -## 实施步骤建议 - -1. **Phase 1（发现 + 使用已有 Pulsing）** - - cluster 模块实现「发现 Pulsing」：检测全局 ActorSystem（Python 侧 `pul.get_system()` / 是否已 init）、或环境变量/配置中的 Pulsing 入口。 - - 若已初始化：在 ActorSystem 上注册集群管理 actor（named，如 `"probing/cluster"`），该 actor 负责拉取 `members()`、映射为 `Vec`，并可由 probing 侧定时请求或订阅更新；合并进 CLUSTER。 - - 配置：仅「是否启用发现」与「集群管理 actor 名称」等，不强制要求配置 Pulsing。 - -2. **Phase 2（可选由 probing 初始化 Pulsing）** - - 当未发现已初始化的 Pulsing 且配置了初始化方式（如 `pulsing_seeds`、standalone）时，cluster 模块（或 Python 桥）尝试调用 `pul.init(...)`（或等价）。 - - 初始化成功后走 Phase 1：注册集群管理 actor，通过该 actor 做节点发现。 - - 统一「成员源」：仅上报 / Pulsing（通过集群管理 actor）/ 合并。 - -3. **Phase 3（可选增强）** - - 集群管理 actor 可对外提供「注册 rank/role」接口，与 PUT /apis/nodes 互补或替代。 - - 分布式存储（TopologyView）的节点列表也可从 Pulsing 成员驱动。 - -## 依赖与约束 - -- **依赖**：若在 Rust 侧直接依赖 Pulsing，需在 probing 的 Cargo 中增加 `pulsing-actor`（或仅通过 HTTP/FFI 调用则无 Rust 依赖）。 -- **进程模型**：当前 report 仅 rank 0 写本地、其余 PUT 到中心；与 Pulsing 集成后，中心或 rank 0 需能访问 Pulsing 视图（同进程或远程）。 -- **身份对应**：需要约定 Pulsing 节点与 probing 节点的对应关系（例如同一进程既上报 `host:addr` 又加入 Pulsing，则用 `addr` 或 node_id 关联）。 - -## 小结 - -- **松耦合**：probing 被注入后自管；cluster 模块主动发现 Pulsing；发现后要么用已有 ActorSystem 并注册集群管理 actor，要么在配置允许时由 probing 自己初始化 Pulsing 再注册 actor。 -- **用 Pulsing 做**：通过「集群管理 actor」做成员发现与存活检测；该 actor 挂在 Pulsing 的 ActorSystem 上，由 probing 注册与使用。 -- **probing 保留**：rank/world_size/role 等业务语义、现有 HTTP API、`nodes` 表与 Web 集群页；无 Pulsing 时行为不变。 -- **集成方式**：发现 → 已初始化则注册 actor / 未初始化且配置则先 init 再注册 actor → 通过该 actor 拉取 members 并与上报合并写入 CLUSTER。 - -这样在不破坏现有使用方式、且不要求应用或 Pulsing 先接好 probing 的前提下，为 probing 带来基于 Pulsing 的自动集群发现与故障感知能力。 - ---- - -## 设计点评 - -### 优点 - -- **松耦合方向正确**：probing 自管、主动发现 Pulsing，无 Pulsing 时退化为纯上报，不绑架部署顺序，对现有用户友好。 -- **职责清晰**：Pulsing 管「谁在、谁活」，probing 管 rank/role 等业务语义；集群管理 actor 作为单一桥梁，边界明确。 -- **兼容与渐进**：GET/PUT、nodes 表、Web 页不变；Phase 1→2→3 可分批落地，风险可控。 -- **发现路径完整**：已初始化 / 未初始化且配置 init 两条路径都覆盖，文档里写清楚了分支逻辑。 - -### 潜在问题与风险 - -1. **发现时机与竞态** - 「适当时机」若只在 server 启动或首次访问时做一次发现，之后 Pulsing 才被用户 `pul.init()`，可能漏掉。建议：发现失败或未发现时，在后续**定时重试**或「首次访问集群视图时重试」，并文档化重试策略（间隔、上限）。 - -2. **集群管理 actor 的归属与生命周期** - actor 由 probing 在「别人的」ActorSystem 上注册，若 probing 先退出而 Pulsing 常驻，会留下 named actor；若 Pulsing 先 shutdown，probing 侧要有**检测断开 + 回退到仅上报**的逻辑，否则会持续请求已失效的 actor。文档可明确：actor 随哪边生命周期、断开后是否自动降级。 - -3. **probing 自己 init Pulsing 的适用边界** - 由 probing 调 `pul.init(seeds=...)` 时，当前进程会加入 Pulsing 集群。若同一台机上多进程都注入 probing 且都配置了 init，可能变成多节点加入同一集群，是否预期需要写清；standalone 与 cluster 模式在「由 probing 初始化」时的行为建议在文档里区分（例如仅允许 standalone 或仅允许指定了 seeds 的 cluster）。 - -4. **身份对应 (host:addr vs node_id)** - Pulsing 的 `node_id` 与 probing 的 `host:addr` 如何稳定对应，文档只说了「约定」。若同一物理机多进程、或容器重启导致 addr 复用，合并时可能错位或重复。建议在实施时约定：例如优先用 `addr` 做关联，或要求上报里带 `pulsing_node_id` 以便精确匹配，并在设计里写一句「合并键策略」。 - -5. **Rust 与 Python 的边界** - cluster 模块若在 Rust（server）里，发现「全局 ActorSystem」必然要经 Python 或 HTTP；若在 Python 扩展里，则可直接 `pul.get_system()`。文档里「发现」和「注册 actor」到底在 Rust 侧还是 Python 侧实现，会直接影响 Phase 1 的落地方式，建议在实施步骤里明确**谁负责发现、谁负责注册 actor、谁负责拉取并写 CLUSTER**（例如：Python 发现+注册 actor，Rust 通过 FFI/HTTP 向该 actor 要成员并写 CLUSTER）。 - -### 建议补充 - -- **配置契约**：列出「发现 / 初始化」相关配置项与默认值（如 `pulsing_auto_discover=true`、未配置 seeds 时不自动 init），避免实现时各说各话。 -- **可观测**：发现成功/失败、是否使用集群管理 actor、降级到仅上报，建议打日志或指标，便于运维判断当前是否在用 Pulsing 视图。 -- **测试策略**：无 Pulsing、已有 Pulsing、probing 自己 init 三种场景各有一个明确测试用例，防止回归。 - -整体上，这个实现方向合理、文档已把主流程和松耦合讲清楚；把上述「发现重试、生命周期与降级、身份对应、Rust/Python 边界」在文档或后续实现里补上，落地会更稳。 diff --git a/docs/src/design/data-layer.md b/docs/src/design/data-layer.md index 33889af8..97a975b9 100644 --- a/docs/src/design/data-layer.md +++ b/docs/src/design/data-layer.md @@ -26,22 +26,7 @@ A single SQL time predicate prunes and queries both tiers at once. ## Architecture -```mermaid -graph LR - APP[push_row / RowWriter] --> MEMT - subgraph HOT[Hot tier · probing-memtable] - MEMT[MEMT ring buffer] --> SEALED[sealed chunk\nmin/max ts + generation] - end - SEALED -->|transpose + Pco| ROLLER[Compactor / Roller] - ROLLER --> MEMC - subgraph COLD[Cold tier · MEMC segments] - MEMC[ColdStore\nimmutable segments] - end - - SQL[SQL query] --> HCT[HotColdTable] - HCT -->|chunk pruning| MEMT - HCT -->|segment + page pruning| MEMC -``` +![MEMT hot layer, MEMC cold layer, and unified query](../assets/architecture/probing-hot-cold-overview.svg) The hot tier is logically read-only at query time, but MEMT mappings are opened writable so readers can update per-chunk pin metadata. The cold tier is read via `SegmentReader`. The @@ -165,20 +150,27 @@ monotonically increasing sequence the `ColdStore` recovers on open. ### Segment Format -A segment is a sequence of 64-aligned blocks. All integrity checks use **xxh3-64 truncated to 32 -bits**. +MEMC must do more than compress old rows to disk. Hot slots are recycled, the process can exit at +any byte boundary, and readers must distinguish "not committed yet" from "committed but corrupt." +The segment therefore carries its own commit protocol and source identity instead of relying on a +separate transaction log. + +A segment is a sequence of 64-aligned blocks. `MCTB` declares a table and schema, `MCPG` stores one +column page, and `MCFT` is the page directory written at seal time. Integrity checks use +**xxh3-64 truncated to 32 bits**. **Segment header (64 bytes):** `magic` (`"MEMC"`), `version` (2), BOM, `flags` (bit 0 = sealed), `writer_pid`, `writer_start`, `created_unix_ms`, `footer_off` (0 until sealed), segment-wide `ts_min`/`ts_max`, `page_count`, header checksum. -**Blocks** share a 64-byte header: +![MEMC physical segment format, column encodings, and recovery paths](../assets/architecture/probing-memc-format.svg) -| magic | meaning | -|---|---| -| `MCTB` | table-definition block — declares a `table_id`, name, column dtypes, ts column | -| `MCPG` | page block — one columnar page for a `table_id` | -| `MCFT` | footer — page directory written on seal | +The writer appends the header, table definitions, and column pages first. It writes the footer only +after every payload and checksum is complete, then rewrites the header as sealed. The sealed bit is +therefore the segment commit point. Before commit, a reader accepts only complete blocks found by a +forward scan. After commit, any footer, block, or payload validation failure is corruption. The +asymmetry follows causality: an unsealed tail may simply be unfinished, while a sealed segment has +already declared itself complete and cannot reinterpret corruption as normal truncation. The page/block header carries `table_id`, `row_count`, `col_count`, `ts_min`/`ts_max`, `payload_len`, `payload_xxh`, and — crucially for restart dedup — `source_instance`, `source_gen`, @@ -193,26 +185,21 @@ that need their historical rows must archive or convert them before upgrading. A single segment holds pages from **multiple tables**, distinguished by `table_id`. This decouples file/directory count from table count: hundreds of tables share one set of segment files. -### Column Encodings - -Each column is encoded independently (`ColEncoding`): +### Why encoding is columnar -- **`Pco`** — numeric columns (`i32/i64/f32/f64/u32/u64`), compressed with Pco (level 8). Monotonic - timestamp columns compress > 4×. -- **`RawFixed`** — `u8` (Pco offers no benefit for byte columns). -- **`RawVarLen`** — `Str`/`Bytes`, stored as concatenated `[u32 len][bytes]` entries (Pco has no - string support). +Once off the hot path, the compactor transposes rows so each data class can use its own statistical +structure. Numeric columns use Pco, which is especially effective for monotonic timestamps; `u8` +remains `RawFixed` to avoid useless compression; strings and byte arrays use length-prefixed +`RawVarLen`. The page header fixes the choice so readers never guess. Moving transpose and +compression to the cold tier is what lets MEMT retain a simple, allocation-free row write path. -### Crash Recovery +### Recovery boundary -- A **sealed** segment is read via its footer page directory — O(1) location of every page. Footer, - block, payload, and page integrity failures reject the segment and fail the SQL scan; queries - never return a successful but incomplete cold result. -- An **unsealed or torn** segment is recovered by **forward scan**. Only an incomplete final - header/payload is dropped as a crash tail; corruption of a complete checksummed block is an - error. Table-definition blocks are always scanned (cheap, and they precede pages). - -There is no heuristic that tries to repair a half-written record. +A sealed segment uses its footer to locate pages, and any integrity failure fails the SQL scan; a +query never succeeds with silently missing cold pages. An unsealed segment has made no completeness +claim, so a forward scan may drop the final incomplete header or payload. A fully written block +with a bad checksum is still an error. Recovery neither stitches partial rows nor guesses writer +intent from payload contents. !!! warning "Durability" Pages are not `fsync`'d individually (only `sync_data` on seal). A `SIGKILL` may lose @@ -223,17 +210,20 @@ There is no heuristic that tries to repair a half-written record. The `Compactor` drains newly-sealed hot chunks into cold segments. -- **Drain semantics.** Only `Sealed` chunks are drained (never the currently-writing chunk). Rows - are transposed to columns; the chunk's `generation` is re-checked before and after — if the ring - recycled it, the page is dropped and retried next pass. Draining is **idempotent**: a per-chunk - `drained_gen` high-water mark skips already-compacted chunk generations. -- **Rolling.** The open segment is sealed and a new one started when it reaches - `target_segment_bytes` (default 64 MiB — the main fragmentation knob), or when it exceeds - `max_segment_age` (default 300 s, so low-rate tables still become queryable), or on explicit - flush. -- **Eviction.** `enforce` deletes oldest sealed segments past a byte budget (`max_total_bytes`) or - TTL, always protecting the newest plus every unsealed/unreadable segment that another writer may - still have open. +![Compaction validates the source generation, encodes columns, then advances the watermark](../assets/architecture/probing-memc-lifecycle.svg) + +The compactor cannot hold the hot tier while compressing; cold-storage jitter would otherwise feed +back into collection. A transaction snapshots a sealed chunk identity and transposes and encodes +outside the hot path, then rereads the generation before commit. If the ring recycled the slot in +the meantime, the result is discarded. `drained_gen` advances only after a complete page append, so +failure causes a retry rather than a false claim that data was retained. + +Segment rolling is constrained by both size and age. Size bounds fragmentation and scan +granularity; age prevents low-rate tables from remaining indefinitely in an unsealed segment. An +explicit flush uses the same seal protocol. Retention deletes only the oldest sealed segments past +the capacity or TTL boundary and protects the newest and any segment another writer may still have +open. Compaction, rolling, and eviction therefore share one rule: destructive decisions apply only +to committed objects. ### Exactly-Once Across Restarts @@ -246,12 +236,11 @@ confusing a same-name replacement whose generations restart from the beginning. ## Runtime Owner -`ColdCompactor` is a process-global singleton (modeled on the task-stats worker) that gives the -compactor a single lifecycle home: - -- a background thread **rediscovers** ring files under `//` each pass (tables appear - over time), drains each into the shared `ColdStore`, rolls by age, and enforces the budget; -- on startup it calls `prime_from_cold()`; on stop it flushes (seals the open segment). +`ColdCompactor` is process-global because multiple background owners would compete for the same hot +sources and advance duplicate watermarks. Each pass rediscovers tables that appear under +`//`, drains them into one `ColdStore`, then applies rolling and budget constraints. +Startup reconstructs watermarks from cold segments; shutdown seals the open segment through the +same commit protocol. Discovery, segment enumeration, write/roll, and retention I/O are fallible operations. The worker logs failures and records an observable `CompactorRuntimeStats` snapshot (`error_count` plus the @@ -303,16 +292,9 @@ from; the hot side then **excludes** any chunk whose `(index, current generation Each row is counted exactly once, and the dedup is immune to ring recycling (the generation check re-validates). -## Configuration Reference - -| `SET memtable.*` | env | meaning | default | -|---|---|---|---| -| `cold_compaction` | `PROBING_COLD` | run the background compactor (`on`/`off`) | off | -| `cold_max_total_mb` | `PROBING_COLD_MAX_TOTAL_MB` | cold-store byte budget (MiB) | unlimited | -| `cold_ttl_secs` | `PROBING_COLD_TTL_SECS` | evict cold segments older than this | none | -| — | `PROBING_COLD_TARGET_MB` | segment roll size (MiB) | 64 | -| — | `PROBING_COLD_POLL_MS` | drain-pass interval | 2000 | -| — | `PROBING_COLD_MAX_AGE_SECS` | seal idle open segment after | 300 | +Runtime switches and capacity values are configuration contracts and are centralized under +[Environment variables — Data storage](../reference/env-vars.md#data-storage) rather than repeated +inside the storage architecture. ## Guarantees & Known Limits diff --git a/docs/src/design/data-layer.zh.md b/docs/src/design/data-layer.zh.md index af354370..f27968e5 100644 --- a/docs/src/design/data-layer.zh.md +++ b/docs/src/design/data-layer.zh.md @@ -20,26 +20,16 @@ Probing 的数据层是一个面向观测数据（指标、采样、trace）的* ## 总体架构 -```mermaid -graph LR - APP[push_row / RowWriter] --> MEMT - subgraph HOT[热层 · probing-memtable] - MEMT[MEMT 环形缓冲] --> SEALED[已封存 chunk\nmin/max ts + generation] - end - SEALED -->|转置 + Pco| ROLLER[Compactor / Roller] - ROLLER --> MEMC - subgraph COLD[冷层 · MEMC 段] - MEMC[ColdStore\n不可变段] - end - - SQL[SQL 查询] --> HCT[HotColdTable] - HCT -->|chunk 剪枝| MEMT - HCT -->|段 + page 剪枝| MEMC -``` +![MEMT 热层、MEMC 冷层与统一查询路径](../assets/architecture/probing-hot-cold-overview.svg) 查询时热层以只读方式 mmap，冷层通过 `SegmentReader` 读取。`HotColdTable` provider 将两者合并为 一次扫描，并对同时存在于两层的 chunk 做去重。 +![MEMT 写入、generation 校验与 Arrow 读取](../assets/architecture/probing-memtable-internals.svg) + +MEMT 的核心并发关系是单写者对多个 mmap 读者：写者用 generation 和 release/acquire 顺序 +发布完整行；读者获取 lease 后复核 chunk 身份，发现环形槽位已被复用时丢弃该批次并重试。 + ## 热层（MEMT） ### 文件布局 @@ -146,19 +136,23 @@ chunk 的 `min_ts`/`max_ts`。这是查询时 chunk 级时间剪枝的基础， ### 段格式 -一个段是一系列 64 对齐的 block。所有完整性校验都使用 **xxh3-64 截断为 32 位**。 +MEMC 要解决的不只是把旧行压缩到磁盘。热层 slot 会被循环复用，进程可能在任意字节处退出，读者又 +必须区分“尚未提交”和“已经提交但损坏”。因此 segment 自身承担提交协议和来源身份，而不依赖额外 +事务日志。 + +一个段是一系列 64 对齐的 block。`MCTB` 声明表与 schema，`MCPG` 保存某张表的一列 page， +`MCFT` 是封存时生成的 page 目录。所有完整性校验都使用 **xxh3-64 截断为 32 位**。 **段头部（64 字节）：** `magic`（`"MEMC"`）、`version`（2）、BOM、`flags`（bit 0 = 已封存）、 `writer_pid`、`writer_start`、`created_unix_ms`、`footer_off`（封存前为 0）、段级 `ts_min`/`ts_max`、`page_count`、头部校验和。 -**Block** 共享 64 字节头部： +![MEMC 的物理段格式、列编码与恢复路径](../assets/architecture/probing-memc-format.svg) -| magic | 含义 | -|---|---| -| `MCTB` | 表定义 block——声明一个 `table_id`、表名、列 dtype、时间戳列 | -| `MCPG` | page（数据）block——某个 `table_id` 的一列页 | -| `MCFT` | footer——封存时写入的 page 目录 | +写者依次追加段头、表定义和列 page；所有 payload 与 checksum 完成后才写 footer，最后回写 sealed +header。sealed 位因而是段级提交点。提交前崩溃，读者只承认前向扫描得到的完整 block；提交后读取， +footer、block 或 payload 任一校验失败都视为损坏。这个非对称策略来自因果区别：未封存尾部可能只是 +写者尚未完成，已封存段则已经声明自己完整，不能再把损坏解释成正常截断。 page/block 头部携带 `table_id`、`row_count`、`col_count`、`ts_min`/`ts_max`、`payload_len`、 `payload_xxh`，以及对重启去重至关重要的 `source_instance`、`source_gen` 与 `source_chunk` @@ -171,38 +165,43 @@ MEMC v2 reader 会明确拒绝 v1 段，因为 v1 缺少实例身份，无法安 单个段可容纳**多张表**的 page，以 `table_id` 区分。这让文件/目录数量与表数量解耦：成百上千张表 共享同一组段文件。 -### 列编码 +### 为什么按列编码 -每列独立编码（`ColEncoding`）： +Compactor 在离开热路径后把行转置为列，使每类数据按自己的统计特征编码。数值列使用 Pco，尤其适合 +单调时间戳；`u8` 保持 `RawFixed`，避免无收益的压缩；字符串和字节串使用带长度的 `RawVarLen`。 +编码类型固定在 page header 中，读者不需要猜测。把转置和压缩留给冷层，换来的是 MEMT 仍能保持 +逐行、无分配的简单写路径。 -- **`Pco`**——数值列（`i32/i64/f32/f64/u32/u64`），用 Pco（level 8）压缩。单调时间戳列压缩比 > 4×； -- **`RawFixed`**——`u8`（Pco 对字节列无收益）； -- **`RawVarLen`**——`Str`/`Bytes`，以连续的 `[u32 len][bytes]` 条目存储（Pco 不支持字符串）。 +### 恢复边界 -### 崩溃恢复 - -- **已封存**段通过 footer 的 page 目录读取——O(1) 定位每个 page。footer、block、payload 或 page - 的完整性校验一旦失败，整个段及 SQL 扫描都会报错，查询不会再成功返回不完整的冷层结果； -- **未封存或撕裂**的段通过**前向扫描**恢复。只有不完整的最后一个头部或 payload 会被当作崩溃尾部 - 丢弃；完整 block 的校验和损坏仍会报错。表定义 block 总会被扫描（开销小，且位于 page 之前）。 - -不存在任何试图修复半行记录的启发式逻辑。 +已封存段通过 footer 直接定位 page，任何完整性失败都会让 SQL 扫描报错；查询不会成功返回一个静默 +缺页的冷层结果。未封存段没有作出完整性承诺，因此允许前向扫描并丢弃最后一个不完整 header 或 +payload，但一个已经完整写出的 block 若 checksum 错误仍然报错。恢复不尝试拼接半行，也不根据内容 +猜测写者意图。 !!! warning "持久性" page 不会逐个 `fsync`（仅在封存时 `sync_data`）。`SIGKILL` 可能丢失当前打开段尚未刷盘的尾部 page。对观测数据可接受，但这是一个明确的取舍。 +![MEMC 恢复、保留策略与冷热查询](../assets/architecture/probing-memc-recovery-query.svg) + +恢复、TTL/容量保留和 SQL 读取共享同一 segment 格式。已封存段必须通过 footer 与 checksum +校验；未封存段只允许丢弃不完整尾部，不能把中间损坏静默解释成没有数据。 + ## Compactor（Roller） `Compactor` 将新封存的热层 chunk 徕出（drain）到冷段。 -- **徕出语义。** 只徕出 `Sealed` 状态的 chunk（绝不动正在写入的 chunk）。行被转置为列；徕出前后 - 复核该 chunk 的 `generation`——若环形已回收它，丢弃该 page 并在下一轮重试。徕出是**幂等**的： - 逐 chunk 的 `drained_gen` 高水位跳过已压缩的 chunk generation。 -- **滚动。** 当打开的段达到 `target_segment_bytes`（默认 64 MiB——主要的碎片化调节旋钮）、超过 - `max_segment_age`（默认 300 s，让低速率表也能及时可查），或显式 flush 时，封存当前段并新开一个。 -- **淘汰。** `enforce` 在超出字节预算（`max_total_bytes`）或 TTL 时删除最旧的已封存段，并始终保护 - 最新段，以及任何可能仍由其他 writer 打开的未封存/暂不可读段。 +![Compactor 复核来源 generation、按列编码并在落盘后推进水位](../assets/architecture/probing-memc-lifecycle.svg) + +Compactor 不能锁住热层等待压缩，否则冷存储抖动会反向阻塞采集。一次事务先快照 sealed chunk 的 +来源身份，再在锁外转置和编码；提交前重新读取 generation。若环形在此期间复用了 slot，当前结果 +直接作废。page 完整追加后才推进 `drained_gen`，因此失败只会导致重试，不会错误声称数据已经持久化。 + +段滚动同时受大小和年龄约束。大小上限控制文件碎片和扫描粒度，年龄上限保证低速率表不会长期停留 +在未封存段；显式 flush 使用同一个封存协议。保留策略只删除越过容量或 TTL 的最旧已封存段，始终 +保护最新段和可能仍被其他 writer 打开的段。整理、滚动和淘汰由此共享“只对已提交对象做破坏性 +决定”的边界。 ### 跨重启的精确一次 @@ -213,11 +212,9 @@ MEMC v2 reader 会明确拒绝 v1 段，因为 v1 缺少实例身份，无法安 ## 运行时 Owner -`ColdCompactor` 是进程级全局单例（仿照 task-stats worker），为 compactor 提供唯一的生命周期归宿： - -- 后台线程每轮**重新发现** `//` 下的环形文件（表会随时间出现），将每个徕出到共享的 - `ColdStore`，按时长滚动，并执行预算约束； -- 启动时调用 `prime_from_cold()`；停止时 flush（封存打开的段）。 +`ColdCompactor` 是进程级单例，因为多个后台 owner 会竞争同一热层来源并重复推进水位。后台线程每轮 +重新发现 `//` 中随运行出现的表，将它们徕出到共享 `ColdStore`，再执行滚动和预算 +约束。启动先从冷段重建 watermark，停止时用相同提交协议封存打开段。 发现、段枚举、写入/滚动与 retention I/O 都是可失败操作。worker 会记录 warning，并通过 `CompactorRuntimeStats` 暴露 `error_count` 和带操作上下文的 `last_error`。启动时的 watermark @@ -263,16 +260,8 @@ MEMC v2 reader 会明确拒绝 v1 段，因为 v1 缺少实例身份，无法安 `(source_chunk, source_gen)` 集合；热侧据此**排除**任何 `(索引, 当前 generation)` 落在该集合中的 chunk。每行恰好计数一次，且去重对环形回收免疫（generation 复核会重新验证）。 -## 配置参考 - -| `SET memtable.*` | 环境变量 | 含义 | 默认 | -|---|---|---|---| -| `cold_compaction` | `PROBING_COLD` | 运行后台 compactor（`on`/`off`） | 关闭 | -| `cold_max_total_mb` | `PROBING_COLD_MAX_TOTAL_MB` | 冷层字节预算（MiB） | 无限 | -| `cold_ttl_secs` | `PROBING_COLD_TTL_SECS` | 淘汰早于此时长的冷段 | 无 | -| — | `PROBING_COLD_TARGET_MB` | 段滚动大小（MiB） | 64 | -| — | `PROBING_COLD_POLL_MS` | 排空轮询间隔 | 2000 | -| — | `PROBING_COLD_MAX_AGE_SECS` | 空闲打开段多久后封存 | 300 | +运行时开关与容量参数属于配置契约，集中在[环境变量 — 数据存储](../reference/env-vars.zh.md#data-storage)， +不在存储架构中重复定义。 ## 保证与已知边界 diff --git a/docs/src/design/debugging.md b/docs/src/design/debugging.md deleted file mode 100644 index af4a2e7e..00000000 --- a/docs/src/design/debugging.md +++ /dev/null @@ -1,216 +0,0 @@ -# Debugging Architecture - -Probing provides powerful debugging capabilities through code injection and stack analysis. - -## Overview - -The debugging subsystem enables: - -- Live code execution in target processes -- Stack trace capture with variable inspection -- Interactive REPL sessions -- Remote debugging support - -## Code Execution - -### Eval Command - -Execute arbitrary Python code in the target process context: - -```bash -probing $ENDPOINT eval "print(model.state_dict().keys())" -``` - -### Execution Flow - -```mermaid -sequenceDiagram - participant CLI - participant Server - participant Python - - CLI->>Server: POST /eval {"code": "..."} - Server->>Python: PyRun_String(code) - Python->>Python: Execute in __main__ - Python-->>Server: Result/Exception - Server-->>CLI: JSON Response -``` - -### Execution Context - -Code executes in the main module's global namespace: - -- Access to all imported modules -- Access to global variables -- Can modify state directly - -### Safety Considerations - -- Code runs with full privileges -- No sandboxing (by design) -- Use with appropriate access controls - -## Stack Analysis - -### Backtrace Capture - -Captures current execution stack: - -```bash -probing $ENDPOINT backtrace -``` - -### Frame Information - -Each stack frame includes: - -| Field | Description | -|-------|-------------| -| func | Function name | -| file | Source file path | -| lineno | Line number | -| depth | Stack depth (0 = innermost) | -| frame_type | Python or Native | -| locals | Local variables (optional) | - -### Query Stack - -```sql -SELECT func, file, lineno, depth -FROM python.backtrace -ORDER BY depth; -``` - -## Interactive REPL - -### Starting REPL - -```bash -probing -t repl -``` - -### REPL Features - -- Tab completion -- Multi-line input -- History support -- Exception display - -### Example Session - -```python ->>> import torch ->>> model = get_model() ->>> model.training -True ->>> torch.cuda.memory_allocated() / 1024**3 -2.5 -``` - -## Remote Debugging - -### Setup - -```bash -# On target machine -PROBING_PORT=8080 python train.py - -# From remote machine -probing -t host:8080 eval "..." -``` - -### Security - -- Authentication via tokens -- TLS encryption support -- IP-based access control - -## Debugging Patterns - -### Finding Hangs - -```bash -# Capture stack -probing $ENDPOINT backtrace - -# Check where execution is blocked -probing $ENDPOINT query " -SELECT func, file, lineno -FROM python.backtrace -WHERE depth < 5" -``` - -### Inspecting State - -```bash -# Check model state -probing $ENDPOINT eval " -for name, param in model.named_parameters(): - print(f'{name}: {param.shape}')" -``` - -### Modifying Behavior - -```bash -# Change learning rate -probing $ENDPOINT eval " -optimizer.param_groups[0]['lr'] = 0.0001" - -# Force checkpoint -probing $ENDPOINT eval " -trainer.save_checkpoint('debug_checkpoint.pt')" -``` - -## Thread Debugging - -### List Threads - -```bash -probing $ENDPOINT eval " -import threading -for t in threading.enumerate(): - print(f'{t.name}: alive={t.is_alive()}')" -``` - -### Main Thread Focus - -Backtrace captures the main thread by default. For other threads: - -```bash -probing $ENDPOINT eval " -import sys, traceback -for thread_id, frame in sys._current_frames().items(): - print(f'Thread {thread_id}:') - traceback.print_stack(frame)" -``` - -## Integration with IDEs - -### VS Code - -The HTTP API can be used for IDE integration: - -```json -{ - "type": "probing", - "request": "attach", - "endpoint": "localhost:8080" -} -``` - -### Jupyter - -There is no `probing.connect()` API. From a notebook cell, shell out to the CLI -(same as any external tool): - -```python -import os, subprocess - -endpoint = os.environ["ENDPOINT"] # pid or host:port -subprocess.run( - ["probing", "-t", endpoint, "eval", "print(list(globals().keys()))"], - check=True, -) -``` - -Or use `!probing -t $ENDPOINT eval "..."` when `ENDPOINT` is set in the notebook environment. diff --git a/docs/src/design/debugging.zh.md b/docs/src/design/debugging.zh.md deleted file mode 100644 index 37d0405d..00000000 --- a/docs/src/design/debugging.zh.md +++ /dev/null @@ -1,215 +0,0 @@ -# 调试架构 - -Probing 通过代码注入和堆栈分析提供强大的调试能力。 - -## 概览 - -调试子系统支持： - -- 在目标进程中实时执行代码 -- 带变量检查的堆栈跟踪捕获 -- 交互式 REPL 会话 -- 远程调试支持 - -## 代码执行 - -### Eval 命令 - -在目标进程上下文中执行任意 Python 代码： - -```bash -probing $ENDPOINT eval "print(model.state_dict().keys())" -``` - -### 执行流程 - -```mermaid -sequenceDiagram - participant CLI - participant 服务器 - participant Python - - CLI->>服务器: POST /eval {"code": "..."} - 服务器->>Python: PyRun_String(code) - Python->>Python: 在 __main__ 中执行 - Python-->>服务器: 结果/异常 - 服务器-->>CLI: JSON 响应 -``` - -### 执行上下文 - -代码在主模块的全局命名空间中执行： - -- 访问所有已导入的模块 -- 访问全局变量 -- 可以直接修改状态 - -### 安全考虑 - -- 代码以完全权限运行 -- 无沙箱（设计如此） -- 使用适当的访问控制 - -## 堆栈分析 - -### 堆栈跟踪捕获 - -捕获当前执行堆栈： - -```bash -probing $ENDPOINT backtrace -``` - -### 帧信息 - -每个堆栈帧包括： - -| 字段 | 描述 | -|------|------| -| func | 函数名 | -| file | 源文件路径 | -| lineno | 行号 | -| depth | 堆栈深度（0 = 最内层）| -| frame_type | Python 或 Native | -| locals | 局部变量（可选）| - -### 查询堆栈 - -```sql -SELECT func, file, lineno, depth -FROM python.backtrace -ORDER BY depth; -``` - -## 交互式 REPL - -### 启动 REPL - -```bash -probing -t repl -``` - -### REPL 功能 - -- Tab 补全 -- 多行输入 -- 历史记录支持 -- 异常显示 - -### 示例会话 - -```python ->>> import torch ->>> model = get_model() ->>> model.training -True ->>> torch.cuda.memory_allocated() / 1024**3 -2.5 -``` - -## 远程调试 - -### 设置 - -```bash -# 在目标机器上 -PROBING_PORT=8080 python train.py - -# 从远程机器 -probing -t host:8080 eval "..." -``` - -### 安全 - -- 通过令牌认证 -- 支持 TLS 加密 -- 基于 IP 的访问控制 - -## 调试模式 - -### 查找卡住位置 - -```bash -# 捕获堆栈 -probing $ENDPOINT backtrace - -# 检查执行阻塞在哪里 -probing $ENDPOINT query " -SELECT func, file, lineno -FROM python.backtrace -WHERE depth < 5" -``` - -### 检查状态 - -```bash -# 检查模型状态 -probing $ENDPOINT eval " -for name, param in model.named_parameters(): - print(f'{name}: {param.shape}')" -``` - -### 修改行为 - -```bash -# 修改学习率 -probing $ENDPOINT eval " -optimizer.param_groups[0]['lr'] = 0.0001" - -# 强制保存检查点 -probing $ENDPOINT eval " -trainer.save_checkpoint('debug_checkpoint.pt')" -``` - -## 线程调试 - -### 列出线程 - -```bash -probing $ENDPOINT eval " -import threading -for t in threading.enumerate(): - print(f'{t.name}: alive={t.is_alive()}')" -``` - -### 主线程焦点 - -默认情况下，backtrace 捕获主线程。对于其他线程： - -```bash -probing $ENDPOINT eval " -import sys, traceback -for thread_id, frame in sys._current_frames().items(): - print(f'线程 {thread_id}:') - traceback.print_stack(frame)" -``` - -## 与 IDE 集成 - -### VS Code - -可通过 HTTP API 做 IDE 集成： - -```json -{ - "type": "probing", - "request": "attach", - "endpoint": "localhost:8080" -} -``` - -### Jupyter - -**没有** `probing.connect()` API。在 notebook 单元格中通过 CLI 调用（与任何外部工具相同）： - -```python -import os, subprocess - -endpoint = os.environ["ENDPOINT"] # pid 或 host:port -subprocess.run( - ["probing", "-t", endpoint, "eval", "print(list(globals().keys()))"], - check=True, -) -``` - -或在 notebook 环境已设置 `ENDPOINT` 时使用：`!probing -t $ENDPOINT eval "..."`。 diff --git a/docs/src/design/distributed-profiler.md b/docs/src/design/distributed-profiler.md new file mode 100644 index 00000000..40045833 --- /dev/null +++ b/docs/src/design/distributed-profiler.md @@ -0,0 +1,138 @@ +# Distributed Profiler Query and Visualization + +> Status: architecture design. Short-window Torch Profiler capture, +> `python.profile_capture` / `python.profile_hotspot`, and basic federation exist today. The unified +> `timeline.*` model and 10K-rank execution described here are not all implemented. + +Related foundations: [Profiling](profiling.md) · [Federation](federation.md) · +[Distributed membership](distributed.md) + +## 1. Goals and architecture + +A capture with 500,000 events on each of 10,000 ranks contains five billion rows. Centralizing all +events makes network, coordinator memory, sorting, and browser work scale with raw trace volume. + +The target is: + +> Every selected rank participates, while each tier exchanges only what the question, time window, +> and display resolution require. + +![Distributed profiler query architecture](../assets/profiler-distributed-query.svg) + +Profiler adapters normalize Torch Profiler, TorchProbe, NCCL/HCCL, Python/native stacks, and GPU/NIC +counters into timeline and call-path semantics. Rank-local reduction feeds node partials and a global +coordinator; SQL, agents, and visualizations consume the same evidence coordinates. + +The design requires complete rank participation, local reduction, continuous drill-down from job to +exact event, and shared quality metadata. It does not render one full timeline per rank on first load +or concatenate independently evaluated complex SQL and call that a global result. + +## 2. Unified data model + +The model is virtual: implementations may read trace partitions, MEMT/MEMC, summaries, or external +profiler files. + +| Entity | Meaning | +|--------|---------| +| `timeline.capture` | Per-rank participation, time range, drops, clocks, and errors | +| `timeline.track` | CPU thread, GPU stream, logical, or counter track hierarchy | +| `timeline.slice` | Step, op, kernel, memcpy, collective, or wait interval | +| `timeline.flow` | Launch, synchronization, wait, or cross-rank causality edge | +| `timeline.counter` | Time-varying GPU, NIC, CPU, or memory value | +| `timeline.tile` | Multi-resolution time/rank aggregate for overview queries | + +Capture manifests distinguish missing captures from captures with no matching event. Slices carry +step, correlation, operation, communicator, collective sequence, stack, and byte coordinates. Flows +prefer logical keys and expose method/confidence when inferred. + +![Multi-resolution timeline](../assets/profiler-timeline-pyramid.svg) + +Tiles reduce both time and rank resolution. Per-rank occupancy first unions overlapping intervals +inside each bucket; only then does the coordinator compute P50/P95/max and outlier counts. Zooming or +narrowing the rank set selects finer tiles and eventually exact slices. + +Flamegraph frames carry stable path identity, inclusive/self values, rank coverage, quantiles, +outlier count, subject/reference deltas, and a rank-set token. Compressed bitmaps or server tokens +replace per-frame arrays of 10,000 rank IDs. + +Current `profile_capture` and `profile_hotspot` remain useful capture/hotspot summary views. Full +Kineto events can map to the same track/slice/flow model without making routine SQL scan Chrome +`traceEvents`. + +## 3. Distributed query and execution + +Web and Agent clients issue a typed `TimelineQuery`; SQL remains available over virtual results. + +```yaml +scope: {capture_group_id: group-42, steps: {from: 1000, to: 1020}, ranks: all} +alignment: {kind: global_step, anchor: step_begin} +tracks: {group_by: [node, behavior_cluster], include: [step, phase, gpu, collective]} +events: {kinds: [cpu_op, gpu_kernel, collective, synchronization]} +reduce: {time: interval_occupancy, ranks: [p50, p95, max, outlier_count]} +resolution: {width_pixels: 1600, max_rows: 200, detail: auto} +output: {kind: timeline_tiles} +``` + +Alignment is explicit: wall clock, global step, collective, operation, or custom marker. Rank +selectors address all ranks, a node/role/cohort, outliers, or a small explicit set. Large selections +continue through `rank_set_token`. + +![Rank, node, and coordinator execution](../assets/profiler-timeline-execution.svg) + +| Plan | Work | +|------|------| +| Rank | filter, time prune, interval union, local top-k, folded stacks | +| Node | merge local ranks and join GPU/NIC/PCIe/NUMA context | +| Coordinator | global quantiles, outliers, behavior cohorts, views, and receipt | + +The exchange uses mergeable quantile sketches, bounded top-k states, compressed rank sets, interval +occupancy, path hashes, and collective alignment tuples. Exact slices move only for a bounded rank +set and narrow time window. Hierarchical transport follows +[coordinator → local0 → leaf](federation.md#hierarchical-fan-out), with explicit merge functions. + +Arrow batches stream results and propagate cancellation. Each result records membership epoch, +expected/seen ranks and nodes, failed partitions, rows/bytes scanned, resolution, exactness, error +bound, partial status, and elapsed time. + +## 4. Cross-rank visualization + +![Cross-rank profiler visualizations](../assets/profiler-cross-rank-visuals.svg) + +All views share capture, alignment, time window, rank-set token, subject/reference cohorts, and event +filters. + +| View | Primary question | +|------|------------------| +| Rank × Time heatmap | Which rank groups and periods are abnormal? | +| P50/P95/max timeline | When does the tail diverge from typical ranks? | +| Collective waterfall | Are ranks arriving late or transferring slowly? | +| Behavior cohorts | How many execution patterns exist, and which ranks share them? | +| Operation × Rank heatmap | Which op/kernel creates the skew? | +| Topology projection | Does the anomaly follow node, PCIe, NIC, or rail layout? | + +Behavior cohorts expose representative ranks, within-cohort variance, topology distribution, and a +rank-set token. Collective views separate predecessor compute, entry, ready, transfer, and complete. + +Flamegraphs support aggregate, differential, variance, and coverage modes. Selecting a frame opens +its contributing rank set and then exact timelines. Timeline, flamegraph, waterfall, and agent +analysis must navigate the same evidence chain rather than act as disconnected pages. + +## 5. Correctness and resource boundaries + +| Decision | Required boundary | +|----------|-------------------| +| local compute, global merge | every cross-rank operation declares mergeable state and coordinator function | +| multi-resolution first | overview returns tiles/sketches; exact events require bounded drill-down | +| interval semantics | union overlaps before wall-time aggregation | +| explicit alignment | every result records anchor and clock/error assumptions | +| explicit coverage | missing capture, no match, dropped event, and failed partition are distinct | +| bounded resources | rank/time/pixel/row/byte/time budgets constrain every query | +| traceability | findings link back to query, rank set, time window, slice, or path | + +Approximate overview results expose `exact`, `error_bound`, `coverage`, and `partial`. Unsupported +global JOINs, window functions, non-mergeable distinct counts, and per-rank LIMIT masquerading as +global top-k must fail or use an explicit coordinator plan. + +Do not emit one Chrome Trace JSON for 10,000 ranks. Full exports retain a manifest and partitions; +selected ranks may be converted to Perfetto/Chrome format. Timeline tiles, bounded exact slices, +distributed flamegraph trees, and structured SQL/Agent results all carry the same query receipt. diff --git a/docs/src/design/distributed-profiler.zh.md b/docs/src/design/distributed-profiler.zh.md new file mode 100644 index 00000000..8fae5b94 --- /dev/null +++ b/docs/src/design/distributed-profiler.zh.md @@ -0,0 +1,293 @@ +# 分布式 Profiler 查询与可视化 + +> 状态：架构设计。当前已经具备短窗口 Torch Profiler 采集、 +> `python.profile_capture` / `python.profile_hotspot` 和基础联邦查询；本文定义的是面向 +> 万 Rank Timeline、火焰图和跨 Rank 分析的统一模型，其中 `timeline.*` 尚未全部实现。 +> +> 相关基础：[性能分析](profiling.zh.md) · [联邦查询](federation.zh.md) · +> [分布式成员](distributed.zh.md) + +## 1. 目标与整体架构 + +一次短窗口如果每个 Rank 产生 50 万个 Event，一万个 Rank 就有 50 亿行。把所有 Event +集中上传再查询，会让网络、协调器内存、排序时间和浏览器渲染量都随原始数据规模增长。 + +Probing 的目标不是减少参与分析的 Rank，而是减少跨层传输的数据： + +> 所有目标 Rank 都参与计算，但只交换当前问题、时间窗口和显示分辨率需要的结果。 + +![分布式 Profiler 查询整体架构](../assets/profiler-distributed-query.svg) + +架构必须同时满足四个目标： + +| 目标 | 含义 | +|------|------| +| 全量参与 | 一次查询可以覆盖全部目标 Rank，并显式报告缺失成员 | +| 本地收敛 | Filter、区间合并、Top-K、分位数状态尽量在数据所在位置计算 | +| 连续下钻 | 从作业概览进入节点、行为组、异常 Rank，最终读取精确 Event | +| 证据一致 | SQL、可视化和 Agent 使用同一时间窗口、Rank 集合与质量信息 | + +系统不尝试在首屏返回一万个完整 Timeline，也不允许把各 Rank 独立执行的复杂 SQL 结果简单 +拼接后冒充全局 JOIN、窗口函数或分位数结果。 + +## 2. 统一数据模型 + +数据模型由 Timeline、Tile 和 Flamegraph 三组虚拟实体组成。它们是查询语义，不要求采用某种 +固定物理存储：实现可以读取 Trace 分区、MEMT/MEMC、预计算摘要或外部 Profiler 文件。 + +### 2.1 Timeline 实体 + +| 虚拟表 | 一行表示什么 | 关键关系 | +|--------|--------------|----------| +| `timeline.capture` | 某 Rank 参与一次分布式采集的状态 | `capture_group_id` 连接全作业 | +| `timeline.track` | CPU 线程、GPU Stream、逻辑轨道或 Counter 轨道 | `parent_track_id` 形成层级 | +| `timeline.slice` | Step、Op、Kernel、Memcpy、Collective 或 Wait 区间 | parent、correlation、stack | +| `timeline.flow` | Launch、同步、等待、Collective peer 等因果边 | source/target slice | +| `timeline.counter` | 某时刻的 GPU、NIC、CPU 或内存数值 | track + timestamp | + +`timeline.capture` 是所有跨 Rank 查询的入口： + +```text +capture_group_id, capture_id, run_id +rank, world_size, role, host, node_rank, local_rank +profiler_type, activities, step_begin, step_end +started_at_ns, ended_at_ns +status, events_total, events_dropped, truncated +clock_domain, clock_error_ns, error +``` + +“Rank 没有 Capture”和“Capture 中没有匹配 Event”必须是两种状态。查询热点或异常之前，先由 +Capture Manifest 确定 `ranks_expected`、`ranks_seen`、丢失事件和时钟质量。 + +`timeline.track` 与 `timeline.slice` 共同表达可查询时间线： + +```text +track: + capture_id, track_id, parent_track_id + track_kind, process_id, thread_id, device_uuid, stream_id, name + +slice: + capture_id, slice_id, parent_slice_id, track_id, rank + kind, name, normalized_name + start_ns, duration_ns, global_step, microbatch_id + correlation_id, operation_id, communicator_id, collective_seq + stack_id, bytes, attributes +``` + +Slice 保存活动区间；Flow 保存跨轨道或跨 Rank 的关系。CPU→GPU Launch、同步等待和 Collective +成员关系不能只靠时间戳猜测，应优先使用 correlation、communicator 和 sequence 等逻辑键，并 +在只能推断时返回 `method` 与 `confidence`。 + +### 2.2 多分辨率 Tile + +`timeline.tile` 是面向作业概览的结果，不是另一份原始 Trace： + +```text +capture_group_id, resolution_level +rank_group_kind, rank_group_id, rank_set_token +ranks_total, ranks_seen +bucket_start_ns, bucket_end_ns +category +occupancy_p50, occupancy_p95, occupancy_max +duration_p50_ns, duration_p95_ns +event_count, outlier_count, dominant_operation +exact, error_bound, coverage +``` + +![时间与 Rank 双轴的多分辨率 Timeline](../assets/profiler-timeline-pyramid.svg) + +Tile 同时压缩两个方向：时间轴按照像素宽度分桶，Rank 轴按照全作业、节点、并行角色、行为组 +或具体 Rank 分组。用户放大窗口或收窄 Rank 集合后，查询自动选择更细 Tile，最终才返回精确 +Slice。 + +时间占用不能直接对 Event Duration 求和。每个 Rank 必须先计算区间并集： + +```text +occupancy(rank, bucket) + = union(matching_intervals ∩ bucket) / bucket_width +``` + +然后才在 Rank 方向计算 P50、P95、Max 和异常数。这样重叠 Stream 不会被重复计时，典型 Rank +和尾部 Rank 也不会被一个平均数掩盖。 + +### 2.3 分布式火焰图 + +火焰图节点需要保存路径、消耗分布和 Rank 覆盖，而不是附加一万个 Rank ID： + +```text +path_id, parent_path_id, frame_name, frame_kind, depth +metric, inclusive_value, self_value +rank_count, rank_coverage, rank_set_token +value_p50, value_p95, value_max, outlier_count +subject_value, reference_value, delta_value, delta_ratio +exact, error_bound, coverage +``` + +Rank 集合以压缩 Bitmap 或服务端 Token 表达。调用路径通过稳定 Path Hash 在 Rank、Node 和 +Coordinator 逐级合并。 + +Torch Profiler 适配器仍可保留 `profile_capture` 和 `profile_hotspot` 作为 Capture 与热点摘要 +视图；完整 Kineto Event 通过相同语义投影为 Track、Slice、Flow，而不是要求日常 SQL 扫描 +Chrome `traceEvents`。 + +## 3. 分布式查询与执行 + +Web 和 Agent 通过类型化 `TimelineQuery` 表达问题，SQL 用于查询虚拟表和结构化结果。客户端 +不需要自己拼接万 Rank Fan-out、分位数合并或区间运算 SQL。 + +### 3.1 查询合同 + +```yaml +scope: + capture_group_id: group-42 + steps: {from: 1000, to: 1020} + ranks: all +alignment: {kind: global_step, anchor: step_begin} +tracks: + group_by: [node, behavior_cluster] + include: [step, phase, gpu, collective] +events: + kinds: [cpu_op, gpu_kernel, collective, synchronization] +reduce: + time: interval_occupancy + ranks: [p50, p95, max, outlier_count] +resolution: + width_pixels: 1600 + max_rows: 200 + detail: auto +output: {kind: timeline_tiles} +``` + +对齐方式必须显式选择： + +| 对齐 | 用途 | 边界 | +|------|------|------| +| `wall_clock` | 已校准机器之间的绝对因果 | 依赖时钟误差 | +| `global_step` | 比较同一训练 Step 的结构和时长 | 不能证明跨机绝对先后 | +| `collective` | 分析到达偏斜和传输阶段 | 依赖 communicator/sequence | +| `operation` | 比较同名第 N 次 Op/Kernel | 依赖规范化名称和匹配规则 | +| `custom_marker` | 用户定义阶段 | 依赖 Marker 覆盖率 | + +Rank 使用结构化选择器：all、node、role、behavior group、outliers 或少量显式 Rank。大集合由 +`rank_set_token` 在后续下钻中复用，避免客户端反复传输巨大整数数组。 + +### 3.2 三段执行计划 + +![Rank、Node 与 Coordinator 三段执行](../assets/profiler-timeline-execution.svg) + +| 计划 | 执行内容 | 输出 | +|------|----------|------| +| Rank | Filter、时间裁剪、区间并集、局部 Top-K、Folded Stack | 有界 Partial State | +| Node | 合并本机 Rank，关联 GPU/NIC/PCIe/NUMA，生成节点摘要 | Node Partial | +| Coordinator | 全局分位数、异常 Rank、行为组、视图与质量收据 | 查询结果 | + +网络交换的是可归并状态，而不是默认交换原始 Event： + +| 目标 | 交换状态 | +|------|----------| +| Duration 分布 | KLL / t-digest 等 Quantile Sketch | +| 热点 | 有界 Top-K State | +| Rank 覆盖 | 压缩 Bitmap / Rank Set Token | +| 时间占用 | 每 Rank 的 Interval Occupancy | +| 火焰图 | Path Hash + Metric + Distribution Sketch | +| Collective | communicator/sequence + entry/ready/complete | + +精确 Slice 只在少量 Rank 和窄时间窗下交换。分层网络拓扑复用 +[联邦查询的 Coordinator→local0→leaf 模型](federation.zh.md#hierarchical-fan-out)，但 Timeline +计划必须声明每一层的归并函数，不能退化成广播任意 SQL 后拼接。 + +### 3.3 流式结果与质量 + +Arrow RecordBatch 按 Partition 流式返回；用户改变视口或取消查询时，取消信号向 Node 和 Rank +传播。每个结果都携带统一收据： + +```text +query_id, capture_group_id, membership_epoch +ranks_expected, ranks_seen, nodes_expected, nodes_seen +partitions_scanned, failed_partitions +rows_scanned, bytes_scanned +resolution_level, exact, error_bound, partial, elapsed_ms +``` + +概览数据量由 `width_pixels`、`max_rows` 和查询预算决定，而不是由原始 Event 数量决定。 + +## 4. 跨 Rank 可视化 + +一万个 Rank 的首屏不是“一 Rank 一条完整 Timeline”，而是多种共享上下文的问题视图。 + +![跨 Rank Timeline、热力图、瀑布图与拓扑视图](../assets/profiler-cross-rank-visuals.svg) + +### 4.1 共享上下文 + +```yaml +capture_group_id: group-42 +alignment: {kind: global_step, step: 1024} +time_window: {start_ns: -2000000, end_ns: 12000000} +rank_set_token: ranks-outlier-7 +cohorts: + subject: slow-group + reference: healthy-group +filters: {event_kinds: [gpu_kernel, collective]} +``` + +时间窗、Rank 集合、Subject/Reference Cohort 和过滤器是所有视图的公共状态。用户在一个视图 +框选后，其他视图重新查询同一上下文；Agent 也接收同一个 Context，而不是重新猜测用户看到 +了哪些 Rank 和时间。 + +### 4.2 视图与问题 + +| 视图 | 表达方式 | 主要回答的问题 | +|------|----------|----------------| +| Rank × Time 热力图 | 行是节点/行为组/Rank，列是时间桶 | 异常发生在哪些 Rank 和时段 | +| 分位数 Timeline | P50/P95/Max 带状曲线 | 尾部从什么时候偏离多数 Rank | +| Collective 瀑布图 | entry→ready→transfer→complete | 是晚到还是传输变慢 | +| 行为组与代表 Timeline | 按 Signature 聚类并展示代表 Rank | 一万 Rank 中有几种执行模式 | +| Operation × Rank 热力图 | Op/Kernel 相对 Peer Baseline 的差值 | 具体哪个 Operation 产生偏斜 | +| 拓扑视图 | Node/NUMA/PCIe/NIC/Rail 空间布局 | 异常是否集中在硬件拓扑 | + +行为 Signature 可由 Step Wall Time、Compute/Communication/Idle Occupancy、Top Operation Ratio +和 Collective 特征组成。聚类输出必须保留组内方差、拓扑分布、代表 Rank 和 Rank Set Token， +不能只返回一个不可解释的 Cluster ID。 + +Collective 瀑布图至少区分：前序计算结束、进入 Collective、成员 Ready、传输和完成。少数 +Rank 晚到与所有 Rank 同时到达但 Transfer 变慢是两类不同问题，不能只用 Collective 总时长 +着色。 + +### 4.3 火焰图模式 + +| 模式 | 宽度 | 颜色 | 用途 | +|------|------|------|------| +| 聚合 | 总量或典型值 | Frame 类型 | 找整体热点 | +| 差分 | `abs(subject-reference)` | Subject 增加/减少 | 慢组多花时间在哪里 | +| 方差 | 典型消耗 | 离散度或异常比例 | 哪条路径跨 Rank 不一致 | +| 覆盖 | 调用路径权重 | Rank Coverage | 控制流是否只出现在部分 Rank | + +选择一个 Frame 后，通过 `rank_set_token` 打开贡献 Rank 的热力图，再进入代表 Rank 的精确 +Timeline。Timeline、火焰图和瀑布图必须能沿同一证据链往返，而不是三个独立页面。 + +## 5. 正确性与资源边界 + +| 设计决策 | 必须遵守的边界 | +|----------|----------------| +| 本地计算、全局归并 | 每个跨 Rank 运算声明可归并状态和 Coordinator 函数 | +| 多分辨率优先 | 首屏只返回 Tile/Sketch；精确 Event 仅用于受限下钻 | +| 区间语义 | 重叠活动先做区间并集，禁止直接累加 Duration 当作 Wall Time | +| 显式对齐 | 每个结果记录 Clock/Step/Collective/Operation Anchor 与误差 | +| 显式覆盖率 | Missing Capture、No Matching Event、Dropped Event、Failed Partition 分开表达 | +| 有界资源 | Rank 数、时间范围、像素、行数、字节数和执行时间都进入查询预算 | +| 可取消 | 视口变化后旧查询必须停止 Rank/Node 侧工作 | +| 可追溯 | 可视化结论必须能回到 Query、Rank Set、时间窗、Slice 或 Flamegraph Path | + +系统允许在概览阶段使用 Sketch 和近似聚合，但必须返回 `exact`、`error_bound`、`coverage` 和 +`partial`。近似结果不能伪装成精确值，部分失败也不能通过空结果隐藏。 + +不支持的全局关系运算必须显式失败或进入专门的 Coordinator Plan： + +- 不把 `global.a JOIN global.b` 在每个 Rank 独立执行后直接拼接成“全局 JOIN”； +- 不对每个 Rank 分别 `LIMIT K` 后声称得到全局 Top-K； +- 不把不可归并的 `count(distinct)`、窗口函数或重叠 Duration 当作普通 Sum； +- 不为一万个 Rank 生成单个 Chrome Trace JSON；完整导出使用 Manifest + 分区文件，选中少量 + Rank 后再生成 Perfetto/Chrome Trace。 + +最终输出保持四类：多分辨率 Timeline Tile、受限的精确 Slice、带 Rank 分布的 Flamegraph +Tree，以及供 SQL/Skill/Agent 使用的结构化分析结果。四类输出共享同一查询收据和证据坐标。 diff --git a/docs/src/design/distributed.md b/docs/src/design/distributed.md index 8935c9a7..6b321a5a 100644 --- a/docs/src/design/distributed.md +++ b/docs/src/design/distributed.md @@ -1,274 +1,117 @@ -# Distributed Architecture +# Distributed Membership and Control Plane -Multi-node probing: per-rank memtable writers, `cluster.nodes` registry, coordinator-side -SQL fan-out via `global.*` and `/apis/cluster/query`. +This page defines how Probing discovers and maintains probe membership for a distributed training +job. Every rank still writes local tables. Cross-rank catalogs, execution paths, and result +correctness belong to the [Federated Query Engine](federation.md). -## Related documents +> Status: implemented. Membership registration lives in `probing-server`; it does not modify torch +> rendezvous data or block `init_process_group`. -| Document | Scope | -|----------|-------| -| [Torchrun cluster heartbeat](torchrun-cluster.md) | TCPStore side channel, hierarchical PUT, backoff | -| [Federated query engine](federation.md) | Catalog rewrite, paths A/B/C, tag injection | -| [Hierarchical fan-out](hierarchical-fanout.md) | coordinator → local0 → leaf HTTP topology | -| [Cluster with Pulsing](cluster-pulsing.md) | Optional `pulsing.*` memtable membership | -| [NCCL Profiler](nccl-profiler.md) | `nccl.proxy_ops` plugin ABI | +## Overall structure -Dependency order: [Core model](../guide/concepts.md) → this page → torchrun heartbeat → federation → hierarchical fan-out. +![Hierarchical membership propagates identity while telemetry remains rank-local](../assets/architecture/probing-cluster-membership.svg) -Reference: [SQL Tables](../reference/sql-tables.md) (`cluster.nodes`, federation tags). +| Role | Owns | Does not own | +|------|------|--------------| +| leaf rank | reports endpoint, rank, and role; executes local SQL | global membership or recursive fan-out | +| local0 | aggregates on-node heartbeats; node query proxy | torch rendezvous mutation | +| global rank 0 | job membership snapshot and query entry | centralized training telemetry | +| `cluster.nodes` | endpoint membership and health | torch process-group semantics | ---- +Heartbeat carries only identity and health metadata. Collected evidence remains rank-local until a +query requests it. -## Topology +## Cluster membership lifecycle {#cluster-membership} -```mermaid -graph TB - subgraph "Node 1" - P1[Process Rank 0] - PROBE1[Probe] - end +### Startup - subgraph "Node 2" - P2[Process Rank 1] - PROBE2[Probe] - end +The Rust constructor starts `maybe_start_torchrun_cluster()` when Probing is enabled, +`WORLD_SIZE > 1`, `PROBING_TORCHRUN_CLUSTER != 0`, and the process is not an elastic supervisor. +It binds HTTP, discovers master/local0 addresses through the job TCPStore, and starts an async +heartbeat worker. Python no longer patches `torch.distributed.init_process_group`; +`setup_torchrun_cluster()` remains an explicit/test facade. - subgraph "Control Plane" - CLI[CLI Client] - AGG[Aggregator] - end +### Hierarchical registration - PROBE1 --> AGG - PROBE2 --> AGG - CLI --> AGG - CLI --> PROBE1 - CLI --> PROBE2 -``` - -## Process Discovery - -### Local Discovery - -```bash -# List all probing-enabled processes on local machine -probing list -``` - -### Remote Discovery - -```bash -# Connect to remote node -probing -t node1:8080 list -probing -t node2:8080 list -``` - -### Cluster View - -```bash -# List registered peers in the cluster view (connect to rank-0 / master endpoint) -probing -t rank0:8080 cluster nodes -``` - -## Cross-Node Queries - -### Query Single Node - -```bash -probing -t node1:8080 query " -SELECT * FROM python.torch_trace -WHERE step = (SELECT MAX(step) FROM python.torch_trace)" -``` - -### Federated query (`global.*`) - -Cross-rank SQL uses the **`global` catalog**. The master fans out to registered peers and -attaches federation tags **`_host`**, **`_addr`**, **`_rank`**, **`_role`** (parallel-role -key from the node registry, e.g. `dp=2,pp=1,tp=0`). - -**Option A — SQL engine (preferred for analytics):** - -```bash -probing -t rank0:8080 query " -SELECT _role, _rank, avg(duration_ms) AS avg_ms -FROM global.python.comm_collective -WHERE global_step > 100 -GROUP BY _role, _rank -ORDER BY avg_ms DESC" -``` - -**Option B — cluster fan-out API:** - -```bash -probing -t rank0:8080 cluster query " -SELECT _role, _rank, avg(duration_ms) AS avg_ms -FROM global.python.comm_collective -GROUP BY _role, _rank -ORDER BY avg_ms DESC" -``` - -Register peers via torchrun (Rust ctor starts hierarchical heartbeat by default — see -[torchrun cluster heartbeat](torchrun-cluster.md)), explicit `setup_torchrun_cluster()`, or -`PUT /apis/nodes` so `_rank` and -`_role` resolve correctly. Override role at runtime with `probing.set_role(...)` in training -scripts. - -See [Federated query engine](federation.md) for engine paths and acceptance tests. - -At wan scale, **`cluster query` defaults to [hierarchical fan-out](hierarchical-fanout.md)** (coordinator → per-machine local0 → on-node leaves). Set `PROBING_CLUSTER_FANOUT_HIERARCHICAL=0` or use CLI `--flat` for legacy flat fan-out. - -Raw `global.*` scans use the same topology: the coordinator reads its own partition, queries its -local leaf ranks directly, and sends node-aggregate requests to remote local0 peers. Hierarchical -execution requires `group_rank` and `local_rank` on every live registry entry; partial metadata is -reported as an error instead of falling back to a potentially incomplete flat/partial scan. - -## Synchronized Debugging - -### Capture All Stacks - -```bash -# Capture stack traces from all ranks -for node in node1 node2 node3; do - echo "=== $node ===" - probing -t $node:8080 backtrace -done -``` - -### Check Distributed State +![Leaf, local0, and global rank 0 merge heartbeat state hierarchically](../assets/architecture/probing-cluster-heartbeat.svg) -```bash -probing -t $ENDPOINT eval " -import torch.distributed as dist +TCPStore keys are isolated under: -if dist.is_initialized(): - print(f'Rank: {dist.get_rank()}') - print(f'World Size: {dist.get_world_size()}') - print(f'Backend: {dist.get_backend()}')" +```text +probing/torchrun//master +probing/torchrun//node//local0 ``` -## Communication Analysis +The store endpoint may be shared with torch rendezvous; rendezvous keys are untouched. +`PUT /apis/nodes` merges heartbeats by rank; `GET /apis/nodes` and `cluster.nodes` expose the sorted +snapshot. Registration carries rank/world size, group/local rank, host, reachable address, and role. -### Collective latency (coarse, built-in) +### Convergence and expiry -`python.comm_collective` records `torch.distributed` calls with wall time. No NCCL plugin required. +Before full membership, heartbeats stay at the base interval. Once stable, they back off +exponentially. One stale TTL marks a member dead; a second removes it. The effective maximum interval +is capped by `STALE_SEC - STALE_SEC/4 - 1`. With the default stale value 25 seconds, the safe maximum +is about 18 seconds. Increase stale to roughly 90 seconds for a stable interval near 60 seconds. -```sql -SELECT rank, op, avg(duration_ms) AS avg_ms, count(*) AS n -FROM python.comm_collective -WHERE global_step >= (SELECT max(global_step) - 20 FROM python.comm_collective) -GROUP BY rank, op -ORDER BY avg_ms DESC; -``` - -```bash -probing -t $ENDPOINT skill run slow_rank -probing -t $ENDPOINT skill run comm_bottleneck -``` - -### NCCL wait decomposition (fine-grained) - -For **culprit vs victim** attribution (`send_gpu_wait_ns` / `recv_wait_ns`), enable the NCCL profiler plugin and query `nccl.proxy_ops`: - -```bash -export NCCL_PROFILER_PLUGIN=$(python -m probing.nccl --plugin-path) -export NCCL_PROFILE_EVENT_MASK=$(python -m probing.nccl --event-mask) -export PROBING=2 -# ... torchrun ... - -probing -t $ENDPOINT skill run nccl_culprit_victim -probing -t $ENDPOINT query " -SELECT rank, sum(send_gpu_wait_ns) AS gpu_wait, sum(recv_wait_ns) AS recv_wait -FROM nccl.proxy_ops -GROUP BY rank -ORDER BY recv_wait DESC" -``` +`probing.set_role(...)` followed by `refresh_node_role()` sends an immediate update for `_role`. -Multi-node: `global.nccl.proxy_ops`. Full setup, schema, and mock workflow: [NCCL profiler plugin](nccl-profiler.md). +## Discovery and control entry points -### RDMA Flow Analysis +| Need | Entry | Scope | +|------|-------|-------| +| local probe processes | `probing list` | local sockets/processes | +| remote endpoint | `probing -t host:port list` | one endpoint | +| job snapshot | `probing -t rank0:port cluster nodes` | `cluster.nodes` | +| local SQL | `probing -t endpoint query "..."` | local `probe.*` | +| cross-rank SQL | `cluster query` or `global.*` | federation | -```bash -# RDMA-specific analysis -probing -t $ENDPOINT rdma -``` - -## Troubleshooting Distributed Issues - -### Rank Synchronization - -```bash -# Check step coordinates on each node (uses probing's step_snapshot, not trainer fields) -for node in node1 node2 node3; do - probing -t $node:8080 eval " -from probing.tracing import step_snapshot -s = step_snapshot() -print(f'rank={s.rank} local_step={s.local_step} global_step={s.global_step}')" -done -``` +HTTP reachability and Engine readiness are distinct states; connection success does not prove the +rank is ready to execute a query. -### Deadlock Detection +## Boundary with federation -```bash -# Check for hanging collective operations -probing -t $ENDPOINT query " -SELECT func, file, lineno -FROM python.backtrace -WHERE func LIKE '%collective%' OR func LIKE '%allreduce%'" -``` +![Membership freezes a peer snapshot that federation consumes](../assets/architecture/probing-membership-federation-boundary.svg) -### Memory Imbalance +Membership supplies peer identity, liveness, and rank/node/role metadata. Federation consumes one +snapshot to choose peers, inject source tags, and report coverage/partial failures. Membership does +not parse SQL or merge DataFrames; federation does not discover torchrun topology. -```sql --- Compare memory across ranks -SELECT - rank, - AVG(allocated) as avg_memory, - MAX(allocated) as peak_memory -FROM python.torch_trace -GROUP BY rank; -``` +The coordinator → local0 → leaf execution topology is defined in +[Federated Query Engine — hierarchical fan-out](federation.md#hierarchical-fan-out). ## Configuration -### Enable Remote Access - -```bash -# Start with TCP server -PROBING_PORT=8080 python train.py - -# Or configure dynamically -probing $ENDPOINT config probing.server.port=8080 -``` - -### Security - -```bash -# Enable authentication -PROBING_AUTH_TOKEN=secret python train.py - -# Connect with token -probing -t host:8080 --token secret query "..." -``` - -Use the same token on every peer. Probing applies the configured credential to internal -node discovery, heartbeat, flat federation queries, and hierarchical fan-out requests; -load-balancer health endpoints remain public. - -## Best Practices - -### Consistent environment - -Same env on all ranks (example): - -```bash -export PROBING_PORT=8080 -export PROBING_TORCH_PROFILING=on -``` - -### Cross-node timestamps - -Use NTP; memtable `ts` columns are wall-clock microseconds per process. - -### Network - -Probing HTTP is control-plane traffic. Isolate or rate-limit fan-out on large clusters; -see [Hierarchical fan-out](hierarchical-fanout.md). +| Variable | Default | Meaning | +|----------|---------|---------| +| `PROBING_TORCHRUN_CLUSTER` | `1` | initialize torchrun membership | +| `PROBING_CLUSTER_REPORT` | `1` | periodic heartbeat | +| `PROBING_CLUSTER_REPORT_BACKOFF` | `1` | back off after convergence | +| `PROBING_CLUSTER_REPORT_INTERVAL_SEC` | `10` | base interval | +| `PROBING_CLUSTER_REPORT_MAX_INTERVAL_SEC` | `120` | configured maximum, stale-capped | +| `PROBING_CLUSTER_REPORT_BACKOFF_FACTOR` | `2` | backoff factor | +| `PROBING_CLUSTER_STALE_SEC` | `25` | dead/removal TTL base | +| `PROBING_CLUSTER_DISCOVER_TIMEOUT_SEC` | `2` | TCPStore discovery timeout | +| `PROBING_CLUSTER_REPORT_TIMEOUT_SEC` | `5` | heartbeat PUT timeout | +| `PROBING_ADVERTISE_ADDR` | inferred | peer-reachable address | +| `PROBING_NODE_HOST` | inferred | host identity for grouping | + +All peers must use the same `PROBING_AUTH_TOKEN`; internal discovery, heartbeat, and query calls +carry credentials. See [Environment variables](../reference/env-vars.md#cluster). + +## Constraints and implementation + +- Heartbeat failure must not terminate the host training process. +- Training callbacks perform no heartbeat network I/O; server workers own it. +- `cluster.nodes` is endpoint membership, not a proof of an ideal torch rank set. +- External mmap schemas such as `pulsing.*` are not implicitly merged into membership. +- NTP/PTP is still required for meaningful cross-node wall-clock alignment. + +| Concern | Location | +|---------|----------| +| torchrun startup and heartbeat | `probing/server/src/torchrun_cluster.rs` | +| registry and snapshots | `probing/core/src/core/cluster.rs` | +| HTTP contract | `probing/server/API.md`, `tests/regression/spec/api_spec.json` | +| multinode example | `examples/cluster/run_multinode.sh` | + +See [Federated Query Engine](federation.md) and +[SQL Tables](../reference/sql-tables.md#cluster-nodes). diff --git a/docs/src/design/distributed.zh.md b/docs/src/design/distributed.zh.md index f2b7619e..11d5b6b2 100644 --- a/docs/src/design/distributed.zh.md +++ b/docs/src/design/distributed.zh.md @@ -1,266 +1,128 @@ -# 分布式架构 +# 分布式成员与控制面 -多节点 probing：各 rank 写本地 memtable；`cluster.nodes` 注册表；coordinator 通过 -`global.*` 与 `/apis/cluster/query` 做 SQL fan-out。 +本文说明 Probing 如何发现和维护一个分布式训练作业中的探针成员。每个 rank 仍只写本地 +表；跨 rank SQL 的 catalog、执行路径和结果正确性由[联邦查询引擎](federation.zh.md)定义。 -## 相关文档 +> 状态：当前实现。成员注册位于 `probing-server`，不会修改 torch rendezvous 数据，也不 +> 阻塞 `init_process_group`。 -| 文档 | 范围 | -|------|------| -| [torchrun 集群心跳](torchrun-cluster.zh.md) | TCPStore 旁路、分层 PUT、退避 | -| [联邦查询引擎](federation.zh.md) | Catalog 改写、路径 A/B/C、标签注入 | -| [分层集群查询](hierarchical-fanout.zh.md) | coordinator → local0 → leaf HTTP 拓扑 | -| [基于 Pulsing 的集群](cluster-pulsing.zh.md) | 可选 `pulsing.*` memtable 成员 | -| [NCCL Profiler](nccl-profiler.zh.md) | `nccl.proxy_ops` 插件 ABI | +## 1. 总体结构 -依赖顺序：[核心模型](../guide/concepts.zh.md) → 本页 → torchrun 心跳 → 联邦 → 分层 fan-out。 +![分层成员控制面传播身份，训练数据仍保留在每个 Rank](../assets/architecture/probing-cluster-membership.svg) -参考：[SQL 表目录](../reference/sql-tables.zh.md)（`cluster.nodes`、联邦标签）。 +| 角色 | 责任 | 不负责 | +|------|------|--------| +| leaf rank | 上报自身 endpoint、rank 与 role；执行本地 SQL | 维护全局成员或递归 fan-out | +| local0 | 汇总本机 rank 心跳；作为分层查询的节点代理 | 改写训练 rendezvous | +| global rank 0 | 提供作业成员快照和查询入口 | 集中保存训练采集数据 | +| `cluster.nodes` | 当前 endpoint membership 与健康状态 | 代替 torch process group | ---- +成员控制面和数据面分离：heartbeat 只传播少量身份与健康元数据，训练采集结果保留在 rank +本地，直到查询发生。 -## 拓扑 +## 2. 集群成员生命周期 {#cluster-membership} -```mermaid -graph TB - subgraph "节点 1" - P1[进程 Rank 0] - PROBE1[探针] - end +### 2.1 启动条件 - subgraph "节点 2" - P2[进程 Rank 1] - PROBE2[探针] - end +| 条件 | 当前行为 | +|------|----------| +| `PROBING=1/2` | 当前进程已启用 Probing | +| `WORLD_SIZE > 1` | 单进程不启动集群 worker | +| `PROBING_TORCHRUN_CLUSTER != 0` | 默认开启 torchrun 集群初始化 | +| `PROBING_CLUSTER_REPORT != 0` | 默认开启周期 heartbeat | +| 非 elastic supervisor | supervisor 不绑定训练 rank 的 HTTP 端口 | - subgraph "控制平面" - CLI[CLI 客户端] - AGG[聚合器] - end +Rust 动态库构造函数调用 `maybe_start_torchrun_cluster()`：绑定 HTTP、通过 job TCPStore +发现 master/local0 地址，并启动异步 heartbeat worker。Python 不再 patch +`torch.distributed.init_process_group`；`setup_torchrun_cluster()` 仅保留为显式入口和测试门面。 - PROBE1 --> AGG - PROBE2 --> AGG - CLI --> AGG - CLI --> PROBE1 - CLI --> PROBE2 -``` - -## 进程发现 +### 2.2 分层注册 -### 本地发现 - -```bash -# 列出本地机器上所有启用 probing 的进程 -probing list -``` +![leaf、local0 与 global rank 0 逐级合并 heartbeat](../assets/architecture/probing-cluster-heartbeat.svg) -### 远程发现 +TCPStore 只使用 Probing 自己的 key 前缀： -```bash -# 连接到远程节点 -probing -t node1:8080 list -probing -t node2:8080 list +```text +probing/torchrun//master +probing/torchrun//node//local0 ``` -### 集群视图 +它与 torch rendezvous 共用 endpoint，但不读写 rendezvous key。`PUT /apis/nodes` 按 rank +合并 heartbeat；`GET /apis/nodes` 和 `cluster.nodes` 返回排序后的当前快照。 -```bash -# 列出集群视图中已注册的节点（连接到 rank 0 / master 端点） -probing -t rank0:8080 cluster nodes -``` +注册至少携带 `rank`、`world_size`、`group_rank`、`local_rank`、`host`、`addr` 和 `role`。 +其中 `addr` 必须是 peer 可访问的探针地址，而不是默认假设 rendezvous 地址就是当前节点地址。 -## 跨节点查询 +### 2.3 收敛、退避与失效 -### 查询单个节点 +- 成员未凑齐时维持基础间隔，优先快速收敛； +- 全员 alive 后按因子指数退避，降低长任务的控制面开销； +- 一个 stale TTL 未收到 heartbeat 时标记 `dead`，第二个 TTL 后从视图移除； +- 实际最大 heartbeat 间隔受 stale 安全窗口约束，不能大于 + `STALE_SEC - STALE_SEC/4 - 1`。 -```bash -probing -t node1:8080 query " -SELECT * FROM python.torch_trace -WHERE step = (SELECT MAX(step) FROM python.torch_trace)" -``` +默认 `STALE_SEC=25` 时，最大安全间隔约为 18 秒；若需要稳定后约 60 秒上报，应把 stale +同时提高到至少约 90 秒。查询必须基于一次 `cluster.nodes` 快照，并在结果中报告成员失败， +不能把训练中动态变化的视图伪装成静态全集。 -### 联邦查询（`global.*`） +训练脚本调用 `probing.set_role(...)` 后，可通过 `refresh_node_role()` 立即补发 heartbeat， +使 `_role` 联邦标签及时更新。 -跨 rank SQL 使用 **`global` catalog**。Master 向已注册节点 fan-out，并为每行附加联邦标签 -**`_host`**、**`_addr`**、**`_rank`**、**`_role`**（并行角色 key，来自节点注册表，如 `dp=2,pp=1,tp=0`）。 +## 3. 发现与控制入口 -**方式 A — SQL 引擎（分析推荐）：** +| 需求 | 入口 | 范围 | +|------|------|------| +| 本机探针进程 | `probing list` | 本机 socket/process 发现 | +| 远程探针状态 | `probing -t host:port list` | 单 endpoint | +| 作业成员快照 | `probing -t rank0:port cluster nodes` | `cluster.nodes` | +| 单 rank SQL | `probing -t endpoint query "..."` | 本地 `probe.*` | +| 跨 rank SQL | `cluster query` 或 `global.*` | 交给 federation | -```bash -probing -t rank0:8080 query " -SELECT _role, _rank, avg(duration_ms) AS avg_ms -FROM global.python.comm_collective -WHERE global_step > 100 -GROUP BY _role, _rank -ORDER BY avg_ms DESC" -``` +HTTP 监听与 Engine readiness 是两个状态。成员可以先发现 endpoint，再由 readiness 判断该 +rank 是否已经能够执行查询；不能用“连接成功”替代“查询引擎已就绪”。 -**方式 B — cluster fan-out API：** +## 4. 与联邦查询的边界 -```bash -probing -t rank0:8080 cluster query " -SELECT _role, _rank, avg(duration_ms) AS avg_ms -FROM global.python.comm_collective -GROUP BY _role, _rank -ORDER BY avg_ms DESC" -``` +![成员发现输出不可变快照，联邦查询据此选择 peer 并报告覆盖率](../assets/architecture/probing-membership-federation-boundary.svg) -通过 torchrun 注入后 Rust ctor **默认**启动分层集群心跳（见 [torchrun 集群心跳](torchrun-cluster.zh.md)），或手动 `setup_torchrun_cluster()` / `PUT /apis/nodes` 注册节点，`_rank` / `_role` 才能正确解析。训练脚本中可用 `probing.set_role(...)` 运行时覆盖 role。 - -引擎实现与正确性测试要求见 **[联邦查询引擎](federation.zh.md)**。 - -万卡场景下 `cluster query` 默认走 **[分层 fan-out](hierarchical-fanout.zh.md)**（coordinator 仅联系各机 local0，local0 再聚合本机 leaf rank），可用 `PROBING_CLUSTER_FANOUT_HIERARCHICAL=0` 或 CLI `--flat` 恢复扁平 fan-out。 - -普通 `global.*` 扫描采用同一拓扑：coordinator 读取自身分区、直接查询本机 leaf ranks，并向异机 -local0 发送节点聚合请求。层级执行要求每个存活注册节点都具有 `group_rank` 与 `local_rank`；元数据 -只要部分缺失就会明确报错，不再降级为可能漏数的 flat/partial 扫描。 - -## 同步调试 - -### 捕获所有堆栈 - -```bash -# 从所有 rank 捕获堆栈跟踪 -for node in node1 node2 node3; do - echo "=== $node ===" - probing -t $node:8080 backtrace -done -``` +成员模块不解析 SQL，也不合并 DataFrame。联邦引擎不发现 torchrun 拓扑，只消费 +`cluster.nodes` 契约。万 rank 下的 coordinator → local0 → leaf 查询拓扑、失败传播和 API +字段统一见[联邦查询引擎 — 分层 fan-out](federation.zh.md#hierarchical-fan-out)。 -### 检查分布式状态 +## 5. 配置 -```bash -probing -t $ENDPOINT eval " -import torch.distributed as dist - -if dist.is_initialized(): - print(f'Rank: {dist.get_rank()}') - print(f'World Size: {dist.get_world_size()}') - print(f'Backend: {dist.get_backend()}')" -``` - -## 通信分析 - -### Collective 延迟（粗粒度，内置） - -`python.comm_collective` 记录 `torch.distributed` 调用墙钟时间，无需 NCCL 插件。 - -```sql -SELECT rank, op, avg(duration_ms) AS avg_ms, count(*) AS n -FROM python.comm_collective -WHERE global_step >= (SELECT max(global_step) - 20 FROM python.comm_collective) -GROUP BY rank, op -ORDER BY avg_ms DESC; -``` - -```bash -probing -t $ENDPOINT skill run slow_rank -probing -t $ENDPOINT skill run comm_bottleneck -``` - -### NCCL 等待分解（细粒度） - -要区分 **culprit / victim**（`send_gpu_wait_ns` / `recv_wait_ns`），需启用 NCCL profiler 插件并查询 `nccl.proxy_ops`： - -```bash -export NCCL_PROFILER_PLUGIN=$(python -m probing.nccl --plugin-path) -export NCCL_PROFILE_EVENT_MASK=$(python -m probing.nccl --event-mask) -export PROBING=2 -# ... torchrun ... - -probing -t $ENDPOINT skill run nccl_culprit_victim -probing -t $ENDPOINT query " -SELECT rank, sum(send_gpu_wait_ns) AS gpu_wait, sum(recv_wait_ns) AS recv_wait -FROM nccl.proxy_ops -GROUP BY rank -ORDER BY recv_wait DESC" -``` +| 变量 | 默认 | 作用 | +|------|------|------| +| `PROBING_TORCHRUN_CLUSTER` | `1` | 启用 torchrun 集群初始化 | +| `PROBING_CLUSTER_REPORT` | `1` | 周期性 heartbeat | +| `PROBING_CLUSTER_REPORT_BACKOFF` | `1` | 收敛后退避 | +| `PROBING_CLUSTER_REPORT_INTERVAL_SEC` | `10` | 基础间隔 | +| `PROBING_CLUSTER_REPORT_MAX_INTERVAL_SEC` | `120` | 配置上限，仍受 stale 钳制 | +| `PROBING_CLUSTER_REPORT_BACKOFF_FACTOR` | `2` | 退避倍数 | +| `PROBING_CLUSTER_STALE_SEC` | `25` | dead/移除 TTL 基准 | +| `PROBING_CLUSTER_DISCOVER_TIMEOUT_SEC` | `2` | TCPStore 发现超时 | +| `PROBING_CLUSTER_REPORT_TIMEOUT_SEC` | `5` | heartbeat PUT 超时 | +| `PROBING_PORT` | rank0 常用 `18080` | global0 固定端口；其他 rank 通常绑定随机端口 | +| `PROBING_ADVERTISE_ADDR` | 自动推断 | 对 peer 发布的可达地址 | +| `PROBING_NODE_HOST` | 自动推断 | UI 与标签使用的主机身份 | -多机使用 `global.nccl.proxy_ops`。完整说明见 [NCCL profiler 插件](nccl-profiler.zh.md)。 - -### RDMA 流分析 - -```bash -# RDMA 特定分析 -probing -t $ENDPOINT rdma -``` - -## 分布式问题排查 - -### Rank 同步 - -```bash -# 检查各节点 step 坐标（使用 probing step_snapshot，而非 trainer 字段） -for node in node1 node2 node3; do - probing -t $node:8080 eval " -from probing.tracing import step_snapshot -s = step_snapshot() -print(f'rank={s.rank} local_step={s.local_step} global_step={s.global_step}')" -done -``` - -### 死锁检测 - -```bash -# 检查挂起的集合操作 -probing -t $ENDPOINT query " -SELECT func, file, lineno -FROM python.backtrace -WHERE func LIKE '%collective%' OR func LIKE '%allreduce%'" -``` - -### 内存不均衡 - -```sql --- 比较各 rank 的内存 -SELECT - rank, - AVG(allocated) as avg_memory, - MAX(allocated) as peak_memory -FROM python.torch_trace -GROUP BY rank; -``` - -## 配置 - -### 启用远程访问 - -```bash -# 以 TCP 服务器启动 -PROBING_PORT=8080 python train.py - -# 或动态配置 -probing $ENDPOINT config probing.server.port=8080 -``` - -### 安全 - -```bash -# 启用认证 -PROBING_AUTH_TOKEN=secret python train.py - -# 带令牌连接 -probing -t host:8080 --token secret query "..." -``` - -所有 peer 必须使用相同令牌。Probing 会把已配置的凭据附加到内部节点发现、心跳、 -普通联邦查询和层级 fan-out 请求；负载均衡器使用的健康检查端点仍保持公开。 - -## 最佳实践 - -### 一致的环境变量 - -各 rank 使用相同配置（示例）： - -```bash -export PROBING_PORT=8080 -export PROBING_TORCH_PROFILING=on -``` +完整列表见[环境变量](../reference/env-vars.zh.md#集群)。所有 peer 必须使用一致的 +`PROBING_AUTH_TOKEN`；内部发现、heartbeat 与查询请求都要携带凭据，健康检查端点可保持公开。 -### 跨节点时间戳 +## 6. 设计约束与实现位置 -使用 NTP；memtable 的 `ts` 列为各进程 wall-clock 微秒。 +- heartbeat 失败不得终止宿主训练进程；按 debug/状态表暴露并重试。 +- 训练 callback 不发送 heartbeat；所有网络操作在服务端异步 worker 中执行。 +- `cluster.nodes` 是 endpoint membership，不保证等同于理想 torch rank 集合。 +- `pulsing.*` 等外部 mmap 表不会被隐式合并进 `cluster.nodes`。 +- 跨节点 wall-clock 需要 NTP/PTP；成员发现本身不提供时钟同步。 -### 网络 +| 关注点 | 位置 | +|--------|------| +| torchrun 初始化与 heartbeat | `probing/server/src/torchrun_cluster.rs` | +| 节点注册与快照 | `probing/core/src/core/cluster.rs` | +| HTTP 契约 | `probing/server/API.md`、`tests/regression/spec/api_spec.json` | +| 多机示例 | `examples/cluster/run_multinode.sh` | -Probing HTTP 属控制面流量。大集群上应隔离或限制 fan-out 速率；见 [分层 fan-out](hierarchical-fanout.zh.md)。 +查询语义见[联邦查询引擎](federation.zh.md)，表列见 +[SQL 表目录](../reference/sql-tables.zh.md#cluster-nodes)。 diff --git a/docs/src/design/extensibility.md b/docs/src/design/extensibility.md index b05c2022..ec418d0b 100644 --- a/docs/src/design/extensibility.md +++ b/docs/src/design/extensibility.md @@ -1,515 +1,137 @@ # Extensibility -Probing exposes **four public extension paths**. Everything else (Rust collectors, HTTP handlers, import hooks) is internal to the core project. - -| Path | You contribute | Consumers use | -|------|----------------|---------------| -| **1. Table plugin** | Python dataclass + `@table` | `SELECT … FROM python.*` (CLI, Web, scripts) | -| **2. Diagnostic skill** | `SKILL.md` + optional `steps.yaml` | Agent / `probing skill run …` / Web | -| **3. REPL Magic** | IPython `Magics` subclass | Python page / `probing eval` REPL | -| **4. Vendor package** | Standalone pip `probing-` | Auto-discovered skills + magics (+ optional tables) | - -**Decision rule:** expose **data** → table plugin; **how to investigate** → skill; **REPL shortcuts** → magic; **vendor/ecosystem bundle** → publish `probing-nvidia`-style package. - -```mermaid -flowchart LR - P["Table plugin\n(@table + init)"] --> T["python.my_table"] - T --> Q["SQL / CLI / Web / Agent"] - S["Diagnostic skill\n(SKILL.md)"] --> Q - T --> G["global.python.my_table\n(multi-rank)"] -``` +This page defines the public extension paths and their boundaries. Detailed `@table` methods live in +[CLI & Python API](../api-reference.md#table-dataclass-plugins), skill fields in +[Skill Format](../reference/skill-format.md), and NCCL deployment/schema in +[NCCL Profiler](nccl-profiler.md). Those details are not duplicated here. ---- +## Extension model -## Path 1: Table plugin {#path-1-table-plugin-dataclass--table} +| Extend | Public mechanism | Contract produced | +|--------|------------------|-------------------| +| training/framework data | Python `@table` plugin | `python.` | +| diagnostic method | `SKILL.md` + `steps.yaml` | reproducible SQL workflow | +| REPL shortcut | `probing.magics` entry point | IPython magic | +| vendor capability bundle | `probing-` wheel | skills, magics, optional tables | +| NCCL internal events | NCCL Profiler C ABI cdylib | `nccl.*` tables | -A table plugin is a Python module that: +New facts become tables; new analysis queries published tables. A skill must not require a server +special case, and collectors must not call each other. -1. Declares one or more **dataclass** tables with `@table` -2. Writes rows at runtime with `.save()` or `.append()` -3. Optionally defines `init()` / `deinit()` for setup and teardown +![New facts enter collectors while analysis and interaction reuse table contracts](../assets/architecture/probing-feature-placement.svg) -Registered tables appear under the **`python`** schema, e.g. `python.my_metrics`. In distributed training, query **`global.python.my_metrics`** to fan out across ranks (Probing adds `_host`, `_addr`, `_rank`, and `_role` automatically). `_role` is the source node's parallel-role key (e.g. `dp=2,pp=1,tp=0`), resolved from the cluster `nodes` registry — see [Distributed](distributed.md). +## Table plugin {#path-1-table-plugin-dataclass--table} -### Minimal example +`@table` turns a dataclass into a fixed append-only schema: ```python -# my_plugin/__init__.py from dataclasses import dataclass - from probing import table - @table @dataclass -class MyMetrics: - step: int - loss: float - - -def init(): - """Called when the plugin is loaded via python.enabled.""" - MyMetrics.init_table() - - -def deinit(): - """Called when the plugin is unloaded via python.disabled.""" - MyMetrics.drop() -``` - -Write data from training code: - -```python -MyMetrics(step=trainer.global_step, loss=loss.item()).save() -``` - -Query: - -```sql -SELECT step, avg(loss) AS avg_loss -FROM python.my_metrics -GROUP BY step -ORDER BY step; -``` - -Reference implementation: `python/probing/ext/example.py`. - -### Table naming - -- Default: class name converted to **snake_case** (`MyMetrics` → `my_metrics`) -- Explicit: `@table("custom_name")` on the dataclass - -The first appended row fixes column types. Python integers/floats use the wide -`I64`/`F64` representation, and later values are coerced to the fixed schema; -dtype variation never recreates the mmap or discards earlier rows. Changing -fields still requires a new table name or `MyMetrics.drop()` before re-init. - -### API added by `@table` - -| Method | Purpose | -|--------|---------| -| `init_table()` | Create or attach mmap backing store | -| `save()` | Append one row (instance method) | -| `append(row)` / `append_many(rows)` | Append from class | -| `take(n)` | Read last *n* rows (debugging) | -| `drop()` | Remove table | - -Storage is mmap-backed under the probing data directory. Rows survive process crashes and are visible to any attached probing client. - -### Enable a plugin - -Load the module and call `init()` by setting **`python.enabled`** to an import path (same string passed to `load_extension()`): - -```bash -# After probing is attached (PROBING=1 or probing inject) -probing -t config python.enabled=my_plugin - -# Or via SQL -probing -t query "SET python.enabled='my_plugin'" -``` - -Unload: - -```bash -probing -t config python.disabled=my_plugin -``` - -The module must be importable in the target process (installed package or on `PYTHONPATH`). Probing calls `init()` after import; `deinit()` runs on disable. - -**Alternative:** import the module directly in your training script. The `@table` decorator registers the table on import; use `init()` / `deinit()` only when you need explicit lifecycle control via `python.enabled`. - -### Framework integration - -Hook your framework inside the plugin module—still write to `@table` rows: - -```python -def init(): - MyMetrics.init_table() - import torch - torch.nn.Module.register_forward_hook(_record_module_stats) -``` - -Official integrations (torch, ray) use the same pattern internally; third-party plugins should not add separate HTTP or hook APIs. - -### Integration examples - -**Weights & Biases (bridge)** - -```python -@table("wandb_run") -@dataclass -class WandbRun: - run_id: str - step: int +class StepStats: + local_step: int + global_step: int loss: float def init(): - WandbRun.init_table() - -def on_wandb_log(step: int, loss: float): - import wandb - if wandb.run: - WandbRun(run_id=wandb.run.id, step=step, loss=loss).save() -``` - -**Custom training metrics** - -```python -@table -@dataclass -class StepStats: - step: int - lr: float - grad_norm: float -``` - -```sql -SELECT step, lr, grad_norm -FROM python.step_stats -WHERE step > (SELECT max(step) - 100 FROM python.step_stats); -``` - ---- - -## Path 2: Diagnostic skill {#path-2-diagnostic-skill} - -A **skill** packages domain knowledge for *how* to investigate a problem. It does not collect data (use Path 1 for that). Each skill is a directory with a **`SKILL.md`** the agent can read, plus an optional machine-readable step list (`steps.yaml`). - -Built-in diagnostics live under `skills//` and run via `probing skill run …`. Install into Cursor/Claude/Codex with `./skills/install.sh`. - -### Directory layout - -``` -skills/ -├── catalog.yaml # index (id, category, path) -└── my_check/ - ├── SKILL.md # required — agent + human readable - ├── steps.yaml # optional — deterministic CLI runner - └── reference.md # optional — deep dives, links -``` - -### `SKILL.md` format - -Frontmatter tells **when** to invoke the skill (for Agent routing). The body tells **how** to think about the problem. Executable steps can live in the body *or* in `steps.yaml`. - -```markdown ---- -name: my_check -description: > - Inspect custom plugin metrics in python.my_metrics. - Use when the user asks about plugin data, missing metrics, - or custom training counters from a table plugin. -category: performance -tables: [python.my_metrics] -parameters: - limit: { type: integer, default: 20 } ---- - -# My check - -## When to use - -- User enabled a table plugin but sees empty charts or SQL results -- Training runs but `python.my_metrics` has no recent rows - -## Prerequisites - -Enable the plugin in the target process: - -```bash -probing -t config python.enabled=my_plugin + StepStats.init_table() ``` -## Procedure - -1. Confirm the table exists in `information_schema.tables` -2. Fetch the last `{limit}` rows ordered by step -3. If empty, warn that the plugin is not writing or not enabled - -## Reading results - -- Steady `loss` with increasing `step` → plugin is healthy -- No rows → check `python.enabled` and that training code calls `.save()` +The training path calls `.save()`; SQL reads `python.step_stats` or +`global.python.step_stats`. Import directly, or manage the module with +`probing -t config python.enabled=`. -## Related skills +Constraints: -- `health_overview` — first triage when unsure where to start -``` - -### `steps.yaml` (optional, for deterministic runs) - -When present, CLI and Web Agent execute steps **without** relying on the LLM to invent SQL. Schema: - -```yaml -# skills/my_check/steps.yaml -apiVersion: probing.dev/v1 -kind: Skill - -metadata: - id: my_check - title: "Check my plugin metrics" - -spec: - parameters: - - name: limit - type: integer - default: 20 - - steps: - - id: recent_metrics - title: "Recent plugin rows" - type: sql - sql: | - SELECT * - FROM python.my_metrics - ORDER BY step DESC - LIMIT {limit} - on_empty: warn - empty_message: "No rows — enable plugin: python.enabled=my_plugin" - - interpretation: - rules: [] - - summary_template: | - Checked python.my_metrics (last {limit} steps). -``` +- field types are fixed after table creation; +- rows contain scalars/small structures, not model weights; +- writer failures are logged and isolated from training; +- use Probing step/rank/role coordinates and SQL JOINs across signals; +- append facts at event time rather than scanning process objects at SQL time. -Splitting **SKILL.md** (knowledge + routing) from **steps.yaml** (execution) lets agents improvise when needed while keeping reproducible runs for CI and `probing skill run`. +See [API reference](../api-reference.md#table-dataclass-plugins) and +[Environment variables](../reference/env-vars.md). -### Consumers +## Diagnostic skill {#path-2-diagnostic-skill} -```bash -probing skill list -probing -t skill run my_check -probing -t skill run slow_rank --set step_window=30 --global +Skills package what to query, how to interpret it, and what to do next: -probing skill install # skills/ -> .cursor/.claude/.agents skill dirs -./skills/install.sh -probing skill update +```text +python/probing/bundled_skills// + ├─ SKILL.md routing and interpretation + └─ steps.yaml parameterized SQL and deterministic rules ``` -Python tool API (discovery / plan only — execution is Rust CLI or MCP): - -```python -from probing.skills.tools import list_skills, plan_skill_run -plan_skill_run("health_overview") # returns CLI command + step SQL preview -``` - -**Execution SSOT:** `probing-skills` crate — CLI `probing skill run`, MCP `run_skill` / -`plan_skill`, Web Investigate Agent (WASM). Python `GET /apis/pythonext/skills/*` endpoints -are discovery-only (catalog, routing, load JSON). - -Web Agent loads skill frontmatter for routing (`description` + `tables`), injects `SKILL.md` -body into context, and runs `steps.yaml` via the shared Rust runner when present. - -### Register a skill - -1. Add `skills/my_check/SKILL.md` (required) -2. Add `skills/my_check/steps.yaml` if you want deterministic execution -3. Register in `skills/catalog.yaml` -4. Run `./skills/install.sh` so agents discover the skill -4. Validate: `python -m probing.skills validate` - -Skills may reference **any** SQL table—Path 1 plugin tables, built-in tables (`cpu.utilization`, `gpu.utilization`, `python.torch_trace`, …), and `global.*` for multi-rank fan-out. - -See `skills/README.md` and [AGENTS.md](https://github.com/DeepLink-org/probing/blob/main/AGENTS.md) for routing, install, and cluster fan-out. - ---- - -## Path 3: REPL Magic {#path-3-repl-magic} +![CLI, MCP, and Web share probing-skills](../assets/architecture/probing-skill-multiclient-runtime.svg) -**Magics** are IPython line commands (e.g. `%query`). Built-ins live under `python/probing/repl/*_magic.py`. Third parties register via the **`probing.magics`** entry point (`Magics` subclass). +| Stage | Owner | +|-------|-------| +| content SSOT | `python/probing/bundled_skills/`; root `skills/` is a symlink alias | +| discovery | Python entry point / skills HTTP API | +| load, validate, execute, interpret | Rust `probing-skills` | +| interaction | CLI, Web WASM, and MCP adapters | -Bundle magics with skills in a **`probing-`** package (Path 4) rather than a one-off unnamed pip project. +Python tools are discovery/plan only; clients must not duplicate the YAML runner. See +[Skill Format](../reference/skill-format.md) and [Diagnostic Skills](../guide/skills.md). ---- +## REPL and vendor packages {#path-3-repl-magic} -## Path 4: Vendor extension package (`probing-`) {#path-4-vendor-extension-package-probing-vendor} +REPL magics register IPython `Magics` subclasses through `probing.magics`. Third parties normally +bundle magics with skills and optional tables in one vendor wheel. -Ecosystem partners (NVIDIA, Huawei, cloud vendors, framework teams) should publish **standalone pip packages** with a unified name: +### Vendor package convention {#path-4-vendor-extension-package-probing-vendor} -| Layer | Convention | Examples | -|-------|------------|----------| -| PyPI / wheel | `probing-` (kebab-case) | `probing-nvidia`, `probing-huawei` | -| Import package | `probing_` (snake_case) | `probing_nvidia`, `probing_huawei` | -| Entry point key | Usually the vendor slug | `nvidia = "probing_nvidia:skill_root"` | - -Layout: - -``` -probing-nvidia/ -├── pyproject.toml -└── src/probing_nvidia/ - ├── __init__.py # skill_root() - ├── magics.py # probing.magics - └── skills/ - ├── catalog.yaml - └── …/ -``` - -### Discovery: entry points only - -Skills and magics use the **same** setuptools registry: - -| Group | Registers | -|-------|-----------| -| `probing.skills` | `skill_root()` → directory with `catalog.yaml` | -| `probing.magics` | `Magics` subclass | - -Use the **same vendor slug** in both groups. With `pip install -e .`, edits under `skills/` or `magics.py` apply immediately — entry points resolve to real source paths. - -`package-data` only ships files inside the wheel; probing does **not** scan it for discovery. - -Prefix skill ids and magic commands (`nvidia_nccl_triage`, `%nvidia_smi`) to avoid runtime overrides. +| Layer | Convention | Example | +|-------|------------|---------| +| wheel | `probing-` | `probing-nvidia` | +| import package | `probing_` | `probing_nvidia` | +| skill / magic ids | vendor-prefixed | `nvidia_nccl_triage` | ```toml [project] name = "probing-nvidia" -dependencies = ["probing", "ipython>=8.0"] +dependencies = ["probing"] [project.entry-points."probing.skills"] nvidia = "probing_nvidia:skill_root" [project.entry-points."probing.magics"] nvidia = "probing_nvidia.magics:NvidiaMagic" - -[tool.setuptools.package-data] -probing_nvidia = ["skills/**"] ``` -```bash -pip install -e . # dev: register once, iterate on skills/magics -python -m probing.extensions extensions -probing skill list -``` - -Template: `examples/probing-acme/`. - -Optional `@table` plugins in the same package: `python.enabled=probing_nvidia`. +Entry points are the discovery contract; package data only ships files. Template: +`examples/probing-acme/`. Optional table modules still require explicit `python.enabled` activation. ---- +## NCCL Profiler special case {#path-5-nccl-profiler-plugin} -## Path 5: NCCL profiler plugin (C cdylib) {#path-5-nccl-profiler-plugin} - -For **fine-grained NCCL wait decomposition** (culprit vs victim), use the standalone Rust profiler loaded by NCCL itself—not a Python table plugin. - -### Enable in training - -```bash -export NCCL_PROFILER_PLUGIN=$(python -m probing.nccl --plugin-path) -export NCCL_PROFILE_EVENT_MASK=$(python -m probing.nccl --event-mask) # default 94 -export PROBING=2 -torchrun --nproc_per_node=8 train.py -``` +NCCL loads this profiler through its C ABI, not Python plugin discovery. It writes `nccl.*` mmap +tables consumed by ordinary SQL, skills, and federation. ABI versions, event masks, schemas, mocks, +and hardware acceptance are maintained only in [NCCL Profiler](nccl-profiler.md). -Requires **NCCL ≥ 2.26** (PyTorch 2.8+). The plugin exports **`ncclProfiler_v4`** (NCCL 2.27+) and **`ncclProfiler_v3`** (NCCL 2.26) and writes mmap tables consumed by probing SQL: - -| Table | Content | -|-------|---------| -| `nccl.proxy_ops` | Per-proxy-op wait: `send_gpu_wait_ns` (culprit), `recv_wait_ns` (victim), `tp_rank`/`pp_rank`/`dp_rank` | -| `nccl.coll_perf` | Collective-level timing (v4) | -| `nccl.inflight_ops` | Watchdog snapshots of hung in-flight ops (`PROBING_NCCL_INFLIGHT_THRESHOLD_SECS`) | -| `nccl.net_qp` | Optional NetPlugin IB QP timing (mask bit 128) | -| `nccl.profiler_counters` | Pool/ring health counters (`pool_exhausted`, `write_errors`, …) | - -### Query and diagnose - -```sql -SELECT rank, sum(send_gpu_wait_ns), sum(recv_wait_ns) -FROM nccl.proxy_ops -GROUP BY rank -ORDER BY 3 DESC; -``` - -```bash -probing -t skill run nccl_culprit_victim -``` - -**Culprit** ranks show high `send_gpu_wait_ns` (local GPU slow); **victim** ranks show high `recv_wait_ns` (waiting on peers/network). See [NCCL profiler plugin](nccl-profiler.md) for schema, mock workflow, and smoke-test checklist. - -Coarse collective latency (`python.comm_collective`) remains available without the plugin; skills `slow_rank` and `comm_bottleneck` optionally join `nccl.proxy_ops` when present. - -### macOS / dev without NCCL - -```bash -PROBING=1 PROBING_NCCL_MOCK=1 python -m probing.nccl --seed-mock -``` - -Seeds synthetic culprit (rank 2, high `send_gpu_wait_ns`) and victim (rank 5, high `recv_wait_ns`) for skill/SQL testing. - -### Build - -```bash -make nccl-profiler-lib # Linux .so → python/probing/libs/ -``` - -Crate: `probing/extensions/nccl-profiler/`. - ---- - -## Best practices - -### Keep rows small - -Write scalars and small structs per step—not full model state or weight tensors. - -```python -# Good -MyMetrics(step=step, loss=float(loss)).save() - -# Avoid -MyMetrics(step=step, payload=model.state_dict()).save() -``` - -### Handle errors in the writer - -Sampling hooks should not crash training: - -```python -def _safe_record(step: int, loss): - try: - MyMetrics(step=step, loss=float(loss)).save() - except Exception: - pass # or log once -``` - -### Prefer append over pull - -Push rows when events happen (step end, collective complete). Do not implement “scan entire process state on every SQL query”—`@table` is append-only storage, not a lazy snapshot API. - -### Use `global.*` for multi-rank analysis - -```sql --- Align by parallel role across ranks (each row also carries a `role` column --- written at record time; `_role` is the federation tag for the source node). -SELECT _role, _rank, avg(duration_ms) AS avg_ms -FROM global.python.comm_collective -WHERE global_step > 100 -GROUP BY _role, _rank -ORDER BY avg_ms DESC; -``` - ---- +The same boundary applies: the plugin produces data and never calls skills, Web, or other collectors. ## Not public extension APIs -The following exist for core development only. **Do not** build third-party plugins on them: - -| Mechanism | Why not public | -|-----------|----------------| -| Rust `ProbeExtension` / `ProbeDataSource` | Compiled into probing; no dynamic plugin loading | -| `@ext_handler` / `/apis/pythonext/*` | Internal HTTP surface; contract-tested for core handlers | -| `add_module_callback` import hook | Used for official torch/ray integration | -| `probing-*` CLI external binaries | Separate tools, not data plugins | +| Mechanism | Status | +|-----------|--------| +| Rust `ProbeExtension` / `ProbeDataSource` | built-in contract; compiled into Probing | +| `@ext_handler` / `/apis/pythonext/*` | core HTTP implementation | +| `add_module_callback` import hook | official framework integration internal | +| `probing-*` external CLI binaries | separate tool discovery, not data plugins | -To extend Probing, use **Path 1 (table plugin)**, **Path 2 (diagnostic skill)**, **Path 3 (REPL Magic)**, **Path 4 (`probing-` package)**, or **Path 5 (NCCL profiler cdylib)** for fine-grained collective wait data. +When a third party needs control, extend a published table, skill, or documented HTTP/proto contract +instead of importing internals. See [Modularity & Boundaries](modularity.md). ---- +## Acceptance checklist -## Related docs +1. New facts are tables, not callback chains. +2. New analysis uses only SQL/documented HTTP. +3. Writer failure is isolated from the host training path. +4. Multi-rank analysis uses fixed federation tags. +5. Skills pass `python -m probing.skills validate`. +6. User-visible schemas/config/APIs update reference and contract tests. -- [Data layer](data-layer.md) — mmap memtable storage behind `python.*` -- [Distributed](distributed.md) — `global.*` federation and cluster queries -- [SQL analytics guide](../guide/sql-analytics.md) — querying patterns -- `skills/README.md` — skill authoring in depth -- `AGENTS.md` — agent install and invoke +Related: [Data Layer](data-layer.md) · [Federation](federation.md) · +[SQL Tables](../reference/sql-tables.md) · [Core model](../guide/concepts.md) diff --git a/docs/src/design/extensibility.zh.md b/docs/src/design/extensibility.zh.md index 43caacd4..7a112cd9 100644 --- a/docs/src/design/extensibility.zh.md +++ b/docs/src/design/extensibility.zh.md @@ -1,576 +1,139 @@ # 扩展机制 -Probing 提供 **四条公开扩展路径**（另有一条可选 NCCL 插件路径）。Rust 采集器、HTTP 处理器、import hook 等属于核心内部实现。 - -| 路径 | 你贡献什么 | 谁在用 | -|------|------------|--------| -| **1. 表插件** | Python dataclass + `@table` | `SELECT … FROM python.*`（CLI、Web、脚本） | -| **2. 诊断 skill** | `SKILL.md` + 可选 `steps.yaml` | Agent / `probing skill run …` / Web | -| **3. REPL Magic** | IPython `Magics` 子类 | Python 页 / `probing eval` REPL | -| **4. 厂商扩展包** | 独立 pip 包 `probing-` | 自动发现 skills + magics（+ 可选表插件） | - -**决策规则：** 要暴露**数据** → 写表插件；要贡献**怎么查** → 写 skill；要 REPL 快捷命令 → 写 magic；**厂商/生态打包** → 发布 `probing-nvidia` 这类扩展包。 - -```mermaid -flowchart LR - P["表插件\n(@table + init)"] --> T["python.my_table"] - T --> Q["SQL / CLI / Web / Agent"] - S["诊断 skill\n(SKILL.md)"] --> Q - T --> G["global.python.my_table\n(多 rank)"] -``` +本文只定义 Probing 的公开扩展路径和它们之间的边界。`@table` 的完整 API 见 +[CLI 与 Python API](../api-reference.zh.md#table-dataclass-plugins)，Skill 字段见 +[Skill 格式规范](../reference/skill-format.md)，NCCL 表和部署见 +[NCCL Profiler](nccl-profiler.zh.md)。这些细节不在本页重复维护。 ---- +## 1. 扩展模型 -## 路径 1：表插件 {#path-1-table-plugin-dataclass--table} +| 想扩展什么 | 公开机制 | 产生的契约 | +|------------|----------|------------| +| 新的训练/框架数据 | Python `@table` 插件 | `python.
` | +| 新的诊断方法 | `SKILL.md` + `steps.yaml` | 可复现 SQL 工作流 | +| REPL 快捷操作 | `probing.magics` entry point | IPython Magic | +| 厂商能力包 | `probing-` wheel | skills、magics、可选表插件 | +| NCCL 内部事件 | NCCL Profiler C ABI cdylib | `nccl.*` 表 | -表插件是一个 Python 模块，它： +![新增事实进入采集层，新的分析与交互复用现有表契约](../assets/architecture/probing-feature-placement.svg) -1. 用 `@table` 声明一个或多个 **dataclass** 表 -2. 运行时用 `.save()` 或 `.append()` 写行 -3. 可选定义 `init()` / `deinit()` 做初始化与清理 +核心原则是：数据扩展只增加表，分析扩展只查询公开表。不要为一个 skill 在 server 中加特例， +也不要让两个采集器直接调用彼此。 -注册后的表在 **`python` schema** 下，例如 `python.my_metrics`。分布式训练查询 -**`global.python.my_metrics`** 可向各 rank fan-out（自动附加 `_host`、`_addr`、`_rank`、`_role`）。 -`_role` 是来源节点的并行角色 key（如 `dp=2,pp=1,tp=0`），从 `cluster.nodes` 注册表解析——见 -[分布式](distributed.zh.md)。 +## 2. 表插件 {#path-1-table-plugin-dataclass--table} -### 最小示例 +`@table` 把 dataclass 固定为一个 append-only schema： ```python -# my_plugin/__init__.py from dataclasses import dataclass - from probing import table - @table @dataclass -class MyMetrics: - step: int - loss: float - - -def init(): - """通过 python.enabled 加载插件时调用。""" - MyMetrics.init_table() - - -def deinit(): - """通过 python.disabled 卸载时调用。""" - MyMetrics.drop() -``` - -在训练代码中写数据（step 建议用 `step_snapshot()`，见 [核心概念](../guide/concepts.zh.md)）： - -```python -from probing.tracing import step_snapshot - -snap = step_snapshot() -MyMetrics(step=snap.local_step, loss=loss.item()).save() -``` - -查询： - -```sql -SELECT step, avg(loss) AS avg_loss -FROM python.my_metrics -GROUP BY step -ORDER BY step; -``` - -参考实现：`python/probing/ext/example.py`。 - -### 表命名 - -- 默认：类名转 **snake_case**（`MyMetrics` → `my_metrics`） -- 显式：`@table("custom_name")` 写在 dataclass 上 - -首行写入后列类型固定。Python 整数/浮点数使用宽类型 `I64`/`F64`，后续值会转换到 -固定 schema；dtype 变化不会重建 mmap 或丢弃旧行。改字段仍需新表名，或先 -`MyMetrics.drop()` 再 `init_table()`。 - -### `@table` 提供的 API - -| 方法 | 用途 | -|------|------| -| `init_table()` | 创建或挂载 mmap 存储 | -| `save()` | 追加一行（实例方法） | -| `append(row)` / `append_many(rows)` | 类方法追加 | -| `take(n)` | 读最近 n 行（调试） | -| `drop()` | 删除表 | - -存储为 probing 数据目录下的 mmap，进程崩溃后仍可被 attach 的客户端查询。 - -### 启用插件 - -设置 **`python.enabled`** 为可 import 的模块路径（与 `load_extension()` 相同）： - -```bash -# 已 attach probing（PROBING=1 或 probing inject）后 -probing -t config python.enabled=my_plugin - -# 或通过 SQL -probing -t query "SET python.enabled='my_plugin'" -``` - -卸载： - -```bash -probing -t config python.disabled=my_plugin -``` - -模块须在目标进程可 import（已安装或在 `PYTHONPATH`）。加载后 probing 调用 `init()`；禁用时调用 `deinit()`。 - -**另一种方式：** 在训练脚本里直接 `import my_plugin`；`@table` 在 import 时注册表；仅当需要通过 `python.enabled` 控制生命周期时才写 `init()` / `deinit()`。 - -### 框架集成 - -在插件模块内挂框架钩子，仍写入 `@table` 行： - -```python -def init(): - MyMetrics.init_table() - import torch - torch.nn.Module.register_forward_hook(_record_module_stats) -``` - -官方 torch / ray 集成内部使用相同模式；第三方插件不要另建 HTTP 或独立 hook API。 - -### 集成示例 - -**Weights & Biases（桥接）** - -```python -@table("wandb_run") -@dataclass -class WandbRun: - run_id: str - step: int +class StepStats: + local_step: int + global_step: int loss: float def init(): - WandbRun.init_table() - -def on_wandb_log(step: int, loss: float): - import wandb - if wandb.run: - WandbRun(run_id=wandb.run.id, step=step, loss=loss).save() -``` - -**自定义训练指标** - -```python -@table -@dataclass -class StepStats: - step: int - lr: float - grad_norm: float -``` - -```sql -SELECT step, lr, grad_norm -FROM python.step_stats -WHERE step > (SELECT max(step) - 100 FROM python.step_stats); -``` - ---- - -## 路径 2：诊断 skill {#path-2-diagnostic-skill} - -**Skill** 打包**如何排查**的领域知识，本身不采集数据（数据用路径 1）。每个 skill 是一个目录：Agent 可读的 **`SKILL.md`**，加上可选的机器可读步骤列表（`steps.yaml`）。 - -内置诊断位于 `skills//`，通过 `probing skill run …` 执行。安装到 Cursor/Claude/Codex：`./skills/install.sh` 或 `probing skill install`。 - -### 目录结构 - -``` -skills/ -├── catalog.yaml # 索引（id、category、path） -└── my_check/ - ├── SKILL.md # 必需 — Agent + 人类可读 - ├── steps.yaml # 可选 — 确定性 CLI 执行 - └── reference.md # 可选 — 深入说明、外链 -``` - -### `SKILL.md` 格式 - -Frontmatter 说明**何时**调用该 skill（供 Agent 路由）。正文说明**如何**思考问题。可执行步骤可写在正文或 `steps.yaml`。 - -```markdown ---- -name: my_check -description: > - 检查 python.my_metrics 中的自定义插件指标。 - 当用户询问插件数据、指标缺失或表插件计数器时使用。 -category: performance -tables: [python.my_metrics] -parameters: - limit: { type: integer, default: 20 } ---- - -# My check - -## 何时使用 - -- 用户已启用表插件但图表或 SQL 结果为空 -- 训练在跑但 `python.my_metrics` 无近期行 - -## 前置条件 - -在目标进程启用插件： - -```bash -probing -t config python.enabled=my_plugin + StepStats.init_table() ``` -## 步骤 +训练路径调用 `StepStats(...).save()`；查询侧读取 `python.step_stats`，多 rank 时读取 +`global.python.step_stats`。插件可以由训练脚本直接 import，也可以通过 +`probing -t config python.enabled=` 管理生命周期。 -1. 在 `information_schema.tables` 中确认表存在 -2. 按 step 倒序取最近 `{limit}` 行 -3. 若为空，提示插件未写入或未启用 +边界要求： -## 结果解读 +- 字段首次创建后类型固定；破坏 schema 时使用新表名或显式重建。 +- 行应是标量和小结构，不保存模型权重或大 payload。 +- writer 失败应被隔离并记录日志，不能拖垮训练。 +- step/rank/role 使用 Probing 的公共坐标，跨数据源关系通过 SQL JOIN 表达。 +- `@table` 是追加事实，不是“每次 SQL 时扫描进程对象”的 pull API。 -- `loss` 随 `step` 递增 → 插件健康 -- 无行 → 检查 `python.enabled` 与训练代码是否调用 `.save()` +方法、命名、容量与启停参数见 [API 参考](../api-reference.zh.md#table-dataclass-plugins)和 +[环境变量](../reference/env-vars.zh.md)。 -## 相关 skill +## 3. 诊断 Skill {#path-2-diagnostic-skill} -- `health_overview` — 不确定从哪查时先做分诊 -``` - -### `steps.yaml`（可选，确定性执行） - -存在时，CLI 与 Web Agent **不依赖 LLM 编造 SQL** 即可执行。Schema 示例： - -```yaml -# skills/my_check/steps.yaml -apiVersion: probing.dev/v1 -kind: Skill - -metadata: - id: my_check - title: "检查我的插件指标" - -spec: - parameters: - - name: limit - type: integer - default: 20 - - steps: - - id: recent_metrics - title: "近期插件行" - type: sql - sql: | - SELECT * - FROM python.my_metrics - ORDER BY step DESC - LIMIT {limit} - on_empty: warn - empty_message: "无数据 — 请启用插件: python.enabled=my_plugin" - - interpretation: - rules: [] - - summary_template: | - 已检查 python.my_metrics（最近 {limit} 个 step）。 -``` - -**SKILL.md**（知识 + 路由）与 **steps.yaml**（执行）分离：Agent 可即兴发挥，CI 与 `probing skill run` 保持可复现。 - -### 使用方 - -```bash -probing skill list -probing -t skill run my_check -probing -t skill run slow_rank --set step_window=30 --global +Skill 不采集数据，它把“应该查什么、如何解释、下一步做什么”封装为版本化工作流： -probing skill install # skills/ → .cursor/.claude/.agents -./skills/install.sh -probing skill update +```text +python/probing/bundled_skills// + ├─ SKILL.md 路由、适用场景与解释 + └─ steps.yaml 参数化 SQL、空结果语义与确定性规则 ``` -Python 工具 API（仅发现 / 展开计划 — 执行走 Rust CLI 或 MCP）： - -```python -from probing.skills.tools import list_skills, plan_skill_run -plan_skill_run("health_overview") # 返回 CLI 命令与步骤 SQL 预览 -``` - -**执行 SSOT：** `probing-skills` crate — CLI `probing skill run`、MCP `run_skill` / -`plan_skill`、Web Investigate Agent（WASM）。Python `GET /apis/pythonext/skills/*` 仅为 -发现接口（catalog、routing、load JSON）。 - -Web Agent 用 frontmatter 做路由（`description` + `tables`），将 `SKILL.md` 正文注入上下文； -有 `steps.yaml` 时由共享 Rust runner 按步骤执行。 +![CLI、MCP 和 Web 复用 probing-skills 运行时](../assets/architecture/probing-skill-multiclient-runtime.svg) -### 注册新 skill - -1. 添加 `skills/my_check/SKILL.md`（必需） -2. 需要确定性执行时添加 `steps.yaml` -3. 在 `skills/catalog.yaml` 登记 -4. 运行 `./skills/install.sh` 让 Agent 发现 -5. 校验：`python -m probing.skills validate` - -Skill 可引用**任意** SQL 表——路径 1 插件表、内置表（`cpu.utilization`、`python.torch_trace` 等）、以及多 rank 的 `global.*`。 - -详见 `skills/README.md`、[诊断 Skill 用户指南](../guide/skills.zh.md)、[AGENTS.md](https://github.com/DeepLink-org/probing/blob/main/AGENTS.md)。 - ---- - -## 路径 3：REPL Magic {#path-3-repl-magic} - -**Magic** 是 IPython REPL 里的行魔法（如 `%query`）。内置 magic 在 `python/probing/repl/*_magic.py`；第三方通过 **`probing.magics`** entry point 注册 `Magics` 子类。 - -```python -from IPython.core.magic import Magics, line_magic, magics_class - -@magics_class -class NvidiaMagic(Magics): - @line_magic - def nvsmi(self, line: str): - ... -``` - -推荐与 skills 一起打进 **`probing-`** 包（路径 4），而不是单独起一个无名 pip 包。 +| 阶段 | 唯一责任方 | +|------|------------| +| 内容 SSOT | `python/probing/bundled_skills/`；根 `skills/` 是符号链接别名 | +| 发现 | Python entry point / skills HTTP API | +| 加载、参数校验、执行、解释 | Rust `probing-skills` | +| 交互适配 | CLI、Web WASM、MCP 各自负责传输和展示 | ---- +因此 Python 的 skills 工具只做发现/计划，不形成另一套 runner；CLI、Web 和 MCP 也不能各自 +复制 YAML 解释逻辑。完整字段、示例和校验命令统一见 +[Skill 格式规范](../reference/skill-format.md)与[诊断 Skill 指南](../guide/skills.zh.md)。 -## 路径 4：厂商扩展包（`probing-`） {#path-4-vendor-extension-package-probing-vendor} +## 4. REPL 与厂商扩展包 {#path-3-repl-magic} -**推荐**第三方（NVIDIA、华为、云厂商、框架团队）以 **独立 pip 包** 贡献 skills、magics，以及可选的表插件。统一命名，便于发现与运维。 +REPL Magic 通过 `probing.magics` 注册 IPython `Magics` 子类。第三方通常不单独发布 magic， +而是与 skill 和可选表插件一起放进厂商 wheel。 -### 命名约定 +### 厂商包约定 {#path-4-vendor-extension-package-probing-vendor} -| 层级 | 规则 | 示例 | +| 层级 | 约定 | 示例 | |------|------|------| -| PyPI / wheel 名 | `probing-`（小写 kebab-case） | `probing-nvidia`、`probing-huawei` | -| Python 导入包 | `probing_`（snake_case） | `probing_nvidia`、`probing_huawei` | -| entry point 键 | 通常与 vendor 一致 | `nvidia = "probing_nvidia:skill_root"` | - -不要用 `nvidia-probing` 或 `probing_ext_nvidia` 这类变体——`python -m probing.extensions extensions` 只索引 **`probing-*`** 发行包。 - -### 包内布局 - -``` -probing-nvidia/ -├── pyproject.toml -└── src/probing_nvidia/ - ├── __init__.py # skill_root() — probing.skills entry point - ├── magics.py # probing.magics entry point - ├── tables.py # 可选 @table 插件 - └── skills/ - ├── catalog.yaml - └── nccl_deep_dive/ - ├── SKILL.md - └── steps.yaml -``` - -### 统一发现：entry point 注册表 - -Skills 与 magics **都只通过 setuptools entry point 发现**（同一套机制）： - -| Entry point group | 注册内容 | -|-------------------|----------| -| `probing.skills` | `skill_root()` → 含 `catalog.yaml` 的目录 | -| `probing.magics` | `Magics` 子类 | - -厂商包在 **两个 group 里用同一个 vendor slug**（如 `nvidia`）。`pip install -e .` 开发模式下改 `skills/` 或 `magics.py` **无需重装**，entry point 指向源码树里的真实路径。 - -`[tool.setuptools.package-data]` 仅用于把 `skills/**` **打进 wheel**；**不会**被 probing 扫描发现——必须写 entry point。 - -### 命名与冲突 - -- **entry point 键**：各包独立注册，多包并存无妨（`nvidia`、`huawei` 各一条） -- **skill id / %magic 命令**：应用层命名空间，须 vendor 前缀（`nvidia_nccl_triage`、`%nvidia_smi`），避免后者覆盖前者 - -### `pyproject.toml` 最小示例 +| wheel | `probing-` | `probing-nvidia` | +| import package | `probing_` | `probing_nvidia` | +| skill / magic id | 带 vendor 前缀 | `nvidia_nccl_triage` | ```toml [project] name = "probing-nvidia" -dependencies = ["probing", "ipython>=8.0"] +dependencies = ["probing"] [project.entry-points."probing.skills"] nvidia = "probing_nvidia:skill_root" [project.entry-points."probing.magics"] nvidia = "probing_nvidia.magics:NvidiaMagic" - -[tool.setuptools.package-data] -probing_nvidia = ["skills/**"] -``` - -```python -# probing_nvidia/__init__.py -from pathlib import Path - -def skill_root() -> Path: - return Path(__file__).resolve().parent / "skills" -``` - -Magics 在训练进程 REPL 启动时通过 `probing.magics` 注册；skills 合并进全局 catalog，CLI / Web / MCP 无需改 Rust 二进制。 - -### 开发模式 - -```bash -cd probing-nvidia -pip install -e . # 一次注册 entry points -# 编辑 src/probing_nvidia/skills/ 或 magics.py，立即生效 -python -m probing.extensions extensions -probing skill list -``` - -### 安装与验证（用户） - -```bash -pip install probing probing-nvidia - -python -m probing.extensions extensions # 列出已装厂商包 -python -m probing.extensions skill-roots # 含 nvidia skills 目录 -probing skill list # 含厂商 skill id ``` -attach 后 HTTP：`GET /apis/pythonext/extensions/list` - -### 可选：表插件 - -同一包内可再提供 `@table` 模块，用户启用： - -```bash -probing -t config python.enabled=probing_nvidia -``` - -### 模板 - -仓库内可复制起点：`examples/probing-acme/`（虚构厂商 **acme**，结构同 `probing-nvidia` / `probing-huawei`）。 +entry point 是发现契约；`package-data` 只负责把文件打进 wheel，不能替代注册。开发模板见 +`examples/probing-acme/`。厂商包中的 `@table` 模块仍由 `python.enabled` 显式启用，不因安装 +wheel 就自动进入训练热路径。 ---- +## 5. NCCL Profiler 特例 {#path-5-nccl-profiler-plugin} -## 路径 5：NCCL profiler 插件（C cdylib） {#path-5-nccl-profiler-plugin} +NCCL Profiler 由 NCCL runtime 通过 C ABI 加载，不是 Python 动态插件。它写 `nccl.*` mmap +表，再由普通 SQL、Skill 和联邦查询消费。ABI 版本、事件 mask、表 schema、mock 和真机验收 +只在 [NCCL Profiler](nccl-profiler.zh.md) 维护，本页不复制。 -**细粒度 NCCL 等待分解**（culprit vs victim）使用 NCCL 自身加载的独立 Rust profiler，**不是** Python 表插件。 +这个特例仍遵守相同上层边界：插件只产出数据，不调用 Skill、Web 或其他 collector。 -### 在训练中启用 +## 6. 哪些不是公开扩展 API -```bash -export NCCL_PROFILER_PLUGIN=$(python -m probing.nccl --plugin-path) -export NCCL_PROFILE_EVENT_MASK=$(python -m probing.nccl --event-mask) # 默认 94 -export PROBING=2 -torchrun --nproc_per_node=8 train.py -``` - -需要 **NCCL ≥ 2.26**（PyTorch 2.8+）。插件仅导出 `ncclProfiler_v3`，写入 mmap 表供 probing SQL 查询： - -| 表 | 内容 | -|----|------| -| `nccl.proxy_ops` | 每 proxy-op 等待：`send_gpu_wait_ns`（culprit）、`recv_wait_ns`（victim）、`tp_rank`/`pp_rank`/`dp_rank` | -| `nccl.net_qp` | 可选 NetPlugin IB QP 计时（mask bit 128） | - -> **与 `role` 的区别：** 训练表（`torch_trace`、`comm_collective`）用 probing **`role`** 字符串；`nccl.proxy_ops` 仍保留 NCCL 插件侧的 tp/pp/dp 列。 - -### 查询与诊断 - -```sql -SELECT rank, sum(send_gpu_wait_ns), sum(recv_wait_ns) -FROM nccl.proxy_ops -GROUP BY rank -ORDER BY 3 DESC; -``` - -```bash -probing -t skill run nccl_culprit_victim -``` - -**Culprit** rank `send_gpu_wait_ns` 高（本机 GPU 慢）；**victim** rank `recv_wait_ns` 高（等 peer/网络）。详见 [NCCL profiler 插件](nccl-profiler.zh.md)。 - -粗粒度 collective 延迟（`python.comm_collective`）无需该插件；`slow_rank`、`comm_bottleneck` 在表存在时会可选 JOIN `nccl.proxy_ops`。 - -### macOS / 无 NCCL 开发环境 - -```bash -PROBING=1 PROBING_NCCL_MOCK=1 python -m probing.nccl --seed-mock -``` - -注入合成 culprit（rank 2，高 `send_gpu_wait_ns`）与 victim（rank 5，高 `recv_wait_ns`）供 skill/SQL 测试。 - -### 构建 - -```bash -make nccl-profiler-lib # Linux .so → python/probing/libs/ -``` - -Crate：`probing/extensions/nccl-profiler/`。 - ---- - -## 最佳实践 - -### 控制行大小 - -每 step 写标量与小结构——不要写完整模型 state 或权重张量。 - -```python -# 推荐 -MyMetrics(step=step, loss=float(loss)).save() - -# 避免 -MyMetrics(step=step, payload=model.state_dict()).save() -``` - -### 写入路径容错 - -采样钩子不应拖垮训练： - -```python -def _safe_record(step: int, loss): - try: - MyMetrics(step=step, loss=float(loss)).save() - except Exception: - pass # 或只 log 一次 -``` - -### 推送优于拉取 - -在事件发生时写行（step 结束、collective 完成）。不要实现「每次 SQL 查询扫描整个进程状态」——`@table` 是 append-only 存储，不是惰性快照 API。 - -### 多 rank 分析用 `global.*` - -```sql --- 按并行 role 跨 rank 对齐（行内还有采集时的 role 列；_role 是联邦标签） -SELECT _role, _rank, avg(duration_ms) AS avg_ms -FROM global.python.comm_collective -WHERE global_step > 100 -GROUP BY _role, _rank -ORDER BY avg_ms DESC; -``` - ---- - -## 非公开扩展 API - -下列机制仅供核心开发，**第三方插件请勿依赖**： - -| 机制 | 为何不公开 | -|------|------------| -| Rust `ProbeExtension` / `ProbeDataSource` | 编译进 probing；无动态插件加载 | -| `@ext_handler` / `/apis/pythonext/*` | 内部 HTTP；核心 handler 契约测试 | -| `add_module_callback` import hook | 官方 torch/ray 集成使用 | -| `probing-*` CLI 外部二进制 | 独立工具，非数据插件 | +| 机制 | 定位 | +|------|------| +| Rust `ProbeExtension` / `ProbeDataSource` | 内置模块契约，需要编译进 Probing | +| `@ext_handler` / `/apis/pythonext/*` | 核心 HTTP 实现，不是第三方稳定接口 | +| `add_module_callback` import hook | 官方框架集成内部能力 | +| `probing-*` 外部 CLI 二进制 | 独立工具发现，不等同于数据插件 | -扩展 Probing 请用 **路径 1（表插件）**、**路径 2（诊断 skill）**、**路径 3（REPL Magic）**、**路径 4（`probing-` 扩展包）**，或 **路径 5（NCCL profiler cdylib）** 获取细粒度 collective 等待数据。 +第三方需要新的控制能力时，应先扩展公开表、Skill 或文档化 HTTP/proto 契约，而不是依赖内部 +模块。内置 crate 的依赖和 ownership 见[模块化与边界](modularity.zh.md)。 ---- +## 7. 验收检查 -## 相关文档 +1. 新事实是否通过表暴露，而不是 callback 链？ +2. 新分析是否只使用 SQL/公开 HTTP？ +3. 写入失败是否与训练主路径隔离？ +4. 多 rank 是否使用固定的 `_rank`、`_role` 等联邦标签？ +5. Skill 是否通过 `python -m probing.skills validate`？ +6. 用户可见 schema、配置或接口是否同步参考手册和契约测试？ -- [数据层](data-layer.zh.md) — `python.*` 背后的 mmap memtable -- [分布式](distributed.zh.md) — `global.*` 联邦与 cluster 查询 -- [SQL 分析指南](../guide/sql-analytics.zh.md) — 查询模式 -- [SQL 表目录](../reference/sql-tables.zh.md) -- [核心概念](../guide/concepts.zh.md) -- [诊断 Skill 指南](../guide/skills.zh.md) -- `skills/README.md` — skill 编写详解 -- `AGENTS.md` — Agent 安装与调用 +相关文档：[数据层](data-layer.zh.md) · [联邦查询](federation.zh.md) · +[SQL 表目录](../reference/sql-tables.zh.md) · [核心模型](../guide/concepts.zh.md) diff --git a/docs/src/design/federation.md b/docs/src/design/federation.md index c28988ce..3d6ef40a 100644 --- a/docs/src/design/federation.md +++ b/docs/src/design/federation.md @@ -224,17 +224,7 @@ Implementation: `probing/core/src/core/federation/`; `cluster=true` routing: ### 4.1 Pipeline -```mermaid -flowchart LR - IN[User SQL] --> CF{cluster?} - CF -->|no| L0[Local DataFusion / probe.*] - CF -->|yes| RT[Path A/B/C] - RT --> RW[Catalog rewrite] - RW --> EX[Execute shards] - EX --> TG[Inject federation tags] - TG --> MG[Merge / re-aggregate] - MG --> OUT[QueryOutcome DataFrame + QueryQuality] -``` +![Federated query pipeline from SQL routing to global merge](../assets/architecture/probing-federation-pipeline.svg) **Conventions** @@ -252,7 +242,7 @@ flowchart LR | Partial HTTP | Cluster/fan-out APIs return **503** with partial `dataframe` when `meta.partial=true` (non-strict only) | | Peer 503 accept | Non-strict mode may accept peer HTTP 503 bodies with partial data during hierarchical merge | | Federated scan logs | Peer drops log at **debug** by default; **warn** when `PROBING_FANOUT_STRICT=1` | -| Hierarchical fan-out | Default on: coordinator → local0 → leaves; see [Hierarchical fan-out](hierarchical-fanout.md) | +| Hierarchical fan-out | Default on: coordinator → local0 → leaves; see [§4.11](#hierarchical-fan-out) | ### 4.2 Path selection @@ -260,16 +250,7 @@ flowchart LR `cluster=true` — **AST-based** (not substring): -```mermaid -flowchart TD - Q[SQL] --> P{Single SELECT?} - P -->|no| C[Path C] - P -->|yes| M{Single table, no JOIN/CTE/UNION/subquery?} - M -->|no| C - M -->|yes| A{global.* + pushdown-safe GROUP BY/agg?} - A -->|yes| PA[Path A] - A -->|no| PB[Path B] -``` +![AST-based selection of aggregate pushdown, federated scan, or broadcast](../assets/architecture/probing-federation-path-selection.svg) | Path | When | §3 examples | |------|------|-------------| @@ -385,6 +366,44 @@ At large `world_size`, prefer path A partials plus coordinator re-aggregation ov | Compute vs comm | C | `probe.comm JOIN probe.torch_trace`; then `GROUP BY _rank` | | Hang / backtrace | A or B | Single-table `GROUP BY _rank` or filtered raw | +### 4.11 Hierarchical fan-out {#hierarchical-fan-out} + +Paths A/B/C define relational work at shards and the coordinator. Hierarchical fan-out defines how +those shard requests reach 10K ranks without requiring `O(world_size)` coordinator connections. + +![Hierarchical query fan-out](../assets/architecture/probing-fanout-internals.svg) + +| Tier | Fan-out targets | Typical size at 8 ranks/node, 1024 nodes | +|------|-----------------|------------------------------------------| +| coordinator | one `local_rank=0` per `group_rank` | about 1023 remote nodes | +| node/local0 | leaves in the same `group_rank` | about 7 per node | +| leaf | none; local SQL only | 1 | + +`POST /apis/cluster/query` carries `cluster`, `hierarchical` (default true), and `scope`: +`auto`, `coordinator`, `node`, or `local`. Coordinator scope merges local and remote nodes; node +scope merges one machine; local scope forbids further fan-out. Peers always execute `probe.*`, so +scope prevents recursion rather than serving as a mere optimization hint. + +Hierarchical execution requires complete `group_rank`, `local_rank`, and `addr` metadata in +`cluster.nodes`. Missing topology returns HTTP 503 instead of silently falling back to a potentially +incomplete flat scan. Flat mode is explicit through `hierarchical=false`, CLI `--flat`, or +`PROBING_CLUSTER_FANOUT_HIERARCHICAL=0`. + +Every tier propagates `nodes_queried`, `nodes_failed`, `node_aggregators_queried`, and +`local_ranks_queried`. A remote local0 must include its leaf coverage rather than appearing as one +opaque successful endpoint. Non-strict mode may merge a lower-tier HTTP 503 partial body; +`PROBING_FANOUT_STRICT=1` fails on any missing peer or dropped batch. + +| Federation path | Hierarchical behavior | +|-----------------|-----------------------| +| A aggregate pushdown | node merges on-node partials; coordinator performs global merge | +| B federated scan | coordinator creates remote lazy partitions only for node aggregators | +| C broadcast | each node runs the SQL on local ranks and concatenates before returning | + +Implementation: `probing/server/src/server/cluster_fanout.rs`, `cluster_query.rs`, and +`probing/core/src/core/federation/fanout_scope.rs`. Membership comes from +[Distributed Membership and Control Plane](distributed.md#cluster-membership). + --- ## 5. Regression queries @@ -409,8 +428,7 @@ Engine changes must pass these on mock multi-node and real clusters: | Document | Content | |----------|---------| -| [Distributed overview](distributed.md) | `cluster query` usage | -| [Hierarchical fan-out](hierarchical-fanout.md) | Wan-scale HTTP topology | +| [Distributed membership](distributed.md) | Discovery, heartbeat, and topology metadata | | [Core model](../guide/concepts.md) | Catalogs, federation tags | | [SQL Tables](../reference/sql-tables.md) | Columns and `cluster.nodes` | | [NCCL Profiler](nccl-profiler.md) | §3.1 ⑤, `nccl.proxy_ops` | diff --git a/docs/src/design/federation.zh.md b/docs/src/design/federation.zh.md index 3148c1de..aabaa121 100644 --- a/docs/src/design/federation.zh.md +++ b/docs/src/design/federation.zh.md @@ -33,6 +33,11 @@ | Web Training 热力图 | `GET /apis/training/step_matrix?cluster=true` | 是 | | 进程内 | rank 0 上 `probing.query("… global.…")` | 视 SQL | +![global catalog 的分层 fan-out 与协调器合并](../assets/architecture/probing-query-federation.svg) + +图中的 `global.*` 是逻辑视图：peer 永远执行 `probe.*`。可归并聚合在数据侧先收敛，协调器 +注入来源标签并完成全局合并；不能正确拆分的表达式不得伪装成全局语义。 + --- ## 2. 两个 Catalog 与联邦标签 @@ -231,17 +236,7 @@ WHERE func LIKE '%collective%' OR func LIKE '%nccl%'; ### 4.1 处理流水线 -```mermaid -flowchart LR - IN[用户 SQL] --> CF{cluster?} - CF -->|否| L0[本地 DataFusion / probe.*] - CF -->|是| RT[路径选型 A/B/C] - RT --> RW[Catalog 改写] - RW --> EX[各分片执行] - EX --> TG[注入联邦标签] - TG --> MG[合并 / 二次聚合] - MG --> OUT[QueryOutcome DataFrame + QueryQuality] -``` +![联邦查询从 SQL 路由到全局合并的处理流水线](../assets/architecture/probing-federation-pipeline.svg) **统一约定** @@ -259,7 +254,7 @@ flowchart LR | Partial HTTP | 集群/fan-out API 在 `meta.partial=true` 时返回 **503** 及 partial `dataframe`（非 strict 模式） | | Peer 503 接受 | 非 strict 模式下，分层 merge 可接受 peer HTTP 503 的 partial body | | 联邦 scan 日志 | 默认 peer 丢弃记 **debug**；`PROBING_FANOUT_STRICT=1` 时记 **warn** | -| 分层 fan-out | 默认开启：`coordinator → 各机 local0 → 本机 leaf`；见 [分层集群查询](hierarchical-fanout.zh.md) | +| 分层 fan-out | 默认开启：`coordinator → 各机 local0 → 本机 leaf`；见 [§4.11](#hierarchical-fan-out) | ### 4.2 路径选型 @@ -267,16 +262,7 @@ flowchart LR `cluster=true` 时按 **AST 解析**（非 substring）依次判断： -```mermaid -flowchart TD - Q[SQL] --> P{单语句 SELECT?} - P -->|否| C[路径 C] - P -->|是| M{单表且无 JOIN/CTE/UNION/子查询?} - M -->|否| C - M -->|是| A{单表 global.* + 可下推 GROUP BY/聚合?} - A -->|是| PA[路径 A] - A -->|否| PB[路径 B] -``` +![联邦查询根据 AST 选择聚合下推、联邦扫描或广播路径](../assets/architecture/probing-federation-path-selection.svg) | 路径 | 进入条件 | §3 典型场景 | |------|----------|-------------| @@ -413,6 +399,64 @@ merge 后再 `GROUP BY` 数据列 + 用户请求的标签列（若有）。 | compute vs comm | C | `probe.comm JOIN probe.torch_trace`；再 `GROUP BY _rank` | | hang / backtrace | A 或 B | 单表 `GROUP BY _rank` 或 filter raw | +### 4.11 分层 fan-out {#hierarchical-fan-out} + +路径 A/B/C 说明 SQL 在每个分片和协调器上如何计算；分层 fan-out 说明这些分片请求如何在 +网络上到达万卡 rank。扁平模式需要 coordinator 直接建立 `O(world_size)` 个连接，分层模式 +把 coordinator 连接数降为 `O(node_count)`。 + +![coordinator 只连接各机代理 rank 的分层查询](../assets/architecture/probing-fanout-internals.svg) + +| 层级 | fan-out 目标 | 8 卡/机、1024 机时的典型规模 | +|------|--------------|-------------------------------| +| coordinator | 每个 `group_rank` 的 `local_rank=0` | 约 1023 个远端 node | +| node/local0 | 同 `group_rank` 的 leaf rank | 每机约 7 个 | +| leaf | 无，只执行本地 SQL | 1 | + +`POST /apis/cluster/query` 的控制字段： + +| 字段 | 默认 | 语义 | +|------|------|------| +| `cluster` | `false` | 是否跨 endpoint | +| `hierarchical` | `true` | 是否使用 node 层 | +| `scope` | `auto` | `auto` / `coordinator` / `node` / `local` | + +`auto` 在 local0 入口选择 coordinator，在 leaf 入口选择 local。`coordinator` 聚合本机和远端 +node；`node` 只聚合当前机器；`local` 禁止继续 fan-out。peer 始终执行本地 `probe.*`，因此 +scope 是防止递归放大的执行契约，不只是性能提示。 + +分层执行依赖 `cluster.nodes` 中完整的 `group_rank`、`local_rank` 和 `addr`。默认分层模式下， +任一活跃成员缺少拓扑字段都会返回 HTTP 503，而不是静默退回可能漏数的扁平查询。只有用户 +明确设置 `hierarchical=false`、CLI `--flat` 或 `PROBING_CLUSTER_FANOUT_HIERARCHICAL=0` +时才使用扁平路径。 + +每一层必须向上合并以下质量信息： + +```json +{ + "hierarchical": true, + "scope": "coordinator", + "nodes_queried": 4, + "nodes_failed": [], + "node_aggregators_queried": 1, + "local_ranks_queried": 1 +} +``` + +远端 local0 返回的 leaf 覆盖率不能被压成“一个成功 endpoint”。非 strict 模式可接收下层 +HTTP 503 中的 partial body，并继续携带 `nodes_failed`；`PROBING_FANOUT_STRICT=1` 时任一 +失败、batch 丢弃或覆盖不完整都使整查失败。 + +| 联邦路径 | 分层执行 | +|----------|----------| +| A 聚合下推 | node 层合并本机可归并 partial，再由 coordinator 全局归并 | +| B 联邦 scan | coordinator 只为各 node aggregator 建远程 lazy partition | +| C broadcast | 每个 node 在本机 rank 上执行同一 SQL并拼接，再向上返回 | + +实现位于 `probing/server/src/server/cluster_fanout.rs`、`cluster_query.rs` 和 +`probing/core/src/core/federation/fanout_scope.rs`。成员层级来源见 +[分布式成员与控制面](distributed.zh.md#cluster-membership)。 + --- ## 5. 回归查询 @@ -437,7 +481,7 @@ merge 后再 `GROUP BY` 数据列 + 用户请求的标签列（若有）。 | 文档 | 内容 | |------|------| -| [分布式架构](distributed.zh.md) | `cluster query` 用法 | +| [分布式成员与控制面](distributed.zh.md) | 成员发现、heartbeat 与拓扑元数据 | | [核心模型](../guide/concepts.zh.md) | Catalog、联邦标签 | | [SQL 表目录](../reference/sql-tables.zh.md) | 表列与 `cluster.nodes` | | [NCCL Profiler](nccl-profiler.zh.md) | §3.1 ⑤、`nccl.proxy_ops` | diff --git a/docs/src/design/hierarchical-fanout.md b/docs/src/design/hierarchical-fanout.md deleted file mode 100644 index 4b4b6483..00000000 --- a/docs/src/design/hierarchical-fanout.md +++ /dev/null @@ -1,196 +0,0 @@ -# Hierarchical cluster query aggregation - -Cross-rank **`cluster query`** and **`global.*`** federation default to **hierarchical fan-out** so the coordinator (usually global rank 0) does not open one HTTP connection per training rank at wan scale. - -Aligns with [Torchrun cluster heartbeat](torchrun-cluster.md) membership tiers. SQL semantics: [Federated query engine](federation.md). - ---- - -## 1. Cost model - -Flat fan-out sends concurrent HTTP from the coordinator to **every** live peer in `cluster.nodes`: - -- Wan scale ≈ **O(world_size)** concurrent connections (e.g. 8192–10240) -- Rank-0 coordinator memory and socket pressure -- One slow rank bounds total latency (≈ slowest peer) - -Hierarchical fan-out splits the query into **coordinator → per-machine local0 → on-node leaf ranks**. Coordinator-side connections ≈ **O(number of nodes)**. - ---- - -## 2. Tiers - -```text -coordinator (global rank 0 / query entry, local_rank=0) - │ - ├─ Local node tier (scope=node) - │ local0 executes SQL locally - │ └─ fan-out → leaf ranks on same group_rank (POST /query, local only) - │ └─ merge rows / aggregate partials → node result - │ - └─ Remote node tier (scope=coordinator → each machine local0) - POST /apis/cluster/query { scope: "node", ... } - each local0 repeats the local node tier, returns to coordinator - coordinator merges node partials + injects federation tags -``` - -| Tier | Who | Fan-out targets | Example (8 GPUs/node, 1024 nodes) | -|------|-----|-----------------|-------------------------------------| -| **Coordinator** | rank0 probe | Each machine `local_rank=0` (one per `group_rank`) | ~1023 remote nodes | -| **Node** | Each machine local0 | Leaf ranks on same machine | ~7 / node | -| **Leaf** | `local_rank>0` | None (local execute only) | — | - ---- - -## 3. Enable and disable - -### Default - -- **`PROBING_CLUSTER_FANOUT_HIERARCHICAL=1`** (on by default) -- `POST /apis/cluster/query` and CLI `probing cluster query` default **`hierarchical: true`** - -### Disable (flat fan-out) - -```bash -export PROBING_CLUSTER_FANOUT_HIERARCHICAL=0 -# Or per request -probing -t rank0:8080 cluster query --flat "SELECT ..." -``` - -```json -POST /apis/cluster/query -{ "expr": "...", "cluster": true, "hierarchical": false } -``` - -### Prerequisites - -Hierarchical mode depends on metadata in `cluster.nodes`: - -| Field | Purpose | -|-------|---------| -| `group_rank` / `NODE_RANK` | Physical node identity | -| `local_rank` | Distinguish local0 (`0`) vs leaf | -| `addr` | HTTP fan-out target | - -Filled automatically by torchrun heartbeat / `PUT /apis/nodes`. If the cluster view **lacks** these fields while hierarchical mode is on (default), **`POST /apis/cluster/query` returns HTTP 503** — probing does **not** silently fall back to flat fan-out. Use `hierarchical=false` (or `--flat`) only when you explicitly accept flat fan-out. - ---- - -## 4. API - -### `POST /apis/cluster/query` - -| Field | Type | Default | Description | -|-------|------|---------|-------------| -| `expr` | string | — | SQL | -| `cluster` | bool | `false` | Cross-node query | -| `hierarchical` | bool | `true` | Use hierarchical fan-out | -| `scope` | string | `auto` | `auto` / `coordinator` / `node` / `local` | - -**`scope` values** - -| Value | Behavior | -|-------|----------| -| `auto` | local0 entry → `coordinator`; leaf → `local` | -| `coordinator` | Local node aggregation + remote node aggregators | -| `node` | This machine only: local0 + leaves (called by coordinator) | -| `local` | Current process only, no fan-out | - -### Response `meta` - -```json -{ - "cluster": true, - "hierarchical": true, - "scope": "coordinator", - "nodes_queried": 4, - "nodes_failed": [], - "peer_batches_dropped": 0, - "partial": false, - "node_aggregators_queried": 1, - "local_ranks_queried": 1 -} -``` - -| Field | Meaning | -|-------|---------| -| `partial` | `true` when any peer failed or merge dropped batches — HTTP **503** with partial `dataframe` (unless `PROBING_FANOUT_STRICT=1`, then the query fails entirely) | -| `peer_batches_dropped` | Partial peer DataFrames dropped during coordinator merge | -| `nodes_queried` | Rank/endpoints attempted across the complete fan-out tree, including the local rank; failed attempts are included | -| `node_aggregators_queried` | Remote **local0** endpoints contacted at coordinator tier | -| `local_ranks_queried` | **Leaf ranks** contacted on the coordinator machine | -| `nodes_failed` | Peers that timed out or returned HTTP errors | - -!!! note "Recursive coverage" - A successful 2-node × 2-GPU hierarchical query reports `nodes_queried=4`. The remote local0 propagates its leaf coverage to the coordinator instead of being counted as one opaque success. - -### CLI - -```bash -# Default: hierarchical -probing -t rank0:8080 cluster query " - SELECT _rank, avg(duration_ms) AS avg_ms - FROM global.python.comm_collective - GROUP BY _rank - ORDER BY avg_ms DESC - LIMIT 10 -" - -# Flat (avoid at wan scale) -probing -t rank0:8080 cluster query --flat "SELECT ..." -``` - -### Web - -`GET /apis/training/step_matrix?cluster=true` uses hierarchical fan-out by default. - ---- - -## 5. Relationship to federation paths - -| Federation path | Hierarchical behavior | -|-----------------|----------------------| -| **A — aggregate pushdown** | Coordinator sends `per_node_sql` to **node aggregators**; local machine also fans out to leaves; see `aggregate_pushdown.rs` | -| **C — broadcast** (JOIN / CTE) | Coordinator runs node aggregation locally; remote nodes recurse via `scope=node` | -| **B — federated scan** | Remote lazy partitions under `FanoutScope::Coordinator` pull **node aggregators** only | - -Complex CTE + window queries should still be split into diagnostic chains (see [Federated query engine §4.7](federation.md#path-c-broadcast)). - ---- - -## 6. Environment variables - -| Variable | Default | Description | -|----------|---------|-------------| -| `PROBING_CLUSTER_FANOUT_HIERARCHICAL` | `1` | `0` = global flat fan-out (legacy O(world_size) path) | -| `PROBING_REMOTE_QUERY_TIMEOUT_SECS` | `30` | Per-peer HTTP timeout (per tier in hierarchical mode); see [Environment variables](../reference/env-vars.md) | -| `PROBING_FANOUT_STRICT` | unset | When `1` or `true`, any peer failure or dropped batch fails the whole query (no partial 503) | - -When hierarchical mode is on (default) but `cluster.nodes` lacks `group_rank` / `local_rank` (heartbeat not converged), **`POST /apis/cluster/query` returns HTTP 503** instead of silently falling back to flat fan-out. Use `hierarchical=false` only when you explicitly accept flat fan-out. - -Cluster heartbeat variables: [Environment variables — cluster](../reference/env-vars.md) and [Torchrun cluster heartbeat](torchrun-cluster.md). - ---- - -## 7. Implementation - -| Module | Path | -|--------|------| -| Fan-out orchestration | `probing/server/src/server/cluster_fanout.rs` | -| HTTP handler | `probing/server/src/server/cluster_query.rs` | -| Peer selection | `probing/core/src/core/cluster.rs` (`node_aggregator_peers`, `local_leaf_peers`) | -| Fan-out scope | `probing/core/src/core/federation/fanout_scope.rs` | -| Remote execution | `probing/core/src/core/federation/cluster_executor.rs` | - -Integration test: `tests/regression/rust/probing/server/hierarchical_fanout_query.rs` (`server_hierarchical_fanout_query`). - ---- - -## 8. Related - -| Document | Content | -|----------|---------| -| [Distributed overview](distributed.md) | `cluster nodes` / `cluster query` | -| [Federated query engine](federation.md) | `global.*`, diagnostic SQL, wan-scale bar | -| [Torchrun cluster heartbeat](torchrun-cluster.md) | Membership tiers | -| [Modularity — cross-rank fan-out](modularity.md) | L3 control-plane ownership | diff --git a/docs/src/design/hierarchical-fanout.zh.md b/docs/src/design/hierarchical-fanout.zh.md deleted file mode 100644 index dac25b17..00000000 --- a/docs/src/design/hierarchical-fanout.zh.md +++ /dev/null @@ -1,192 +0,0 @@ -# 分层集群查询聚合 - -跨 rank 的 `cluster query` / `global.*` 查询默认走 **分层 fan-out**，避免 coordinator（通常是 global rank 0）在万卡规模下直连每一个训练 rank。 - -与 [torchrun 集群心跳](torchrun-cluster.zh.md) 的成员注册层级对齐；联邦 SQL 语义见 [联邦查询引擎](federation.zh.md)。 - ---- - -## 1. 代价模型 - -扁平 fan-out 从 coordinator 向 `cluster.nodes` 中 **每个** 存活 peer 并发 HTTP： - -- 万卡 ≈ **O(world_size)** 并发连接（例如 8192～10240） -- 单点 rank0 内存与 socket 压力大 -- 慢 rank 拖垮整次查询（总延迟 ≈ 最慢 peer） - -分层 fan-out 将查询拆成 **coordinator → 各机 local0 → 同机 leaf rank**，coordinator 侧连接数 ≈ **O(节点数)**。 - ---- - -## 2. 层级 - -```text -coordinator (global rank 0 / 查询入口, local_rank=0) - │ - ├─ 本机 node 层 (scope=node) - │ local0 本地执行 SQL - │ └─ fan-out → 同 group_rank 的 leaf ranks (POST /query, 仅本地) - │ └─ 行合并 / 聚合 partial → 本机结果 - │ - └─ 远程 node 层 (scope=coordinator → 各机 local0) - POST /apis/cluster/query { scope: "node", ... } - 各机 local0 重复「本机 node 层」逻辑后返回 - coordinator 合并各 node partial + 注入联邦标签 -``` - -| 层级 | 谁 | fan-out 目标 | 典型规模 (8 卡/机, 1024 机) | -|------|-----|---------------|------------------------------| -| **Coordinator** | rank0 探针 | 各机 `local_rank=0`（每 `group_rank` 一个） | ~1023 远程 node | -| **Node** | 各机 local0 | 同机 leaf ranks | ~7 / node | -| **Leaf** | `local_rank>0` | 无（仅本地执行） | — | - ---- - -## 3. 启用与关闭 - -### 默认 - -- **`PROBING_CLUSTER_FANOUT_HIERARCHICAL=1`**（默认开启） -- `POST /apis/cluster/query` 与 CLI `probing cluster query` 的 **`hierarchical` 默认为 `true`** - -### 关闭（恢复扁平 fan-out） - -```bash -export PROBING_CLUSTER_FANOUT_HIERARCHICAL=0 -# 或单次请求 -probing -t rank0:8080 cluster query --flat "SELECT ..." -``` - -```json -POST /apis/cluster/query -{ "expr": "...", "cluster": true, "hierarchical": false } -``` - -### 前置条件 - -分层依赖 `cluster.nodes` 中的元数据： - -| 字段 | 用途 | -|------|------| -| `group_rank` / `NODE_RANK` | 区分物理节点 | -| `local_rank` | 识别 local0（`0`）与 leaf | -| `addr` | HTTP fan-out 目标 | - -由 torchrun 心跳 / `PUT /apis/nodes` 自动填充。若集群视图 **缺少** 上述字段且分层模式开启（默认），**`POST /apis/cluster/query` 返回 HTTP 503** — 不会静默降级为扁平 fan-out。仅在明确接受扁平 fan-out 时使用 `hierarchical=false`。 - ---- - -## 4. API - -### `POST /apis/cluster/query` - -| 字段 | 类型 | 默认 | 说明 | -|------|------|------|------| -| `expr` | string | — | SQL | -| `cluster` | bool | `false` | 是否跨节点 | -| `hierarchical` | bool | `true` | 是否分层 fan-out | -| `scope` | string | `auto` | `auto` / `coordinator` / `node` / `local` | - -**`scope` 含义** - -| 值 | 行为 | -|----|------| -| `auto` | local0 入口 → `coordinator`；leaf → `local` | -| `coordinator` | 本机 node 聚合 + 远程 node aggregators | -| `node` | 仅本机 local0 + leaf（供 coordinator 调用） | -| `local` | 仅当前进程，不 fan-out | - -### 响应 `meta` - -```json -{ - "cluster": true, - "hierarchical": true, - "scope": "coordinator", - "nodes_queried": 4, - "nodes_failed": [], - "node_aggregators_queried": 1, - "local_ranks_queried": 1 -} -``` - -| 字段 | 含义 | -|------|------| -| `nodes_queried` | 整棵 fan-out 树实际尝试的 rank/endpoint 数（包含本 rank 和失败尝试） | -| `node_aggregators_queried` | coordinator 层联系的远程 **local0** 数量 | -| `local_ranks_queried` | 本机 node 层联系的 **leaf rank** 数量 | -| `nodes_failed` | 超时或 HTTP 失败的 peer 地址 | - -!!! note "递归覆盖率" - 成功的 2 机 × 2 卡分层查询通常为 `nodes_queried=4`。远程 local0 会把其 leaf 覆盖率递归传回 coordinator，而不是被当成一个不透明的成功 endpoint。 - -### CLI - -```bash -# 默认分层 -probing -t rank0:8080 cluster query " - SELECT _rank, avg(duration_ms) AS avg_ms - FROM global.python.comm_collective - GROUP BY _rank - ORDER BY avg_ms DESC - LIMIT 10 -" - -# 扁平（万卡慎用） -probing -t rank0:8080 cluster query --flat "SELECT ..." -``` - -### Web - -`GET /apis/training/step_matrix?cluster=true` 默认走分层 fan-out。 - ---- - -## 5. 与联邦路径的关系 - -| 联邦路径 | 分层行为 | -|----------|----------| -| **A 聚合下推** | coordinator 向 **node aggregators** 发送 `per_node_sql`；本机额外 fan-out 至 leaf；见 `aggregate_pushdown.rs` | -| **C broadcast**（JOIN / CTE） | coordinator 对本机做 node 聚合，远程 node 通过 `scope=node` 递归 | -| **B 联邦 scan** | 远程 lazy 分区在 `FanoutScope::Coordinator` 下仅拉 node aggregator | - -复杂 CTE + 窗口仍建议拆诊断链（见 [联邦查询引擎 §4.7](federation.zh.md#47-路径-c-broadcast)）。 - ---- - -## 6. 环境变量 - -| 变量 | 默认 | 说明 | -|------|------|------| -| `PROBING_CLUSTER_FANOUT_HIERARCHICAL` | `1` | `0` = 全局扁平 fan-out（legacy O(world_size) 路径） | -| `PROBING_REMOTE_QUERY_TIMEOUT_SECS` | `30` | 单 peer HTTP 超时（分层下为 per-tier）；见 [环境变量](../reference/env-vars.zh.md) | -| `PROBING_FANOUT_STRICT` | unset | `1`/`true` 时任一 peer 失败或 batch 丢弃则整查失败（无 partial 503） | - -分层模式开启（默认）但 `cluster.nodes` 缺少 `group_rank` / `local_rank`（心跳未收敛）时，**`POST /apis/cluster/query` 返回 HTTP 503**，不再静默回退扁平 fan-out。仅在明确接受扁平 fan-out 时使用 `hierarchical=false`。 - -集群心跳相关变量见 [环境变量 — 集群](../reference/env-vars.zh.md#集群) 与 [torchrun 集群心跳](torchrun-cluster.zh.md)。 - ---- - -## 7. 实现位置 - -| 模块 | 路径 | -|------|------| -| Fan-out 编排 | `probing/server/src/server/cluster_fanout.rs` | -| HTTP handler | `probing/server/src/server/cluster_query.rs` | -| Peer 选择 | `probing/core/src/core/cluster.rs`（`node_aggregator_peers`, `local_leaf_peers`） | -| Fan-out scope | `probing/core/src/core/federation/fanout_scope.rs` | -| 远程执行 | `probing/core/src/core/federation/cluster_executor.rs` | - -集成测试：`tests/regression/rust/probing/server/hierarchical_fanout_query.rs`（`server_hierarchical_fanout_query`）。 - ---- - -## 8. 相关文档 - -| 文档 | 内容 | -|------|------| -| [分布式架构](distributed.zh.md) | `cluster nodes` / `cluster query` | -| [联邦查询引擎](federation.zh.md) | `global.*`、诊断 SQL、万卡五连 | -| [torchrun 集群心跳](torchrun-cluster.zh.md) | 成员注册层级 | -| [模块化 — 跨 rank fan-out](modularity.zh.md) | L3 控制面职责 | diff --git a/docs/src/design/index.md b/docs/src/design/index.md index 3e58560f..142bd3fa 100644 --- a/docs/src/design/index.md +++ b/docs/src/design/index.md @@ -7,45 +7,40 @@ Vocabulary: **[Core model](../guide/concepts.md)**. ## Reading order -1. **[Modularity & boundaries](modularity.md)** — four-layer model, crate map, dependency rules (start here) -2. **[Data Layer](data-layer.md)** — MEMT/MEMC, mmap, SQL integration -3. **[Distributed → Overview](distributed.md)** — multi-node mental model, then nested pages below +1. **[Modularity & boundaries](modularity.md)** — ownership and dependency direction. +2. **[Activation & runtime control](activation-injection.md)** → **[Data Layer](data-layer.md)** — + how Probing enters a process and retains evidence. +3. **[Profiling and tracing](profiling.md)** — TorchProbe, spans, training phases, and stacks. +4. **[Distributed membership](distributed.md)** → **[Federation](federation.md)** — membership, + hierarchical fan-out, and cross-rank SQL. +5. **[Distributed Profiler query and visualization](distributed-profiler.md)** — 10K-rank + timeline semantics, hierarchical execution, and cross-rank drill-down. + +“Current” describes implemented behavior. “Draft/target” pages are not complete product contracts. + +## Foundations + +| Document | Status | Description | +|----------|--------|-------------| +| [Modularity & boundaries](modularity.md) | Current | Layers, public contracts, ownership | +| [Activation & runtime control](activation-injection.md) | Current | `.pth`, ptrace trampoline, and service readiness | +| [Data Layer](data-layer.md) | Current | MEMT/MEMC hot/cold storage and SQL integration | +| [Extensibility](extensibility.md) | Current | Table, collector, skill, and service contracts | -## Platform core +## Collectors & profiling -| Document | Description | -|----------|-------------| -| [Modularity & boundaries](modularity.md) | L1–L4 layers, public contracts, ownership | -| [Data Layer](data-layer.md) | Hot/cold columnar store and SQL integration | -| [Extensibility](extensibility.md) | `@table` plugins, skills, NCCL profiler hook-in | -| [CLI command tree](cli.md) | Command grouping, target rules, migration (draft) | +| Document | Status | Description | +|----------|--------|-------------| +| [Profiling and tracing](profiling.md) | Current | TorchProbe, spans/phases, Python/native stacks, and system collection | +| [NCCL Profiler](nccl-profiler.md) | Current | Plugin ABI and wait decomposition | +| [Overhead](overhead.md) | Current | Shadow-step formulas, change invariants, and offline benchmarks | -## Collectors & profiling +## Distributed query & analysis -| Document | Description | -|----------|-------------| -| [Profiling](profiling.md) | Torch hooks, sampling, table write path | -| [Torch Profiler SQL](torch-profiler-sql.md) | On-demand Kineto capture → virtual SQL tables | -| [Overhead](overhead.md) | Formulas, shadow-step measurement, offline benchmarks | -| [NCCL Profiler](nccl-profiler.md) | Plugin ABI, proxy-op wait decomposition | -| [Debugging Engine](debugging.md) | eval / backtrace / REPL implementation | -| [Training Phases](/zh/design/training-phase/) | Phase transitions and span model *(中文)* | -| [Span API](tracing-spans.md) | `span` / `record_span` / backends / performance | - -## Distributed - -| Document | Description | -|----------|-------------| -| [Overview](distributed.md) | Multi-node topology, control plane, federation intro | -| [Torchrun cluster heartbeat](torchrun-cluster.md) | Hierarchical registration, backoff, env presets | -| [Federated query engine](federation.md) | Cross-rank SQL paths A/B/C, tags, regression queries | -| [Hierarchical fan-out](hierarchical-fanout.md) | Coordinator → local0 → leaf query aggregation | -| [Cluster with Pulsing](cluster-pulsing.md) | Optional Pulsing-based membership | - -## Legacy - -| Document | Description | -|----------|-------------| -| [System Architecture (legacy)](architecture.md) | Two-layer overview — superseded by [Modularity](modularity.md); kept for historical diagrams | +| Document | Status | Description | +|----------|--------|-------------| +| [Distributed membership](distributed.md) | Current | Torchrun registration, heartbeat, TTL, and member metadata | +| [Federated query engine](federation.md) | Current | `global.*`, plan selection, hierarchical fan-out, tags, and partial results | +| [Distributed Profiler](distributed-profiler.md) | Target | 10K-rank timelines, hierarchical query, multi-resolution views, and flamegraphs | User-facing workflows: **[User Guide](../guide/index.md)** · Reference: **[SQL Tables](../reference/sql-tables.md)** · **[CLI & Python API](../api-reference.md)** diff --git a/docs/src/design/index.zh.md b/docs/src/design/index.zh.md index d90cfe63..9508bc16 100644 --- a/docs/src/design/index.zh.md +++ b/docs/src/design/index.zh.md @@ -4,47 +4,49 @@ 术语：**[核心模型](../guide/concepts.zh.md)**。 +## 总体架构 + +![Probing 从使用入口到本地存储的整体架构](../assets/architecture/probing-top-down-overview.svg) + +图中的两条方向需要分开理解：训练回调只向本机表追加数据；CLI、Web、Skill 和 MCP 的 +读取请求则通过服务端进入查询引擎。采集器之间不直接调用，跨数据源关系在 SQL 层组合。 + ## 阅读顺序 -1. **[模块化与边界](modularity.zh.md)** — 四层模型、crate 地图、依赖规则（从这里开始） -2. **[数据层](data-layer.zh.md)** — MEMT/MEMC、mmap、SQL 集成 -3. **[分布式 → 概览](distributed.zh.md)** — 多节点心智模型，再读下方子页 +1. **[模块化与边界](modularity.zh.md)** — 先建立模块归属和依赖方向。 +2. **[启用、注入与运行时控制](activation-injection.zh.md)** → **[数据层](data-layer.zh.md)** — + 理解 Probing 如何进入进程，以及数据如何被持续保存。 +3. **[性能分析与 Tracing](profiling.zh.md)** — 理解 TorchProbe、Span、训练阶段和堆栈采集。 +4. **[分布式成员与控制面](distributed.zh.md)** → **[联邦查询](federation.zh.md)** — + 理解成员发现、分层 fan-out 与跨 rank SQL。 +5. **[分布式 Profiler 查询与可视化](distributed-profiler.zh.md)** — 理解万 Rank Timeline + 数据模型、分层执行和跨 Rank 下钻。 -## 平台核心 +表中“当前”表示描述现有实现；“草案/目标设计”表示尚未全部落地，不能当作已发布能力。 -| 文档 | 说明 | -|------|------| -| [模块化与边界](modularity.zh.md) | L1–L4 分层、公开契约、归属边界 | -| [数据层](data-layer.zh.md) | 热/冷列存与 SQL 集成 | -| [扩展机制](extensibility.zh.md) | `@table` 插件、skills、NCCL profiler | -| [CLI 命令树](cli.zh.md) | 命令分组、target 规则、迁移方案（草案） | +## 基础架构 + +| 文档 | 状态 | 说明 | +|------|------|------| +| [模块化与边界](modularity.zh.md) | 当前 | 分层、公开契约、归属边界 | +| [启用、注入与运行时控制](activation-injection.zh.md) | 当前 | `.pth`、ptrace、shellcode 跳板与服务就绪 | +| [数据层](data-layer.zh.md) | 当前 | MEMT/MEMC 热冷列存与 SQL 集成 | +| [扩展机制](extensibility.zh.md) | 当前 | `@table`、Rust collector、Skill 和公开服务契约 | ## 采集与 Profiling -| 文档 | 说明 | -|------|------| -| [性能分析](profiling.zh.md) | Torch hook、采样、落表路径 | -| [Torch Profiler SQL](torch-profiler-sql.zh.md) | 按需 Kineto 采集 → 虚拟 SQL 表 | -| [开销测量](overhead.zh.md) | 公式定义、shadow step、离线基准 | -| [NCCL Profiler](nccl-profiler.zh.md) | 插件 ABI、proxy 等待分解 | -| [调试引擎](debugging.zh.md) | eval / backtrace / REPL 实现 | -| [训练阶段](training-phase.zh.md) | 阶段转换与 span 模型 | -| [Span API](tracing-spans.zh.md) | `span` / `record_span` / backend / 性能与选型 | - -## 分布式 - -| 文档 | 说明 | -|------|------| -| [概览](distributed.zh.md) | 多节点拓扑、控制面、联邦入门 | -| [torchrun 集群心跳](torchrun-cluster.zh.md) | 分层注册、退避、环境变量 | -| [联邦查询引擎](federation.zh.md) | 跨 rank SQL 路径 A/B/C、标签、回归查询 | -| [分层集群查询](hierarchical-fanout.zh.md) | coordinator → local0 → leaf 聚合 | -| [基于 Pulsing 的集群](cluster-pulsing.zh.md) | 可选 Pulsing 成员发现 | - -## 旧版 - -| 文档 | 说明 | -|------|------| -| [系统架构（旧版）](architecture.zh.md) | 两层概览 — 已由 [模块化](modularity.zh.md) 取代 | +| 文档 | 状态 | 说明 | +|------|------|------| +| [性能分析与 Tracing](profiling.zh.md) | 当前 | TorchProbe、Span/Phase、Python/Native 堆栈与系统采集 | +| [NCCL Profiler](nccl-profiler.zh.md) | 当前 | 插件 ABI、事件层次和等待分解 | +| [开销测量](overhead.zh.md) | 当前 | shadow step、统计口径、回归不变量和离线基准 | + +## 分布式查询与分析 + +| 文档 | 状态 | 说明 | +|------|------|------| +| [分布式成员与控制面](distributed.zh.md) | 当前 | torchrun 注册、heartbeat、TTL 与成员元数据 | +| [联邦查询引擎](federation.zh.md) | 当前 | `global.*`、路径选择、分层 fan-out、标签和 partial 语义 | +| [分布式 Profiler 查询与可视化](distributed-profiler.zh.md) | 目标设计 | 万 Rank Timeline、分层查询、多分辨率视图和分布式火焰图 | 用户向工作流：**[用户指南](../guide/index.zh.md)** · 参考：**[SQL 表目录](../reference/sql-tables.zh.md)** · **[CLI 与 Python API](../api-reference.zh.md)** diff --git a/docs/src/design/modularity.md b/docs/src/design/modularity.md index 742dd420..62231d7a 100644 --- a/docs/src/design/modularity.md +++ b/docs/src/design/modularity.md @@ -3,8 +3,8 @@ This document defines **core vs feature modules**, public interfaces, dependency rules, and ownership boundaries. Goal: parallel development without cross-cutting churn. -Read with [Architecture](architecture.md), [Data Layer](data-layer.md), and -[Extensibility](extensibility.md). Shared vocabulary: [Core Concepts](../guide/concepts.md). +Read with [Activation & runtime control](activation-injection.md), [Data Layer](data-layer.md), +and [Extensibility](extensibility.md). Shared vocabulary: [Core Concepts](../guide/concepts.md). --- @@ -13,45 +13,7 @@ Read with [Architecture](architecture.md), [Data Layer](data-layer.md), and Probing is organized in four layers. **Dependencies only flow downward** (higher layers may call lower; never the reverse). -```mermaid -flowchart TB - subgraph L4["L4 — Experience (no engine logic)"] - WEB[web/ WASM UI] - SKILLS[skills/ diagnostic workflows] - PYSDK[python/probing/ SDK & hooks] - end - - subgraph L3["L3 — Control plane"] - CLI[probing-cli] - SERVER[probing-server] - end - - subgraph L2["L2 — Collectors & extensions"] - PYEXT[probing-python] - CC[probing-cc] - GPU[probing-gpu] - NCCL[probing-nccl-profiler cdylib] - end - - subgraph L1["L1 — Platform core"] - CORE[probing-core] - MEM[probing-memtable] - PROTO[probing-proto] - MACROS[probing-macros] - end - - WEB --> PROTO - CLI --> PROTO - SERVER --> CORE - SERVER --> L2 - L2 --> CORE - L2 --> MEM - CORE --> MEM - CORE --> PROTO - SKILLS --> PYSDK - PYSDK --> PYEXT - NCCL --> MEM -``` +![Module dependency direction and collector isolation](../assets/architecture/probing-module-dependencies.svg) | Layer | Role | Changes when… | |-------|------|----------------| @@ -250,21 +212,7 @@ Collectors must not invent alternate peer tags. ## 4. Dependency rules -```text -Allowed: - L4 → L3 (HTTP only) - L3 → L2, L1 - L2 → L1 - L1 internal: core → memtable, proto - -Forbidden (fix if found): - L1 → L2/L3/L4 - L2 → L2 (collector cross-deps) - L2 → L3 (extensions must not import server) - L2 → probing-cli — collectors must not depend on CLI (wheel `cli_main` is wired in root `src/lib.rs`; see §4.1) - skills → Rust internals - web → probing-core / pyo3 -``` +![Dependencies flow down through published contracts and collectors remain isolated](../assets/architecture/probing-module-dependencies.svg) ### Dependency matrix (target state) @@ -283,11 +231,7 @@ Forbidden (fix if found): Maturin builds **one native artifact** (`probing._core` cdylib from root `Cargo.toml`). The `probing` console script is **not** a separate Rust binary on PyPI: -```text -pip install probing - → probing._core.so (core + server + python ext + cli linked in) - → probing.cli.__main__ → _core.cli_main() → probing_cli::cli_main() -``` +![The wheel composes the CLI entry at the root without a runtime reverse dependency](../assets/architecture/probing-wheel-composition.svg) This is an **accepted compile-time coupling** for the wheel workflow (`pyproject.toml` `[tool.maturin]` + `[project.scripts]`). It is **not** the Python collector calling the CLI @@ -308,22 +252,7 @@ commands, rather than letting collectors or the cdylib spread imports across the --- -```mermaid -sequenceDiagram - participant C as Collector L2 - participant M as memtable L1 - participant E as Engine L1 - participant S as Server L3 - participant X as CLI/Web/Skill L4 - - C->>M: push_row / mmap file - Note over M: MEMT ring (+ optional MEMC cold) - X->>S: POST /query - S->>E: async_query(SQL) - E->>M: scan via TableProvider - E-->>S: DataFrame - S-->>X: JSON / render -``` +![Writes and reads meet at the table contract](../assets/architecture/probing-data-query-contract.svg) **Implications:** @@ -399,38 +328,23 @@ Track and fix incrementally: | Composition sprawl | All wiring in `server/engine.rs` | Optional: manifest TOML listing enabled extensions | | Skills triple loader | ~~Rust + Python + Web compile-time embed~~ | **Done** — `probing-skills` is loader/interpret/runner SSOT; Python keeps discovery entry-points + PyO3 serialize bridge; Web deserializes API into shared types | | kmsg collector | Registered (Linux/kmsg feature gate) | Done | -| Architecture doc | 2-layer diagram | Superseded by this doc + [Data Layer](data-layer.md) | - -### Cluster membership: Torchrun heartbeat vs Pulsing -Two **complementary** paths populate `cluster.nodes` and power `global.*` federation. -They do not replace each other. +### Cluster membership and external runtime tables -| Path | Layer | When | Mechanism | -|------|-------|------|-----------| -| **Torchrun cluster heartbeat** | L3 `probing-server` | Default for `torchrun` / elastic jobs (`WORLD_SIZE > 1`, `PROBING=1/2`) | Hierarchical HTTP PUT + TCPStore side channel (`probing/torchrun//…`). Does **not** touch torch rendezvous keys. See [Torchrun cluster heartbeat](torchrun-cluster.md). | -| **Pulsing integration** | L4 passive + external runtime | Another process already runs [Pulsing](cluster-pulsing.md) and writes `pulsing.*` memtables | Probing discovers `pulsing.*` mmap tables; no probing-owned heartbeat thread. Optional bootstrap via Pulsing APIs. | +The current source for `cluster.nodes` membership is the **Torchrun cluster heartbeat**. L3 +`probing-server` uses hierarchical HTTP PUT plus TCPStore side-channel keys without modifying +torch rendezvous keys. The Rust ctor starts `maybe_start_torchrun_cluster()` by default; see +[Distributed membership and control plane](distributed.md#cluster-membership). -**Default for torchrun users:** heartbeat auto-starts from the Rust ctor (`maybe_start_torchrun_cluster()`). -**Pulsing:** use when the job already centers on Pulsing for membership/failure detection, or you need Pulsing actors alongside probing tables. +`pulsing.*` is an externally produced mmap schema discovered through the same table contract as +other vendor data. Probing does not currently merge Pulsing gossip members into `cluster.nodes` +or bootstrap a Pulsing ActorSystem. --- ## 9. Adding a new feature (decision tree) -```text -Need new raw signals? - └─ Yes → L2 collector - ├─ System/host/GPU/NCCL → Rust extension crate - └─ Training semantics → Python @table + hook in python/probing/ - └─ No - Need new analysis workflow? - └─ Yes → L4 skill (steps.yaml) referencing existing tables - Need new UI? - └─ Yes → L4 web page calling existing HTTP/SQL - Need new transport/command? - └─ Yes → L3 CLI + server endpoint (proto DTO first) -``` +![Place new capabilities according to the facts and state they introduce](../assets/architecture/probing-feature-placement.svg) **Anti-patterns:** @@ -445,12 +359,12 @@ Need new raw signals? | Doc | Scope | |-----|-------| -| [Architecture](architecture.md) | Historical overview (being aligned with this doc) | +| [Activation & runtime control](activation-injection.md) | Process entry and service readiness | | [Data Layer](data-layer.md) | MEMT/MEMC internals | | [Extensibility](extensibility.md) | Public extension paths (table + skill) | | [Distributed](distributed.md) | Federation & cluster | -| [Torchrun cluster heartbeat](torchrun-cluster.md) | Hierarchical torchrun membership | -| [Cluster with Pulsing](cluster-pulsing.md) | Optional Pulsing-based membership | +| [Distributed membership](distributed.md) | Hierarchical torchrun membership and health | +| [Distributed Profiler](distributed-profiler.md) | Target 10K-rank timeline query and visualization | | [NCCL Profiler](nccl-profiler.md) | NCCL plugin boundary | | [web/DESIGN.md](https://github.com/DeepLink-org/probing/blob/main/web/DESIGN.md) | UI module layout | | [AGENTS.md](https://github.com/DeepLink-org/probing/blob/main/AGENTS.md) | Agent skill usage | diff --git a/docs/src/design/modularity.zh.md b/docs/src/design/modularity.zh.md index b6cc2ed6..6f94a82c 100644 --- a/docs/src/design/modularity.zh.md +++ b/docs/src/design/modularity.zh.md @@ -2,7 +2,8 @@ 本文定义 **核心模块 vs 功能模块**、公共接口、依赖规则与建议 ownership，目标是让并行开发**互不打扰**。 -配合阅读：[系统架构](architecture.zh.md)、[数据层](data-layer.zh.md)、[扩展机制](extensibility.zh.md)；术语见 [核心概念](../guide/concepts.zh.md)。 +配合阅读：[启用、注入与运行时控制](activation-injection.zh.md)、[数据层](data-layer.zh.md)、 +[扩展机制](extensibility.zh.md)；术语见[核心概念](../guide/concepts.zh.md)。 --- @@ -10,45 +11,7 @@ Probing 分四层。**依赖只能向下**（上层可调用下层，反之禁止）。 -```mermaid -flowchart TB - subgraph L4["L4 — 体验层（不含引擎逻辑）"] - WEB[web/ WASM UI] - SKILLS[skills/ 诊断工作流] - PYSDK[python/probing/ SDK 与 hooks] - end - - subgraph L3["L3 — 控制面"] - CLI[probing-cli] - SERVER[probing-server] - end - - subgraph L2["L2 — 采集器 / 扩展"] - PYEXT[probing-python] - CC[probing-cc] - GPU[probing-gpu] - NCCL[probing-nccl-profiler cdylib] - end - - subgraph L1["L1 — 平台核心"] - CORE[probing-core] - MEM[probing-memtable] - PROTO[probing-proto] - MACROS[probing-macros] - end - - WEB --> PROTO - CLI --> PROTO - SERVER --> CORE - SERVER --> L2 - L2 --> CORE - L2 --> MEM - CORE --> MEM - CORE --> PROTO - SKILLS --> PYSDK - PYSDK --> PYEXT - NCCL --> MEM -``` +![模块依赖方向与采集器隔离边界](../assets/architecture/probing-module-dependencies.svg) | 层 | 职责 | 典型变更 | |----|------|----------| @@ -209,17 +172,7 @@ HTTP 契约：`probing/server/API.md` + `tests/regression/spec/api_spec.json`。 ## 4. 依赖规则 -```text -允许：L4→L3(HTTP) → L3→L2/L1 → L2→L1 - -禁止： - L1 → 上层 - L2 ↔ L2（采集器互调） - L2 → server - L2 → probing-cli — 采集器不得依赖 CLI（wheel 的 `cli_main` 仅在根 `src/lib.rs` 组装；见 §4.1） - skills → Rust 内部 - web → probing-core -``` +![依赖只沿公开契约向下，采集器之间彼此隔离](../assets/architecture/probing-module-dependencies.svg) ### 依赖矩阵（目标态） @@ -238,11 +191,7 @@ HTTP 契约：`probing/server/API.md` + `tests/regression/spec/api_spec.json`。 Maturin 只构建 **一个 native 产物**（根 `Cargo.toml` 的 `probing._core` cdylib）。PyPI 上的 `probing` 命令 **不是** 独立 Rust 二进制： -```text -pip install probing - → probing._core.so （core + server + python ext + cli 链进同一 .so） - → probing.cli.__main__ → _core.cli_main() → probing_cli::cli_main() -``` +![Wheel 在根动态库中组装 CLI 入口，但不形成运行时反向依赖](../assets/architecture/probing-wheel-composition.svg) 这是为适配 maturin wheel 工作流（`pyproject.toml` 的 `[tool.maturin]` + `[project.scripts]`）的 **可接受编译期耦合**，不是采集器在运行时反向调用控制面。 @@ -259,21 +208,7 @@ pip install probing --- -```mermaid -sequenceDiagram - participant C as 采集 L2 - participant M as memtable L1 - participant E as Engine L1 - participant S as Server L3 - participant X as CLI/Web/Skill L4 - - C->>M: push_row / mmap - X->>S: POST /query - S->>E: async_query - E->>M: TableProvider scan - E-->>S: DataFrame - S-->>X: JSON -``` +![训练写入与查询读取通过表契约汇合](../assets/architecture/probing-data-query-contract.svg) **推论：** 新指标 → 新表；跨信号分析 → SQL 或 skill； retention 只动 memtable 配置。 @@ -340,34 +275,22 @@ sequenceDiagram | 组装集中 | 全在 server/engine.rs | 可选 extension manifest | | Skill 三份 loader | ~~Rust/Python/Web 编译期 embed~~ | **已完成** — `probing-skills` 为 loader/interpret/runner SSOT；Python 保留发现 entry point + PyO3 序列化桥；Web 从 API 反序列化为共享类型 | | kmsg 采集器 | 已注册（Linux/kmsg feature gate） | Done | -| Architecture 文档 | 二层旧图 | 以本文 + [数据层](data-layer.zh.md) 为准 | - -### 集群成员：Torchrun 心跳 vs Pulsing -两条**互补**路径填充 `cluster.nodes` 并支撑 `global.*` 联邦，互不取代。 +### 集群成员与外部运行时表 -| 路径 | 层 | 适用场景 | 机制 | -|------|-----|----------|------| -| **Torchrun 集群心跳** | L3 `probing-server` | 默认：`torchrun`/elastic（`WORLD_SIZE > 1`，`PROBING=1/2`） | 分层 HTTP PUT + TCPStore 旁路键（`probing/torchrun//…`），**不**写 rendezvous 键。见 [Torchrun 集群心跳](torchrun-cluster.zh.md)。 | -| **Pulsing 集成** | L4 被动 + 外部运行时 | 作业已跑 [Pulsing](cluster-pulsing.zh.md) 并写 `pulsing.*` memtable | Probing 发现 mmap 表；无 probing 自有心跳线程。 | +`cluster.nodes` 的现行成员来源是 **Torchrun 集群心跳**：L3 `probing-server` 通过分层 HTTP PUT +和 TCPStore 旁路键注册 rank，不写 torch rendezvous 键。Rust ctor 默认调用 +`maybe_start_torchrun_cluster()`；细节见 [分布式成员与控制面](distributed.zh.md#cluster-membership)。 -**torchrun 用户默认**：Rust ctor 自动 `maybe_start_torchrun_cluster()`。 -**Pulsing**：作业以 Pulsing 为中心做成员/故障检测，或需要 Pulsing actor 与 probing 表并存时使用。 +`pulsing.*` 只是被 mmap catalog 发现的外部表，与其他厂商表遵守相同的数据源契约。Probing +目前不会把 Pulsing gossip 成员自动合并进 `cluster.nodes`，也没有由 Probing 启动 Pulsing +ActorSystem 的现行实现。 --- ## 9. 新功能决策树 -```text -要新原始信号？ - └─ 是 → L2 采集 - ├─ 系统/GPU/NCCL → Rust extension - └─ 训练语义 → Python @table + hook - └─ 否 - 要新分析流程？ → L4 skill（仅 SQL） - 要新 UI？ → L4 web（仅 HTTP/SQL） - 要新命令？ → L3 CLI + proto DTO -``` +![新能力依据新增的事实和状态选择所属模块](../assets/architecture/probing-feature-placement.svg) **反模式：** 在 engine.rs 写业务；Web 查不存在的表；采集器在写路径里 query；Skill 内嵌 Rust 分支；在 `probing-skills` 之外执行 skill（Python/Web 重复 runner）。 @@ -377,12 +300,12 @@ sequenceDiagram | 文档 | 范围 | |------|------| -| [系统架构](architecture.zh.md) | 历史概览（逐步与本页对齐） | +| [启用、注入与运行时控制](activation-injection.zh.md) | 运行时进入目标进程和服务就绪 | | [数据层](data-layer.zh.md) | MEMT/MEMC 内部实现 | | [扩展机制](extensibility.zh.md) | 对外扩展路径（表 + skill + NCCL） | | [分布式](distributed.zh.md) | 联邦与集群 | -| [Torchrun 集群心跳](torchrun-cluster.zh.md) | 分层 torchrun 成员注册 | -| [基于 Pulsing 的集群](cluster-pulsing.zh.md) | 可选 Pulsing 成员发现 | +| [分布式成员与控制面](distributed.zh.md) | 分层 torchrun 成员注册与健康状态 | +| [分布式 Profiler 查询与可视化](distributed-profiler.zh.md) | 万 Rank Timeline 查询和可视化目标设计 | | [NCCL Profiler](nccl-profiler.zh.md) | NCCL 插件边界 | | [web/DESIGN.md](https://github.com/DeepLink-org/probing/blob/main/web/DESIGN.md) | 前端模块布局 | | [AGENTS.md](https://github.com/DeepLink-org/probing/blob/main/AGENTS.md) | Agent 使用 skill | diff --git a/docs/src/design/nccl-profiler.md b/docs/src/design/nccl-profiler.md index 0e1fafc3..f37285aa 100644 --- a/docs/src/design/nccl-profiler.md +++ b/docs/src/design/nccl-profiler.md @@ -1,245 +1,99 @@ -# NCCL profiler plugin - -Fine-grained **NCCL wait decomposition** for distributed training: distinguish a **culprit** rank (local GPU slow to produce data) from a **victim** rank (waiting on peers or the network). - -This is **Path 3** in [Extensibility](extensibility.md)—a Rust `cdylib` loaded by NCCL, not a Python table plugin. - -## When to use - -| Signal | Tool | -|--------|------| -| Step time high, unsure if comm or compute | `python.comm_collective` + skill `comm_bottleneck` | -| Which rank is the straggler? | skill `slow_rank` | -| Straggler identified — **why** (GPU vs network wait)? | `nccl.proxy_ops` + skill `nccl_culprit_victim` | -| Suspect RoCE / IB congestion | `nccl.net_qp` + `rdma.mlx_hca` | - -Coarse collective tracing (`python.comm_collective`) is **off by default**; enable with -`PROBING_TORCH_COLLECTIVE_ENABLE=1` or `SET probing.torch.collective.enable=1` -(in addition to `PROBING=1`). The NCCL profiler plugin requires **NCCL ≥ 2.26** (PyTorch **2.8+** recommended); it exports both **`ncclProfiler_v4`** (NCCL ≥ 2.27, preferred) and **`ncclProfiler_v3`** (NCCL 2.26) — NCCL negotiates the highest version automatically. - -## Three collective data sources — keep them apart - -probing has three independent collective-communication collectors. They have -**different timing semantics** and must not be conflated: - -| Source | Tables | What it measures | Role | -|--------|--------|------------------|------| -| **NCCL profiler plugin** (this doc) | `nccl.coll_perf`, `nccl.proxy_ops`, `nccl.inflight_ops`, `nccl.net_qp` | NCCL-native events: reconstructed execution time, wait decomposition, bandwidth | **Precise source of truth** | -| Torch-API tracer (legacy, `probing/profiling/collective/`) | `python.comm_collective` | Python wall-clock around the `torch.distributed` API call (launch layer) | Coarse fallback; carries `global_step` context | -| PyTorch Flight Recorder bridge | `python.torch_nccl_flight_record`, `python.torch_nccl_pg_status` | torch's internal watchdog ring buffer | Watchdog-timeout / desync forensics | - -Rules of engagement: - -- The Torch-API tracer is **disabled by default** (including multi-rank jobs) to - avoid silent overhead. Enable with `PROBING_TORCH_COLLECTIVE_ENABLE=1` or - `SET probing.torch.collective.enable=1` (e.g. when you need per-step - `global_step` alignment alongside precise NCCL timing). -- For execution time, bandwidth, and wait attribution, always query `nccl.*`. - `python.comm_collective.duration_ms` is **not** NCCL execution time — for - `async_op` calls it closes at `work.wait()`, otherwise at API return. -- Joining the layers: `nccl.*` rows carry no training step; correlate by - epoch-ns time window against `python.comm_collective.global_step` if needed. - -## Quick start (Linux training) - -```bash -pip install probing # wheel bundles libprobing_nccl_profiler.so on Linux - -export NCCL_PROFILER_PLUGIN=$(python -m probing.nccl --plugin-path) -export NCCL_PROFILE_EVENT_MASK=$(python -m probing.nccl --event-mask) # default 94 -export PROBING=2 - -torchrun --nproc_per_node=8 train.py - -# Same process or after inject: -probing -t skill run nccl_culprit_victim -probing -t query " - SELECT rank, sum(send_gpu_wait_ns) AS gpu_wait, sum(recv_wait_ns) AS recv_wait - FROM nccl.proxy_ops - GROUP BY rank - ORDER BY recv_wait DESC" -``` - -### Optional: NetPlugin (IB QP timing) - -```bash -export NCCL_PROFILE_EVENT_MASK=222 # 94 + NetPlugin bit 128 -probing -t query "SELECT * FROM nccl.net_qp LIMIT 20" -``` - -## macOS / dev without NCCL - -```bash -PROBING=1 PROBING_NCCL_MOCK=1 python -m probing.nccl --seed-mock -probing -t skill run nccl_culprit_victim -``` - -On macOS, `PROBING_NCCL_MOCK=auto` (default) seeds mock tables when `PROBING=1` and no plugin `.so` is present. - -Mock scenario: - -- **rank 2** — culprit (`send_gpu_wait_ns` high) -- **rank 5** — victim (`recv_wait_ns` high) - -## Tables - -### `nccl.proxy_ops` - -Per NCCL proxy operation, with ProxyStep waits aggregated at op stop. - -| Column | Meaning | -|--------|---------| -| `ts` | Event timestamp (ns) | -| `rank` | `torch.distributed` rank | -| `tp_rank`, `pp_rank`, `dp_rank` | Parallel roles from env (`TP_RANK`, `PP_RANK`, `DP_RANK`, Megatron names); `-1` if unset | -| `comm_hash` | NCCL communicator hash | -| `coll_func` | Collective name (`AllReduce`, …) | -| `seq` | Collective sequence number | -| `channel_id` | NCCL channel | -| `peer` | Peer rank for this proxy op | -| `is_send` | `1` = send proxy, `0` = recv | -| `n_steps` | ProxyStep count aggregated | -| `trans_bytes` | Bytes transferred (v4: summed from per-step `transSize` updates) | -| `send_gpu_wait_ns` | **Culprit signal** — local GPU not ready to send | -| `send_peer_wait_ns` | Waiting for receiver clear-to-send credits (**v4 ABI only**, 0 on v3) — receiver-congestion signal | -| `send_wait_ns` | Send-side network wait | -| `recv_wait_ns` | **Victim signal** — waiting on peer data | -| `recv_flush_wait_ns` | Recv flush wait | - -Multi-node: `global.nccl.proxy_ops` with `_host`, `_addr`, `_rank` federation columns. - -> `ts` columns in all `nccl.*` tables are **UNIX-epoch nanoseconds**, so -> timestamps are comparable across ranks/hosts in `global.nccl.*` queries. - -### `nccl.coll_perf` - -Per collective / P2P operation. - -**Timing model.** NCCL's own docs state that a collective's `stopEvent` only -marks the end of the **host-side enqueue** — the kernel and proxy threads keep -working after it. Following the official ext-profiler recommendation, the -plugin reference-counts child events (`ProxyOp`, `KernelCh`) and reconstructs -the real execution window from them. The `timing_source` column records which -signal was available: - -| `timing_source` | Window | Quality | -|-----------------|--------|---------| -| `kernel_gpu` | GPU **globaltimer** window: `kernelCh.pTimer` (start) + `KernelChStop` state (stop) | Best — device clock, **v4 ABI only** | -| `kernel_ch` | Kernel-channel activity observed by the proxy thread (`ncclProfileKernelCh`) | NCCL's own kernel-activity signal, host clock | -| `proxy` | Proxy-op start→stop envelope | Good for inter-node ops | -| `enqueue` | Coll start→stop (launch only) | Fallback — intra-node ops without proxy/kernel events | - -| Column | Meaning | -|--------|---------| -| `ts` | Op completion timestamp (epoch ns) | -| `rank`, `tp_rank`, `pp_rank`, `dp_rank` | Same as `nccl.proxy_ops` | -| `comm_hash`, `coll_func`, `seq` | Collective identity (`seq` = 0 for P2P) | -| `n_ranks` | Communicator size (v4 per-comm `init` metadata; `-1` on v3) | -| `is_p2p` | `1` = Send/Recv, `0` = collective | -| `peer` | P2P peer rank (`-1` for collectives) | -| `count`, `msg_size_bytes`, `dtype` | Payload: element count × dtype size | -| `algo`, `proto`, `n_channels` | NCCL algorithm (Ring/Tree…), protocol (LL/LL128/Simple), channels (v4: P2P too) | -| `exec_time_ns` | Reconstructed execution duration (see `timing_source`) | -| `enqueue_time_ns` | Host-side enqueue duration (NCCL coll start→stop) | -| `timing_source` | `kernel_gpu` / `kernel_ch` / `proxy` / `enqueue` | -| `algobw_gbps` | Algorithm bandwidth `msg_size / exec_time` (GB/s). **Bus bandwidth**: multiply by the collective factor using `n_ranks`, e.g. AllReduce `2(n_ranks-1)/n_ranks`, in SQL | - -```sql --- Slowest AllReduce buckets by bandwidth -SELECT coll_func, msg_size_bytes, AVG(algobw_gbps) AS gbps, COUNT(*) AS n -FROM nccl.coll_perf -GROUP BY coll_func, msg_size_bytes -ORDER BY gbps ASC LIMIT 10 -``` - -### `nccl.inflight_ops` - -Periodic watchdog snapshot of operations that **started but never stopped** — -the hang signal that `nccl.proxy_ops` cannot capture (a hung op never reaches -`stop_event`). Columns: `ts`, `rank`, `comm_hash`, `coll_func`, `seq`, `kind` -(`coll`/`p2p`/`proxy_op`), `channel_id`, `peer`, `is_send`, `start_ns`, `age_ns`. - -```sql --- Which rank is stuck, and in what? -SELECT rank, coll_func, seq, kind, MAX(age_ns)/1e9 AS stuck_secs -FROM nccl.inflight_ops -GROUP BY rank, coll_func, seq, kind -ORDER BY stuck_secs DESC -``` - -### `nccl.net_qp` - -IB queue-pair completion timing (NetPlugin mask). Columns: `ts`, `rank`, `device`, `qp_num`, `wr_id`, `opcode`, `length`, `duration_ns`. - -## Culprit vs victim - -From NCCL ProxyStep state transitions (paper mapping): - -- **Culprit** — dominant `send_gpu_wait_ns` on a rank: that GPU is slow to produce tensors for the collective. -- **Victim** — dominant `recv_wait_ns`: the rank spends time waiting for peers or the network. - -A single rank can appear as culprit for one collective and victim for another. Compare both columns per rank; use `tp_rank`/`pp_rank`/`dp_rank` to align with Megatron-style topology. - -## Diagnostic skill: `nccl_culprit_victim` - -Bundled under `skills/nccl_culprit_victim/` (wheel: `python/probing/_skills/`). - -```bash -probing skill list -probing -t skill run nccl_culprit_victim -probing -t skill run nccl_culprit_victim --set seq_window=50 --global -``` - -Steps include: - -1. Per-rank wait summary (`send_gpu_wait_ns` / `recv_wait_ns`) -2. Culprit ranking (by `send_gpu_wait_ns`) -3. Victim ranking (by `recv_wait_ns`) -4. Role-aligned view (`tp` / `pp` / `dp`) -5. Optional `global.nccl.proxy_ops` fan-out -6. Optional `nccl.net_qp` hint - -Related skills: `slow_rank`, `comm_bottleneck` (coarse layer; optionally join `nccl.proxy_ops` when present). - -## Environment variables - -| Variable | Purpose | -|----------|---------| -| `NCCL_PROFILER_PLUGIN` | Path to `libprobing_nccl_profiler.so` | -| `NCCL_PROFILE_EVENT_MASK` | Event mask; default `94` = Coll \| P2P \| ProxyOp \| ProxyStep \| KernelCh | -| `PROBING_DATA_DIR` | Memtable directory (default `/dev/shm/probing`) | -| `PROBING_NCCL_MIN_MSG_BYTES` | Skip ops smaller than this (bytes); default `0` = record all. Same idea as NCCL Inspector's `DUMP_MIN_SIZE_BYTES` | -| `PROBING_NCCL_INFLIGHT_THRESHOLD_SECS` | Watchdog: snapshot in-flight ops older than this into `nccl.inflight_ops`; default `10`, `0` disables | -| `PROBING_NCCL_POOL_SHARDS` | Shard slot pools by comm hash (default `8`, range 1–64); reduces callback lock contention on multi-comm jobs | -| `PROBING_NCCL_MOCK` | `auto` / `1` / `0` — mock tables for dev | -| `TP_RANK`, `PP_RANK`, `DP_RANK` | Written into `nccl.proxy_ops` role columns | - -CLI helpers: - -```bash -python -m probing.nccl --plugin-path -python -m probing.nccl --event-mask -python -m probing.nccl --seed-mock --ranks 8 --ops 5 -``` - -## Build from source - -```bash -make nccl-profiler-lib # Linux .so → python/probing/libs/ -cargo test -p probing-nccl-profiler -``` - -Crate: `probing/extensions/nccl-profiler/`. See crate [README](https://github.com/DeepLink-org/probing/blob/main/probing/extensions/nccl-profiler/README.md) for architecture (slot pools, Coll→ProxyOp→ProxyStep hierarchy, batch flush). - -## Smoke test checklist (P0) - -1. `python -c "import torch; print(torch.__version__, torch.cuda.nccl.version())"` — NCCL ≥ 2.26 -2. `NCCL_PROFILER_PLUGIN` set before `torchrun` -3. After a few collectives: `SELECT count(*) FROM nccl.proxy_ops` > 0 -4. `probing skill run nccl_culprit_victim` returns rank breakdown - -## See also - -- [Distributed training](distributed.md) — cluster fan-out, `global.*` -- [Extensibility](extensibility.md) — Path 1 (table plugin), Path 2 (skills), Path 3 (this plugin) -- [AGENTS.md](https://github.com/DeepLink-org/probing/blob/main/AGENTS.md) — agent skill install and routing +# NCCL Profiler Architecture + +The NCCL Profiler is not designed merely to persist callbacks. It must answer a distributed causal +question: when a collective slows down, is time spent waiting for this rank to produce data, for a +peer, for the network, or for device execution? A single timestamp cannot answer that question. +The architecture therefore preserves event lifetimes and lets the query layer align evidence +across ranks. + +## Observation boundary: why the collector lives inside NCCL + +The PyTorch API layer knows `global_step` and parallel roles, but the return of a synchronous call +or an asynchronous `work.wait()` is a host boundary, not proof that device and proxy work completed. +PyTorch Flight Recorder preserves watchdog ring records and is useful for post-timeout collective +alignment, but it does not continuously decompose waits in normal communication. Only the NCCL +profiler plugin observes Collective, KernelCh, ProxyOp, ProxyStep, and NetPlugin callbacks directly. + +The evidence planes remain separate. Python supplies training coordinates, NCCL reconstructs +runtime execution and waits, and Flight Recorder preserves timeout state. They do not call one +another on callback paths; queries correlate them by communicator, sequence, rank, and epoch-ns +windows. This preserves each layer's time semantics and avoids pulling Python state into NCCL +communication threads merely to simplify a later join. + +The plugin exports both `ncclProfiler_v4` and `ncclProfiler_v3`, allowing NCCL to negotiate the ABI. +V4 provides GPU globaltimer, per-communicator metadata, and fuller peer-wait evidence. Missing v3 +signals degrade explicitly through `timing_source` and sentinel values rather than pretending to +have equal precision. + +## Event lifetime: completion comes from child events + +![NCCL child events reconstruct the execution window and decompose waits](../assets/architecture/probing-nccl-event-model.svg) + +A collective `stopEvent` closes host enqueue while kernels and proxy work may still be running. +Publishing at that point would label launch time as execution time. The plugin therefore keeps the +Collective as a parent of active KernelCh and ProxyOp events; each ProxyOp, in turn, owns ProxyStep +progress. Only the final child close gives the parent a complete window and makes it publishable. + +Timing degrades through an evidence hierarchy: GPU globaltimer first, then the KernelCh activity +window, then the ProxyOp envelope, and finally host enqueue. The selected source is stored in +`timing_source`. This is not presentation metadata; it is part of query semantics. Two +`exec_time_ns` values should be compared directly only when their evidence quality is comparable. + +ProxyStep is not published as an unbounded detail table. Its transitions accumulate within a +ProxyOp into send-side GPU wait, peer-credit wait, network send, receive, and flush wait. This +trades bounded state for the decomposition needed by diagnosis and prevents message fragmentation +from multiplying storage volume. The waits remain evidence rather than conclusions: high +`send_gpu_wait_ns` implicates local production, while high `recv_wait_ns` implicates a peer or the +network. Culprit/victim attribution still requires the same sequence on other ranks, parallel +topology, and system state. + +## Callback concurrency: communication threads never yield to diagnostics + +![NCCL callbacks update sharded fixed pools and write completed rows outside the lock](../assets/architecture/probing-nccl-write-path.svg) + +Callbacks arrive from host, proxy, NetPlugin, and watchdog threads. A global lock or dynamic growth +inside those callbacks could make the profiler alter communication timing. The plugin uses +fixed-capacity slot pools sharded by communicator hash. A callback normally touches one shard, and +capacity plus worst-case allocation cost are fixed at startup. + +A handle contains shard, slot, and generation. Reuse changes the generation, so a late stop cannot +close a newer event that occupies the same slot. Under the shard lock the callback updates parent/ +child state and counters and materializes a completed row. MEMT append happens only after releasing +the lock, preventing storage jitter from widening the NCCL critical section. + +The watchdog uses `try_lock`. If a shard is busy, it skips and counts that snapshot instead of +waiting for the communication thread. This is an explicit priority decision: an observable data +gap is acceptable; creating a new hang while trying to diagnose one is not. + +## Publication model: tables are projections of lifecycle state + +Completed communication is published to `nccl.coll_perf` with its reconstructed window, +algorithm, protocol, message size, and `timing_source`. Proxy wait decomposition for the same work +goes to `nccl.proxy_ops`. An operation that never completes cannot produce either completed row, so +the watchdog writes read-only snapshots to `nccl.inflight_ops`. With NetPlugin enabled, QP +completion latency enters `nccl.net_qp` independently rather than being attached to a collective +whose relationship has not been proven. + +These are not four competing answers; they are four projections of the event lifecycle. A query +starts with `coll_perf` to locate an anomalous window and uses `proxy_ops` to explain its waits. If +no completion exists it turns to `inflight_ops`; only network-wait evidence justifies joining +`net_qp` and RDMA metrics. Cross-rank queries use `global.nccl.*` to filter locally before merging, +while epoch-ns windows connect NCCL evidence to Python training-step coordinates. Aggregation and +causal inference belong to the query layer and never feed back into the collector. + +`nccl.profiler_counters` defines the integrity boundary for all four projections. Pool exhaustion, +stale handles, write failures, and watchdog skips are counted. A diagnosis must inspect these +signals before interpreting an absence of events as an absence of anomalies. + +## Failure boundary and implementation constraints + +The callback path never waits for a remote node, calls another collector, or changes NCCL control +flow because diagnostics failed. A full pool drops and counts an event, MEMT failures are recorded +outside the shard lock, and watchdog contention skips a snapshot. The evidence can therefore have +an explicit gap while training communication retains its original control flow. + +Exact schemas are in the [SQL table reference](../reference/sql-tables.md), capacity and runtime +controls in [Environment variables](../reference/env-vars.md), deployment and query examples in +[Performance analysis](../examples/performance-analysis.md), and cross-rank diagnostic orchestration +in [Diagnostic skills](../guide/skills.md). The implementation lives under +`probing/extensions/nccl-profiler/`. diff --git a/docs/src/design/nccl-profiler.zh.md b/docs/src/design/nccl-profiler.zh.md index e3b8fecf..eb642a85 100644 --- a/docs/src/design/nccl-profiler.zh.md +++ b/docs/src/design/nccl-profiler.zh.md @@ -1,225 +1,81 @@ -# NCCL profiler 插件 +# NCCL Profiler 架构 -面向分布式训练的 **NCCL 等待分解**：区分 **culprit**（本 rank GPU 产出慢）与 **victim**（等待 peer / 网络）。 +NCCL Profiler 的目标不是把 NCCL 回调逐条存下来，而是回答一个分布式因果问题：一次 collective +变慢时，时间消耗在本 rank 尚未产出、等待对端、网络传输，还是设备执行。这个结论不能由单个 +时间戳给出，因此设计重点是保留事件之间的生命周期关系，并让查询层能够跨 rank 对齐证据。 -属于 [扩展机制](extensibility.zh.md) 中的 **路径 3**——由 NCCL 加载的 Rust `cdylib`，不是 Python 表插件。 +## 观测边界：为什么必须进入 NCCL -## 何时使用 +PyTorch API 层知道 `global_step` 和并行角色，但同步调用的返回时间、异步调用的 `work.wait()` 都只是 +host 侧边界，不能代表设备和 proxy 何时真正完成。PyTorch Flight Recorder 保存 watchdog 环形记录， +适合超时后的 collective 对齐，却不能持续分解正常通信的等待。只有 NCCL profiler plugin 能直接看到 +Collective、KernelCh、ProxyOp、ProxyStep 与 NetPlugin 回调。 -| 现象 | 工具 | -|------|------| -| step 变慢，不确定是通信还是计算 | `python.comm_collective` + skill `comm_bottleneck` | -| 哪个 rank 是 straggler？ | skill `slow_rank` | -| 已定位慢 rank，要区分 GPU 慢还是等网络 | `nccl.proxy_ops` + skill `nccl_culprit_victim` | -| 怀疑 RoCE / IB 拥塞 | `nccl.net_qp` + `rdma.mlx_hca` | +这三种证据因此保持独立：Python 层提供训练坐标，NCCL 层恢复运行时执行与等待，Flight Recorder +保存超时现场。它们不在回调路径互相调用，而是在查询时通过 communicator、sequence、rank 和 epoch +纳秒窗口关联。这样既保留各层真实的时间语义，也避免为方便 JOIN 而把 Python 状态带入 NCCL 通信线程。 -粗粒度 collective（`python.comm_collective`）**默认关闭**；需显式开启 -`PROBING_TORCH_COLLECTIVE_ENABLE=1` 或 `SET probing.torch.collective.enable=1` -（并配合 `PROBING=1`）。NCCL profiler 插件需要 **NCCL ≥ 2.26**（建议 PyTorch **2.8+**）；插件同时导出 **`ncclProfiler_v4`**（NCCL ≥ 2.27，优先）与 **`ncclProfiler_v3`**（NCCL 2.26），NCCL 自动协商最高版本。 +插件同时导出 `ncclProfiler_v4` 与 `ncclProfiler_v3`，由 NCCL 协商 ABI。v4 能提供 GPU globaltimer、 +per-communicator 信息和更完整的 peer 等待；v3 缺少的证据必须通过 `timing_source` 和默认值显式退化， +不能伪装成同等精度。 -## 三条 collective 采集路径 — 保持分叉，勿混淆 +## 事件生命周期：完成时间来自子事件 -probing 有三条相互独立的集合通信采集路径，**计时语义不同**，不可混用： +![NCCL 子事件重建执行窗口并分解等待位置](../assets/architecture/probing-nccl-event-model.svg) -| 来源 | 表 | 测的是什么 | 定位 | -|------|----|-----------|------| -| **NCCL profiler 插件**（本文档） | `nccl.coll_perf`、`nccl.proxy_ops`、`nccl.inflight_ops`、`nccl.net_qp` | NCCL 原生事件：重建的执行时间、等待分解、带宽 | **精准数据源** | -| Torch API 层插桩（遗留，`probing/profiling/collective/`） | `python.comm_collective` | `torch.distributed` API 调用的 Python 墙钟（launch 层） | 粗粒度回退；独有 `global_step` 上下文 | -| PyTorch Flight Recorder 桥 | `python.torch_nccl_flight_record`、`python.torch_nccl_pg_status` | torch 内部 watchdog 环形缓冲 | watchdog timeout / desync 取证 | +collective 的 `stopEvent` 只表示 host enqueue 已结束，kernel 和 proxy 可能仍在运行。如果在这里立即 +写行，`exec_time` 实际测到的只是 launch 开销。插件因而把 Collective 作为父对象，持有仍然活动的 +KernelCh 和 ProxyOp；ProxyOp 再持有 ProxyStep 的进展。只有最后一个子事件关闭，父对象才获得完整 +执行窗口并进入发布阶段。 -约定： +计时信号按证据质量逐级退化：优先使用 GPU globaltimer，其次使用 KernelCh 活动窗口，再其次使用 +ProxyOp 包络，最后才退回 host enqueue。选择结果写入 `timing_source`。这个字段不是展示信息，而是 +查询解释的组成部分：两个 `exec_time_ns` 只有在时间源质量可比较时才应直接比较。 -- Torch API 层插桩**默认关闭**（含多 rank 作业），避免隐性开销。需显式开启： - `PROBING_TORCH_COLLECTIVE_ENABLE=1` 或 `SET probing.torch.collective.enable=1` - （例如要与精准 NCCL 计时按 `global_step` 对齐）。 -- 查执行时间、带宽、等待归因，一律用 `nccl.*`。 - `python.comm_collective.duration_ms` **不是** NCCL 执行时间——`async_op` - 调用在 `work.wait()` 处收口，同步调用在 API 返回处收口。 -- 跨层关联：`nccl.*` 行不带训练步；需要时用 epoch 纳秒时间窗对 - `python.comm_collective.global_step` 对齐。 - -## 快速开始（Linux 训练） - -```bash -pip install probing # Linux wheel 自带 libprobing_nccl_profiler.so - -export NCCL_PROFILER_PLUGIN=$(python -m probing.nccl --plugin-path) -export NCCL_PROFILE_EVENT_MASK=$(python -m probing.nccl --event-mask) # 默认 94 -export PROBING=2 - -torchrun --nproc_per_node=8 train.py - -probing -t skill run nccl_culprit_victim -probing -t query " - SELECT rank, sum(send_gpu_wait_ns) AS gpu_wait, sum(recv_wait_ns) AS recv_wait - FROM nccl.proxy_ops - GROUP BY rank - ORDER BY recv_wait DESC" -``` - -### 可选：NetPlugin（IB QP 时延） - -```bash -export NCCL_PROFILE_EVENT_MASK=222 # 94 + NetPlugin 位 128 -probing -t query "SELECT * FROM nccl.net_qp LIMIT 20" -``` - -## macOS / 无 NCCL 开发机 - -```bash -PROBING=1 PROBING_NCCL_MOCK=1 python -m probing.nccl --seed-mock -probing -t skill run nccl_culprit_victim -``` +ProxyStep 不作为无限增长的明细表发布，而是在 ProxyOp 生命周期内累积为发送端 GPU 等待、peer +credit 等待、网络发送、接收和 flush 等待。这一决策用有界状态换取诊断所需的等待分解，避免消息 +切片数量直接放大存储量。等待字段仍只是证据：高 `send_gpu_wait_ns` 指向本 rank 产出不足，高 +`recv_wait_ns` 指向等待对端或网络；最终的 culprit/victim 判断还必须结合相同 sequence 的其他 rank、 +并行拓扑和系统状态。 -macOS 默认 `PROBING_NCCL_MOCK=auto`：在 `PROBING=1` 且无插件 `.so` 时自动写入 mock 表。 +## 回调并发：通信线程不能为诊断让路 -Mock 场景：**rank 2** = culprit（`send_gpu_wait_ns` 高），**rank 5** = victim（`recv_wait_ns` 高）。 +![NCCL 回调更新分片固定状态池，并在锁外写入完整事件](../assets/architecture/probing-nccl-write-path.svg) -## 数据表 +回调来自 host、proxy、NetPlugin 和 watchdog 等不同线程。若它们共享一把大锁或在回调中动态扩容， +诊断器本身就可能改变通信时序。插件使用固定容量 slot pool，并按 communicator hash 分片；一次回调 +通常只触碰一个 shard，容量和最坏分配成本在启动时已经确定。 -### `nccl.proxy_ops` +handle 由 shard、slot 和 generation 组成。slot 被回收后 generation 改变，因此迟到的 stop 回调 +无法关闭后来复用该 slot 的新事件。锁内只更新父子关系、状态和计数，并在事件完成时生成独立 row； +真正的 MEMT 追加发生在释放 shard 锁之后。这样存储抖动不会扩大 NCCL 临界区。 -每个 NCCL proxy op 一行，ProxyStep 等待在 op 结束时聚合。 +watchdog 采用 `try_lock`。分片繁忙时，它宁可跳过一次在途快照并增加计数，也不等待通信线程。 +这是诊断系统的优先级决策：允许可见的数据缺口，不允许为了观测 hang 而制造新的 hang。 -| 列 | 含义 | -|----|------| -| `ts` | 时间戳（纳秒） | -| `rank` | `torch.distributed` rank | -| `tp_rank`, `pp_rank`, `dp_rank` | 并行角色（`TP_RANK` / `PP_RANK` / `DP_RANK` 等 env）；未设置则为 `-1` | -| `comm_hash` | NCCL communicator hash | -| `coll_func` | collective 名称 | -| `seq` | collective 序号 | -| `channel_id` | NCCL channel | -| `peer` | 对端 rank | -| `is_send` | `1` 发送 proxy，`0` 接收 | -| `n_steps` | 聚合的 ProxyStep 数 | -| `trans_bytes` | 传输字节数（v4 按 step 级 `transSize` 累计） | -| `send_gpu_wait_ns` | **culprit 信号** — 本 GPU 未就绪 | -| `send_peer_wait_ns` | 等待接收端 clear-to-send credits（**仅 v4 ABI**，v3 为 0）— 对端拥塞信号 | -| `send_wait_ns` | 发送侧网络等待 | -| `recv_wait_ns` | **victim 信号** — 等待对端数据 | -| `recv_flush_wait_ns` | 接收 flush 等待 | - -多机：`global.nccl.proxy_ops`，带 `_host`、`_addr`、`_rank`。 - -> 所有 `nccl.*` 表的 `ts` 均为 **UNIX epoch 纳秒**，`global.nccl.*` 跨 rank 查询时时间戳可直接比较。 - -### `nccl.coll_perf` - -每个 collective / P2P 操作一行。 - -**计时模型。** NCCL 官方文档明确：collective 的 `stopEvent` 只表示 **host 侧 -enqueue 结束**——内核与 proxy 线程在其后继续工作。插件按官方 ext-profiler -推荐做法，对子事件（`ProxyOp`、`KernelCh`）做引用计数，并用子事件窗口重建 -真实执行时间。`timing_source` 列标注实际使用的信号： +## 发布模型：表代表不同阶段的事实 -| `timing_source` | 窗口 | 质量 | -|-----------------|------|------| -| `kernel_gpu` | GPU **globaltimer** 窗口：`kernelCh.pTimer`（起点）+ `KernelChStop` 状态（终点） | 最佳 — 设备时钟，**仅 v4 ABI** | -| `kernel_ch` | proxy 线程观测到的内核活动窗口（`ncclProfileKernelCh`） | NCCL 自身内核活动信号，host 时钟 | -| `proxy` | proxy op start→stop 包络 | 跨机操作较准 | -| `enqueue` | coll start→stop（仅 launch） | 回退 — 无 proxy/kernel 事件的机内操作 | - -| 列 | 含义 | -|----|------| -| `ts` | 完成时间戳（epoch 纳秒） | -| `rank`, `tp_rank`, `pp_rank`, `dp_rank` | 同 `nccl.proxy_ops` | -| `comm_hash`, `coll_func`, `seq` | 操作标识（P2P 的 `seq` 为 0） | -| `n_ranks` | 通信组大小（v4 per-comm `init` 提供；v3 为 `-1`） | -| `is_p2p` | `1` = Send/Recv，`0` = collective | -| `peer` | P2P 对端 rank（collective 为 `-1`） | -| `count`, `msg_size_bytes`, `dtype` | 消息负载：元素数 × dtype 字节数 | -| `algo`, `proto`, `n_channels` | NCCL 算法（Ring/Tree…）、协议（LL/LL128/Simple）、channel 数（v4 起 P2P 也有值） | -| `exec_time_ns` | 重建的真实执行耗时（见 `timing_source`） | -| `enqueue_time_ns` | host 侧 enqueue 耗时（NCCL coll start→stop） | -| `timing_source` | `kernel_gpu` / `kernel_ch` / `proxy` / `enqueue` | -| `algobw_gbps` | 算法带宽 `msg_size / exec_time`（GB/s）。**busbw** 在 SQL 中用 `n_ranks` 乘集合通信系数（如 AllReduce `2(n_ranks-1)/n_ranks`） | - -```sql --- 按带宽找最慢的 collective 分桶 -SELECT coll_func, msg_size_bytes, AVG(algobw_gbps) AS gbps, COUNT(*) AS n -FROM nccl.coll_perf -GROUP BY coll_func, msg_size_bytes -ORDER BY gbps ASC LIMIT 10 -``` - -### `nccl.inflight_ops` - -watchdog 周期快照：**已 start 未 stop** 的操作——挂死的 op 永远不会触发 -`stop_event`，因此不会出现在 `nccl.proxy_ops` 里；这张表补上了 hang 场景的盲区。 -列：`ts`, `rank`, `comm_hash`, `coll_func`, `seq`, `kind`（`coll`/`p2p`/`proxy_op`）, -`channel_id`, `peer`, `is_send`, `start_ns`, `age_ns`。 - -```sql --- 哪个 rank 卡在哪个操作上？ -SELECT rank, coll_func, seq, kind, MAX(age_ns)/1e9 AS stuck_secs -FROM nccl.inflight_ops -GROUP BY rank, coll_func, seq, kind -ORDER BY stuck_secs DESC -``` - -### `nccl.net_qp` - -IB QP 完成时延（需 NetPlugin mask）。列：`ts`, `rank`, `device`, `qp_num`, `wr_id`, `opcode`, `length`, `duration_ns`。 - -## Culprit 与 Victim - -- **Culprit**：某 rank `send_gpu_wait_ns` 突出 → 本地 GPU/计算慢，拖慢 collective 产出。 -- **Victim**：某 rank `recv_wait_ns` 突出 → 在等他人或网络。 - -同一 rank 可能在不同 collective 上同时出现两种模式。结合 `tp_rank`/`pp_rank`/`dp_rank` 与 Megatron 拓扑对齐分析。 - -## 诊断 skill：`nccl_culprit_victim` - -目录：`skills/nccl_culprit_victim/`（wheel 内：`python/probing/_skills/`）。 - -```bash -probing skill list -probing -t skill run nccl_culprit_victim -probing -t skill run nccl_culprit_victim --set seq_window=50 --global -``` - -步骤包括：各 rank wait 汇总、culprit/victim 排行、tp/pp/dp 角色视图、可选 `global` fan-out 与 `nccl.net_qp` 提示。 - -关联 skill：`slow_rank`、`comm_bottleneck`（粗粒度；有 `nccl.proxy_ops` 时会附带 NCCL 步骤）。 - -## 环境变量 - -| 变量 | 作用 | -|------|------| -| `NCCL_PROFILER_PLUGIN` | `libprobing_nccl_profiler.so` 路径 | -| `NCCL_PROFILE_EVENT_MASK` | 事件 mask；默认 `94` = Coll \| P2P \| ProxyOp \| ProxyStep \| KernelCh | -| `PROBING_DATA_DIR` | memtable 目录 | -| `PROBING_NCCL_MIN_MSG_BYTES` | 小于该字节数的操作不记录；默认 `0`（全记）。对应 NCCL Inspector 的 `DUMP_MIN_SIZE_BYTES` | -| `PROBING_NCCL_INFLIGHT_THRESHOLD_SECS` | watchdog：在途超过该秒数的操作快照进 `nccl.inflight_ops`；默认 `10`，`0` 关闭 | -| `PROBING_NCCL_POOL_SHARDS` | 按 comm hash 分片 slot pool（默认 `8`，范围 1–64）；降低多 comm 场景下回调锁竞争 | -| `PROBING_NCCL_MOCK` | 开发 mock：`auto` / `1` / `0` | -| `TP_RANK`, `PP_RANK`, `DP_RANK` | 写入 proxy_ops 角色列 | - -```bash -python -m probing.nccl --plugin-path -python -m probing.nccl --event-mask -python -m probing.nccl --seed-mock --ranks 8 --ops 5 -``` +完成的通信首先写入 `nccl.coll_perf`，其中保存重建后的执行窗口、算法、协议、消息规模和 +`timing_source`。同一通信的 proxy 等待被压缩到 `nccl.proxy_ops`，用于解释时间消耗在生产、peer +还是网络。尚未结束的事件没有完成行，watchdog 通过只读快照写入 `nccl.inflight_ops`，补上“永远 +不会触发 stop”的挂死盲区。启用 NetPlugin 后，QP 完成时延独立进入 `nccl.net_qp`，不强行嫁接到 +尚未证明的一次 collective 上。 -## 源码构建 - -```bash -make nccl-profiler-lib -cargo test -p probing-nccl-profiler -``` - -实现：`probing/extensions/nccl-profiler/`。架构细节见 crate [README](https://github.com/DeepLink-org/probing/blob/main/probing/extensions/nccl-profiler/README.md)。 - -## 真机验收清单（P0） - -1. 确认 NCCL ≥ 2.26 -2. `torchrun` 前设置 `NCCL_PROFILER_PLUGIN` -3. 若干 collective 后：`SELECT count(*) FROM nccl.proxy_ops` > 0 -4. `probing skill run nccl_culprit_victim` 有 rank 分解结果 - -## 相关文档 - -- [分布式训练](distributed.zh.md) -- [扩展机制](extensibility.zh.md) -- [AGENTS.md](https://github.com/DeepLink-org/probing/blob/main/AGENTS.md) +这些表不是四套相互竞争的答案，而是事件生命周期的四个投影。查询先用 `coll_perf` 找异常窗口，再用 +`proxy_ops` 分解等待；没有完成记录时转向 `inflight_ops`；只有出现网络等待证据时才继续关联 +`net_qp` 与 RDMA 指标。跨 rank 查询通过 `global.nccl.*` 在各进程本地过滤后汇总，训练 step 则由 +epoch 纳秒窗口与 Python 层坐标连接。聚合与因果判断属于查询层，不回灌到 collector。 + +`nccl.profiler_counters` 是上述证据的完整性边界。pool 耗尽、陈旧 handle、写入失败和 watchdog +跳过都会计数；诊断结果必须先检查这些信号，才能判断“没有事件”究竟是没有异常还是采集不完整。 + +## 失败边界与实现约束 + +NCCL 回调路径遵守三个约束：不等待远程节点，不执行跨 collector 调用，不因诊断失败改变 NCCL 的 +返回路径。状态池满时丢弃并计数，MEMT 写失败在锁外记录，watchdog 竞争失败只跳过快照。由此得到的 +数据可能带有明确缺口，但训练通信仍保持原有控制流。 + +精确表 schema 见 [SQL 表参考](../reference/sql-tables.zh.md)，环境与容量开关见 +[环境变量](../reference/env-vars.zh.md)，部署和查询示例见 +[性能分析示例](../examples/performance-analysis.zh.md)，跨 rank 诊断编排见 +[诊断 Skills](../guide/skills.zh.md)。实现位于 `probing/extensions/nccl-profiler/`。 diff --git a/docs/src/design/overhead-invariants.md b/docs/src/design/overhead-invariants.md deleted file mode 100644 index 5225aeef..00000000 --- a/docs/src/design/overhead-invariants.md +++ /dev/null @@ -1,32 +0,0 @@ -# TorchProbe overhead invariants (agents / maintainers) - -**Do not change semantics below without updating tests.** Read this before editing `web/src/overhead/`, `python/probing/profiling/torch_probe.py`, `python/probing/profiling/deferred_drain.py`, or `skills/health_overview/steps.yaml` overhead SQL. - -Background: [overhead.md](overhead.md). 中文: [中文版](/zh/design/overhead-invariants/). - -## Invariants (summary) - -| ID | Rule | -|----|------| -| **I1** | Primary % use **median** ratios, not `mean(probed)/mean(shadow)` | -| **I2** | Amortized = `(1−rate)×dispatch + rate×sampled`, not mean amortization | -| **I3** | `_record_step_timing` before `_drain_deferred` in `_close_step_wall` | -| **I4** | Deferred drain async by default (`PROBING_TORCH_DEFER_ASYNC=1`) | -| **I5** | Stable gate: `shadow_n≥5`, `dispatch_n≥16`, `shadow_baseline>0` | -| **I6** | UI soft formatting: `≈0%`, `~2%` for <5%; Typical vs Effective labels | - -## Test map - -```bash -cd web && cargo test overhead -PROBING=0 pytest tests/regression/profiling/ -q -``` - -| Tests | Guards | -|-------|--------| -| `web/src/overhead/metrics.rs` | I1, I2, I5, I6 | -| `tests/regression/profiling/test_overhead_invariants.py` | I3, I4 | -| `test_torch_probe_sampling.py` | I3, defer settle | -| `test_deferred_drain_worker.py` | I4 | - -See the Chinese doc for formulas, checklist, and the user regression fixture. diff --git a/docs/src/design/overhead-invariants.zh.md b/docs/src/design/overhead-invariants.zh.md deleted file mode 100644 index 8dd09bc7..00000000 --- a/docs/src/design/overhead-invariants.zh.md +++ /dev/null @@ -1,141 +0,0 @@ -# TorchProbe Overhead 不变量（Agent / 维护者必读） - -本文档列出 **不可随意改动的语义与公式**。修改 `web/src/overhead/`、`python/probing/profiling/torch_probe.py`、`python/probing/profiling/deferred_drain.py` 或 `skills/health_overview/steps.yaml` 中与 overhead 相关的逻辑前，请先读本文并更新对应测试。 - -完整背景见 [overhead.zh.md](overhead.zh.md)。 - ---- - -## 1. 核心不变量 - -### I1 — 主告警与 UI 主数字用 median，不用 mean 比值 - -| 指标 | 公式 | 禁止替代 | -|------|------|----------| -| `dispatch_overhead_pct` | `median(dispatch) ÷ median(shadow) − 1` | `mean(dispatch) ÷ mean(shadow)` | -| `blended_overhead_pct` | `median(all probed) ÷ median(shadow) − 1` | 混合 mean | -| `sampled_overhead_pct` | `median(sampled) ÷ median(shadow) − 1` | — | - -**原因**：训练步墙钟抖动大、`shadow_n` 少时，mean 比值会与 median 差一个数量级（例如 median ≈2% 而 mean 比值 ≈20%+），会误导用户。 - -**守护测试**：`web/src/overhead/metrics.rs` → `amortized_not_mean_ratio_when_means_diverge` - ---- - -### I2 — Amortized（Effective overhead）= 采样率加权，不是 mean 摊销 - -``` -amortized = (1 − rate) × dispatch_overhead + rate × sampled_overhead -``` - -- `rate`：配置 `sample_rate`，否则 `sampled_n / probed_n` -- 无采样步时：`amortized == dispatch_overhead` - -**禁止**：`mean(probed) ÷ mean(shadow) − 1` 作为 Web UI 的 Amortized / Effective overhead。 - -**守护测试**：`amortized_blends_dispatch_and_sampled_by_rate`、`amortized_not_mean_ratio_when_means_diverge` - ---- - -### I3 — `step_duration_sec` 记时边界 - -在 `TorchProbe.post_step_hook` / `_close_step_wall` 中，顺序必须为： - -``` -_record_step_timing() # 墙钟终点 -_drain_deferred() # deferred 回收（可在后台线程执行 save） -_advance_step_cycle_for_next() -_mark_step_wall_start() # 下一步起点 -``` - -**禁止**：在 `_record_step_timing()` **之前**调用 `_drain_deferred()`（会把前几步 event 回收算进本步墙钟）。 - -**守护测试**：`tests/regression/profiling/test_torch_probe_sampling.py::test_post_step_hook_drains_deferred_after_step_timing`、`test_overhead_invariants.py::test_close_step_wall_source_order` - ---- - -### I4 — Deferred 回收默认异步 - -- 默认 `PROBING_TORCH_DEFER_ASYNC=1`：ready 的 `DelayedRecord` 入队，后台线程 `elapsed_time` + `save()` -- 队列满 → 主线程同步 `save()` 回退（不丢数据） -- 进程退出 `atexit` flush - -**禁止**：在无测试、无文档的情况下改回「仅在训练线程同步 drain」作为唯一路径。 - -**守护测试**：`tests/regression/profiling/test_deferred_drain_worker.py` - ---- - -### I5 — 稳定性门控 - -百分比在 UI 上视为「稳定」需同时满足： - -- `shadow_baseline > 0` -- `shadow_n ≥ 5`（`MIN_SHADOW_SAMPLES`） -- `dispatch_n ≥ 16`（`MIN_DISPATCH_SAMPLES`） - -`dispatch_overhead_pct` / `blended_overhead_pct` 在不稳定时不应展示为精确告警数字（可为 `—` 或 muted）。 - -**守护测试**：`snapshot_computes_dispatch_overhead`、`skills/health_overview` SQL 中的 `dispatch_n` / `shadow_n` - ---- - -### I6 — 展示语义（L4 Web） - -| 规则 | 说明 | -|------|------| -| 低开销显示 | `|pct| < 0.5%` → `≈0%`；`< 5%` → `~N%`（避免 `+1.9%` 告警感） | -| 主指标命名 | UI 主卡：「Typical overhead」= dispatch；「Effective overhead」= amortized | -| 训练日志对齐 | `torch_step_timing` 为 hook-to-hook 墙钟，含 DataLoader 等待；不等于仅 compute 的 `time=49ms` 打印 | - -**守护测试**：`format_pct_signed_*`、`sidebar_copy_when_stable` - ---- - -## 2. 测试地图 - -| 文件 | 守护的不变量 | -|------|----------------| -| `web/src/overhead/metrics.rs` (`#[cfg(test)]`) | I1, I2, I5, I6 | -| `tests/regression/profiling/test_overhead_invariants.py` | I3, I4（源码顺序 / 默认 env） | -| `tests/regression/profiling/test_torch_probe_sampling.py` | I3, deferred settle 窗口 | -| `tests/regression/profiling/test_deferred_drain_worker.py` | I4 | -| `skills/health_overview/steps.yaml` | I1 告警列 `dispatch_overhead_pct` | - -本地命令： - -```bash -# Rust Web 指标 -cd web && cargo test overhead - -# Python hook / drain -PROBING=0 pytest tests/regression/profiling/ -q -``` - ---- - -## 3. Agent 修改前检查清单 - -1. 是否改动 overhead **公式**？→ 更新本文 §1 + `metrics.rs` 测试 + `overhead.zh.md` -2. 是否改动 **hook 顺序**？→ 更新 `test_post_step_hook_drains_deferred_after_step_timing` 与 `test_close_step_wall_source_order` -3. 是否改动 **异步 drain** 默认？→ 更新 `deferred_drain.py` 测试与 §I4 -4. 是否只改 UI 文案？→ 保持 I6；跑 `cargo test -p web overhead` -5. **不要** 用 mean 比值「修复」amortized 与 median 不一致 — 那是预期行为 - ---- - -## 4. 参考场景（回归夹具） - -用户实测（median 一致、mean 失真）： - -| 观测 | 值 | -|------|-----| -| shadow median | 180 ms | -| dispatch median | 166 ms | -| dispatch overhead | ≈ +1.9% | -| shadow mean | 130 ms | -| probed mean | 533 ms | -| mean 比值（禁止作 amortized） | ≫ 20% | -| 期望 amortized（rate≈5%） | ≈ +1.5% ~ +2% | - -此场景编码在 `amortized_not_mean_ratio_when_means_diverge` 测试中。 diff --git a/docs/src/design/overhead.md b/docs/src/design/overhead.md index 731b8ab3..753fcfa6 100644 --- a/docs/src/design/overhead.md +++ b/docs/src/design/overhead.md @@ -1,334 +1,144 @@ -# Overhead measurement and formulas +# Overhead Control and Measurement Architecture -Canonical design doc for **instrumentation overhead** in Probing: terminology, formulas, measurement boundaries, and offline vs in-run methods. Implementation details: [Profiling](profiling.md); table columns: [SQL tables — torch_step_timing](../reference/sql-tables.md#python-torch_step_timing). +Instrumentation overhead is not one counter for time spent inside a collector. Hook dispatch, +sampled-step work, GPU timing reads, MEMT writes, and NCCL callbacks cross different execution +boundaries. Combining them into one percentage cannot guide sampling or locate a regression. +Probing first isolates cost architecturally, then defines what each measurement can explain. -**Invariants and regression tests** (read before changing formulas or hook order): [overhead-invariants.md](overhead-invariants.md). +See [Profiling architecture](profiling.md) for implementation and +[SQL tables — torch_step_timing](../reference/sql-tables.md#python-torch_step_timing) for the data +contract. -## 1. Goals and scope +## 1. How cost is decomposed -### Questions answered +Each TorchProbe optimizer step first enters either the probed or shadow path. A shadow step retains +the complete training workload and all other collectors, but TorchProbe module and optimizer hooks +return at entry. It is an interleaved baseline in the same job, not "pure training with no +observation." -| Question | Typical consumer | -|----------|------------------| -| How much slower is each training step with TorchProbe hooks? | Production jobs, `health_overview`, Web Overhead panel | -| What does span / memtable persistence cost? | `make bench` tracing layer, regression tests | -| End-to-end delta on a real TinyNet loop? | `make bench` torch_train layer | -| NCCL profiler impact on collective latency? | `run_nccl_profiler_bench.sh` (offline) | +A probed step then follows one of two paths. An unsampled step pays only hook dispatch and step +timing. A sampled step also records module events, prepares results, and writes tables. The first is +a fixed cost that long-running observation may pay every step; the second is an occasional heavy +cost amortized by the sample rate. They must remain separate because lowering the rate reduces the +heavy path but cannot remove dispatch through hooks already attached to the model tree. -### Out of scope +In-run shadow isolates TorchProbe only. Standalone instrumentation benchmarks measure Span/MEMT +components, and paired end-to-end training benchmarks check their composition. The NCCL profiler +has no in-run shadow because disabling NCCL callbacks would change the communication path being +measured; it requires an offline A/B over the same collectives. -- **Business throughput SLOs** (tokens/s, time-to-convergence) — requires your own baseline job -- **NCCL collective execution time** as “probe tax” — that is training work, not instrumentation -- **“Pure PyTorch” on shadow steps** — other collectors (NCCL, CPU/GPU) still run on shadow steps +## 2. Why the control path is organized this way -### vs `probing bench` CLI +![TorchProbe combines deterministic sampling, deferred GPU reads, shadow measurement, and optional adaptation](../assets/architecture/probing-sampling-overhead.svg) -| Entry | Measures | -|-------|----------| -| `make bench` → `examples/overhead/bench_instrumentation.py` | Python/Rust **instrumentation** wall time | -| `probing bench write` (hidden CLI) | memtable **write-path** throughput/latency — not training hook tax | +Step and layer gates control different dimensions: how often to inspect deeply and how much of one +inspection to cover. The step gate depends only on step number, so ranks select the same steps. The +layer gate hashes `(step, layer)` deterministically, reducing coverage while preserving cross-rank +comparability. The default `rate=0.05`, `layer_rate=1.0` keeps complete module relationships for a +small set of steps instead of producing unrelated fragments on every step. -These are not interchangeable. +GPU event recording and reading are separated. A sampled step submits events; elapsed time is read +after a settle window. The default asynchronous worker uses a bounded queue of 4096 items. A full +queue falls back to synchronous save rather than growing memory without bound, and process exit +flushes it. This makes the resource bound and the no-silent-loss behavior explicit. ---- +Shadow steps are interleaved at `4:1` by default so probed and baseline paths experience similar +data, collective, and system noise. Adaptive sampling is off by default. When enabled, it may act +only after `shadow_n ≥ 5` and `dispatch_n ≥ 16`, and it may never raise the rate above the user's +initial value. The controller can reduce cost when evidence is sufficient; it cannot autonomously +increase observation intensity. -## 2. Terminology +## 3. Timing boundaries and statistical semantics -| Term | Definition | -|------|------------| -| **probed step** | `python.torch_step_timing.is_shadow = 0`; TorchProbe module hooks run per config | -| **shadow step** | `is_shadow = 1`; module/optimizer hooks **return immediately**; no `python.torch_trace`, but a timing row is written | -| **shadow cadence** | Default `4:1` — 4 probed steps then 1 shadow step (`shadow=4:1`) | -| **sampled step** | probed with `sampled = 1` — module trace flush (may include GPU defer/sync) | -| **hook tax** | Median overhead %: all probed steps vs shadow | -| **sampled overhead** | Median overhead %: sampled probed steps only vs shadow | -| **total overhead (amortized)** | Weighted: `(1−rate)×dispatch_overhead + rate×sampled_overhead` (Web UI) | -| **in-run** | Continuous comparison via `python.torch_step_timing` during training | -| **offline** | Standalone benchmark scripts with A/B or paired comparison | +![TorchProbe step timing boundary and deferred-drain order](../assets/architecture/probing-overhead-timing-window.svg) ---- +`step_duration_sec` starts at `_mark_step_wall_start()` at the end of the previous optimizer +`post_step_hook` and ends at `_record_step_timing()` in the current one. Only then does +`_drain_deferred()` run, followed by state advance and the next start marker. The current step thus +contains its training work and hook teardown but is not charged for GPU-event recovery from earlier +steps. -## 3. Common formulas +The `train.step` span measures the user-wrapped compute interval and excludes hook dispatch and +persistence. `step_duration_sec` intentionally includes those boundary costs. The two metrics +answer different questions and cannot be subtracted or substituted for one another. -### 3.1 Relative overhead (ratio) - -Used for TorchProbe in-run and many offline cases: - -$$ -\text{overhead\_pct} = \left(\frac{M_{\text{probed}}}{M_{\text{shadow}}} - 1\right) \times 100 -$$ - -$M$ is an aggregate (see §5). SQL uses `nullif(..., 0)` when the denominator is zero. - -### 3.2 Relative baseline (offline A/B) +Runtime aggregation uses medians to resist data-loading, collective, and scheduling spikes. Let +$M_s$ be median shadow duration, $M_d$ median unsampled probed duration, and $M_p$ median sampled +probed duration: $$ -\text{vs\_baseline\_pct} = \left(\frac{T_{\text{measured}}}{T_{\text{baseline}}} - 1\right) \times 100 +\text{dispatch} = \left(\frac{M_d}{M_s}-1\right)\times100\%, \qquad +\text{sampled} = \left(\frac{M_p}{M_s}-1\right)\times100\% $$ -### 3.3 Paired training delta +For sample rate $r$, amortized effective overhead is: $$ -\Delta T = T_{\text{instrumented}} - T_{\text{baseline}}, \quad T_{\text{inst\_med}} = \text{median}(T_{\text{baseline}}) + \text{median}(\Delta T) +\text{effective}=(1-r)\times\text{dispatch}+r\times\text{sampled} $$ -### 3.4 NCCL latency delta - -$$ -\text{pct\_delta}(ref, x) = 100 \times \frac{x - ref}{ref} -$$ - -Positive means the profiled mode is slower. - ---- - -## 4. In-run measurement: TorchProbe shadow step - -### 4.1 Mechanism - -With `PROBING_TORCH_PROFILING=on` (default includes `shadow=4:1`), each optimizer step writes one row to `python.torch_step_timing`. - -**Shadow detection** (`shadow_step_in_cycle`): - -```text -cycle_len = shadow_normal + shadow_baseline # default 4 + 1 = 5 -is_shadow = (cycle_index % cycle_len) >= shadow_normal -``` - -Indices 0–3 are probed; index 4 is shadow (default cadence). - -### 4.2 Timing window (`step_duration_sec`) - -Uses `time.perf_counter()` wall clock — **not** GPU kernel time. - -**Boundaries**: - -- **Start**: `_mark_step_wall_start()` at the end of the previous step’s `post_step_hook` -- **End**: `_record_step_timing()` inside the current step’s `post_step_hook` (**before** `_drain_deferred()`) - -So `step_duration_sec` covers **full step compute (forward/backward/opt) plus in-step hook teardown** (including same-step trace flush on sampled steps). It **excludes** deferred GPU `elapsed_time` drain from earlier sampled steps (drain runs after timing, before the next `_mark_step_wall_start()`). - -**vs `train.step` span**: from `python.trace_event`, measures the wrapped compute interval only — **excludes** hook dispatch and persistence. The Web UI shows both; do not compare numbers directly. - -### 4.3 Hook behavior by step type - -| Type | Module hooks | Writes torch_trace | Timing row | -|------|--------------|-------------------|------------| -| discovery (first step) | register modules | no | special path | -| probed, not sampled | short-circuit | no | `is_shadow=0, sampled=0` | -| probed, sampled | full path | yes | `is_shadow=0, sampled=1` | -| shadow | immediate return | no | `is_shadow=1` | - -`hook_tax` blends sampled and non-sampled probed steps. **`dispatch_overhead`** (`sampled=0` only) is the stable primary metric. `sampled_overhead` is the heavy path. - ---- - -## 5. Metric catalog - -### 5.1 In-run metrics - -| Metric | Numerator | Denominator | Aggregate | Window | Consumer | -|--------|-----------|-------------|-----------|--------|----------| -| **dispatch_overhead_pct** | probed & `sampled=0` | shadow | **median** | rolling **80** steps | **primary alert** | -| **hook_tax_pct** | all probed | shadow | **median** | rolling **80** steps | blended / soak | -| **sampled_overhead_pct** | probed & `sampled=1` | shadow | **median** | rolling **80** steps | Web | -| **Dispatch (Web)** | `sampled=0` probed | shadow | **median** | last **80** steps | Web sidebar / panel | -| **Total overhead (Web)** | dispatch + sampled weighted by sample rate | — | **weighted %** | last 80 steps | amortized display | - -**Reference SQL** (rolling window + stratified metrics): - -```sql -WITH bounds AS ( - SELECT GREATEST(COALESCE(MAX(local_step), 0) - 80, 1) AS win_start - FROM python.torch_step_timing -) -SELECT - round((median(CASE WHEN is_shadow = 0 AND sampled = 0 THEN step_duration_sec END) - / nullif(median(CASE WHEN is_shadow = 1 THEN step_duration_sec END), 0) - 1) * 100, 2) - AS dispatch_overhead_pct, - round((median(CASE WHEN is_shadow = 0 THEN step_duration_sec END) - / nullif(median(CASE WHEN is_shadow = 1 THEN step_duration_sec END), 0) - 1) * 100, 2) - AS hook_tax_pct, - sum(CASE WHEN is_shadow = 0 AND sampled = 0 THEN 1 ELSE 0 END) AS dispatch_n, - sum(CASE WHEN is_shadow = 1 THEN 1 ELSE 0 END) AS shadow_n -FROM python.torch_step_timing, bounds -WHERE local_step >= bounds.win_start AND local_step > 1; -``` - -**Display** (Web): $|\text{pct}| < 0.5$ → `≈0%`; treat as stable when `shadow_n ≥ 5` and `dispatch_n ≥ 16` (§5.2, §11). - -### 5.2 Sample size - -| Condition | Behavior | -|-----------|----------| -| `shadow_baseline = 0` (`shadow=off`) | overhead % undefined | -| `shadow_n < 5` | low-sample hint; % is indicative only | -| `dispatch_n < 16` | high variance on dispatch overhead | -| `probed_n = 0` or `shadow_n = 0` | soak skips ratio assertion | - -### 5.3 Auxiliary metrics - -| Metric | Source | Purpose | -|--------|--------|---------| -| `train_step_median_ms` | `median(span_end - span_start)` for `train.step` | compute-only reference | -| `nccl.profiler_counters` | latest row | data health, not overhead % | - ---- - -## 6. Offline benchmark: `make bench` +This must not become `mean(probed)/mean(shadow)`: the probed set mixes light and heavy paths, while +the smaller shadow set is sensitive to long-tail steps. Historical `hook_tax` uses all probed-step +medians and remains only for compatibility and as a conservative upper bound. -Script: `examples/overhead/bench_instrumentation.py`. Run inside a **probing-injected** process: +## 4. When the measurement is trustworthy -```bash -PROBING=1 make bench -PROBING=1 make bench-quick -``` - -### 6.1 Parameters - -| Parameter | full | `--quick` | -|-----------|------|-----------| -| span_iters | 300 | 80 | -| probe_steps | 40 | 12 | -| train batches | 30 | 8 | -| warmup (outer rounds) | 2 | 1 | -| runs (outer rounds) | 5 | 3 | - -### 6.2 Three layers - -**A — tracing**: median total wall time per scenario; `vs_baseline_pct` vs `span (no backend)`. - -**B — torch_probe (synthetic)**: fake single module; median per-step timing; `hook_tax_pct` on `shadow=4:1` uses §3.1. Not representative of large production models. - -**C — torch_train (TinyNet)**: paired back-to-back baseline vs instrumented; reports `paired_delta_ms` and `inst_med`. - -### 6.3 JSON export - -```bash -PROBING=1 python examples/overhead/bench_instrumentation.py --json-out /tmp/bench.json -``` - ---- - -## 7. NCCL profiler overhead (offline only) - -No in-run shadow. Runtime health: `nccl.profiler_counters`. - -E2E: `examples/overhead/nccl_profiler_overhead.py`, `examples/overhead/run_nccl_bench.sh`. Baseline vs plugin + `PROBING=2`. Compare with §3.4 on latency and throughput. - -Micro: `probing/extensions/nccl-profiler/benches/callback_path.rs` (Criterion) — component-level only. - ---- - -## 8. Other subsystems - -| Subsystem | Method | Threshold | -|-----------|--------|-----------| -| Span + memtable | `tests/regression/profiling/test_span_overhead.py` | `T_on < T_off × 8 + 0.05s` | -| TorchProbe module spans | same | `med_on < med_off × 6 + 0.02s` | -| memtable writes | `probing bench write` | throughput / latency | -| pprof | `probing.pprof.sample_freq` | qualitative | - -Isolate stack cost: `PROBING_SPAN_BACKENDS=none`. +The Web UI and diagnostic skills use the latest 80 steps, covering several shadow cycles without +retaining distant cold-start noise. With `shadow_n < 5` or `dispatch_n < 16`, the result remains a +collecting or low-confidence estimate and cannot trigger a stable alert. With `shadow=off`, the +denominator does not exist and in-run overhead percentage is undefined. Absolute values below +`0.5%` render as `≈0%` rather than turning timer resolution and natural jitter into false precision. ---- +Noise is handled by source separation. Sampled heavy steps do not enter dispatch; rolling medians +suppress step spikes; discovery, JIT, and cache warmup stay outside the stable window; deferred +drain occurs after timing; and each rank computes its stratified metrics before cross-rank +comparison rather than mixing different workloads into one mean. -## 9. Gates (current repo) +`nccl.profiler_counters`, queue saturation, and write failures describe evidence integrity, not an +overhead percentage. When events are absent, a diagnosis must exclude a collection gap before +claiming there was no additional cost. -| Gate | Condition | Location | -|------|-----------|----------| -| diagnostic warning | `dispatch_overhead_pct > 5%` | `health_overview` | -| soak failure | `hook_tax_pct > 75%` default | `soak_assert.py` | -| CI regression | span ratio bounds | `test_span_overhead.py` | +## 5. Why offline validation remains necessary -5%, 75%, and 8× serve different purposes — not a single SLO. +In-run measurement matches production workload but sees only TorchProbe relative to shadow. The +offline benchmark therefore has three layers: tracing isolates span-stack and persistence cost; +synthetic TorchProbe validates hook and sampling state transitions; TinyNet uses back-to-back paired +deltas to validate real forward/backward/optimizer composition. These are not three product +metrics, but a chain of evidence that narrows a regression from end to end toward a component. ---- +NCCL follows a separate chain. Baseline and profiled runs use the same message size, warmup, and +synchronization boundary and compare collective latency and throughput. Criterion measures only +callback, slot-pool, and clock-read components and cannot replace collective E2E results. -## 10. Known biases +The repository's 5% diagnostic warning, 75% soak bound, and component ratio gates serve different +layers. They are not one performance SLO and do not replace release calibration on the target model +and hardware. -1. Shadow is not vacuum — other collectors still run. -2. Sampled steps may include GPU defer/sync in wall time. -3. Web uses 80-step window; skills use full history. -4. Synthetic bench ≠ production. -5. Filter `local_step > 1` to skip discovery. +## 6. Invariants for changes {#change-invariants} ---- +This table is a change-safety contract, not a second overhead model. -## 11. Anti-noise and stability +| Invariant | Required behavior | Guard | +|-----------|-------------------|-------| +| Primary percentage | median dispatch/shadow ratio; never `mean(probed)/mean(shadow)` | `web/src/overhead/metrics.rs` | +| Amortized overhead | `(1-rate)×dispatch + rate×sampled` | `amortized_blends_dispatch_and_sampled_by_rate` | +| Hook order | `_record_step_timing()` → `_drain_deferred()` → advance → `_mark_step_wall_start()` | Python overhead/sampling regression tests | +| Async drain | `PROBING_TORCH_DEFER_ASYNC=1` default; bounded queue, sync fallback, exit flush | `test_deferred_drain_worker.py` | +| Stability | stable only when `shadow_n ≥ 5` and `dispatch_n ≥ 16` | Web metrics and `health_overview` | +| UI meaning | Typical=dispatch; Effective=rate-weighted; `abs(pct)<0.5%` renders `≈0%` | Web formatting/copy tests | -### 11.1 Noise sources - -| Source | Mitigation | -|--------|------------| -| Sampled steps mixed into hook tax | Use **`dispatch_overhead`** (`sampled=0`) as primary | -| Step-time jitter (data, collectives) | **median** + 80-step rolling window | -| Few shadow points | Require `shadow_n ≥ 5`; prefer long runs | -| Cold start / discovery | `local_step > 1`; offline warmup | -| Blended aggregation | Report dispatch / sampled / blended separately | -| Deferred drain blocks hook | After timing: async worker (`PROBING_TORCH_DEFER_ASYNC=1`, default) + bounded queue; sync fallback when full | - -### 11.2.1 Async deferred drain - -Ready `DelayedRecord` items are enqueued after `_record_step_timing`; a daemon thread runs `elapsed_time` + `save()`. Set `PROBING_TORCH_DEFER_ASYNC=0` for synchronous drain (tests). Queue size: `PROBING_TORCH_DEFER_QUEUE_SIZE` (default 4096). `atexit` flushes the queue; tests may call `flush_deferred_drain()`. - -### 11.2 Principles - -1. **Stratify** light (dispatch) vs heavy (sampled) paths. -2. **Robust aggregates**: median for alerts; amortized uses sample-rate-weighted blend, not `mean(probed)/mean(shadow)`. -3. **Aligned window**: Web and `health_overview` use **80 steps**. -4. **Paired offline bench** for torch_train layer. -5. **Explicit gates**: no alert until `shadow_n ≥ 5` and `dispatch_n ≥ 16`. - -### 11.3 Metric pick list - -| Goal | Metric | -|------|--------| -| Daily alert / sidebar | `dispatch_overhead_pct` | -| Sampling cost | `sampled_overhead_pct` | -| Legacy / soak | `hook_tax_pct` (blended, conservative) | - -### 11.4 Tuning - -```bash -PROBING_TORCH_PROFILING=on,shadow=8:2,rate=0.05 # more shadow points per window -``` - -### 11.5 Future work - -Trimmed mean, per-cadence aggregation, EWMA sidebar, confidence bands, NCCL in-run shadow. - ---- - -## 12. Operations +After changing formulas, hook order, or async-drain defaults, run: ```bash -PROBING=1 PROBING_TORCH_PROFILING=on python train.py -PROBING=1 python examples/overhead/torch_probe_overhead_smoke.py -PROBING=1 make bench-quick +cd web && cargo test overhead +PROBING=0 pytest tests/regression/profiling/test_overhead_invariants.py \ + tests/regression/profiling/test_torch_probe_sampling.py \ + tests/regression/profiling/test_deferred_drain_worker.py -q ``` -Reduce overhead: lower `rate` / `layer_rate`; disable `trace_spans`, `sync=on`, `backward=on`; use `shadow=off` only when in-run estimates are not needed. - ---- - -## 13. Implementation index - -| Component | Path | -|-----------|------| -| Shadow + timing | `python/probing/profiling/torch_probe.py` | -| Offline bench | `examples/overhead/bench_instrumentation.py` | -| Web SQL | `web/src/overhead/sql.rs` | -| Web formatting | `web/src/overhead/metrics.rs` | -| Skill SQL | `skills/health_overview/steps.yaml` | -| soak | `examples/imagenet/soak_assert.py` | -| NCCL E2E | `examples/overhead/nccl_profiler_overhead.py` | - ---- - -## Related docs +## Related documentation -- [Profiling](profiling.md) -- [Tracing spans](tracing-spans.md) -- [NCCL Profiler](nccl-profiler.md) -- [Troubleshooting — High Overhead](../guide/troubleshooting.md) +- [Profiling architecture](profiling.md) +- [Data layer](data-layer.md) +- [NCCL Profiler architecture](nccl-profiler.md) +- [SQL table reference](../reference/sql-tables.md) diff --git a/docs/src/design/overhead.zh.md b/docs/src/design/overhead.zh.md index d86aefb7..db88d475 100644 --- a/docs/src/design/overhead.zh.md +++ b/docs/src/design/overhead.zh.md @@ -1,443 +1,125 @@ -# Overhead 测量与公式定义 +# 开销控制与测量架构 -本文是 Probing **instrumentation overhead** 的顶层设计文档：统一术语、公式、测量边界与离线/在线基准方法。实现细节见 [性能分析](profiling.zh.md)；表字段见 [SQL 表 — torch_step_timing](../reference/sql-tables.zh.md#python-torch_step_timing)。 +观测开销不是采集器运行时间的一个简单计数。hook 派发、被采样 step 的重路径、GPU 时间读取、MEMT +写入和 NCCL 回调跨越不同执行边界。如果把它们混成一个百分比，既无法指导采样决策，也无法判断回归 +来自哪里。Probing 因而先在架构上隔离成本，再为每条路径定义可解释的测量方法。 -**不可变语义与回归测试**（修改公式 / hook 顺序前必读）：[overhead-invariants.zh.md](overhead-invariants.zh.md)。 +实现见[性能采集架构](profiling.zh.md)，字段契约见 +[SQL 表 — torch_step_timing](../reference/sql-tables.zh.md#python-torch_step_timing)。 -## 1. 目标与范围 +## 1. 成本如何被拆开 -### 要回答的问题 +TorchProbe 的每个 optimizer step 先进入 probed 或 shadow 路径。shadow step 仍保留完整训练负载和 +其他采集器，但 TorchProbe module/optimizer hook 在入口短路；它因此是同一次作业内的交错基线，而 +不是“没有任何观测的纯训练”。 -| 问题 | 典型场景 | -|------|----------| -| TorchProbe 模块 hook 让每步慢多少？ | 生产训练、`health_overview`、Web Overhead 面板 | -| Span / memtable 持久化成本？ | `make bench` tracing 层、回归测试 | -| 真实 TinyNet 训练上的端到端增量？ | `make bench` torch_train 层 | -| NCCL profiler 对 collective 延迟的影响？ | `run_nccl_profiler_bench.sh`（离线） | +probed step 再分成两条路径。未命中采样时只支付 hook dispatch 和 step timing，命中采样时还要记录 +module 事件、整理结果并写表。前者代表长期运行中每步都可能支付的固定成本，后者代表按采样率偶发的 +重路径成本。把两者分开，是因为降低采样率只能摊薄重路径，不能消除 hook 已经挂到模型树上的派发成本。 -### 明确不衡量 +运行中 shadow 只能隔离 TorchProbe。Span/MEMT 的组件成本通过独立 instrumentation benchmark 测量， +完整训练路径通过成对端到端基准验证。NCCL profiler 没有运行中 shadow，因为关闭 NCCL 回调会改变 +被测通信路径本身；它必须用相同 collective 的离线 A/B 衡量。 -- **业务吞吐 SLO**（tokens/s、收敛速度）——需用户自有 baseline job 对比 -- **NCCL collective 本身的执行时间**（那是训练逻辑，不是探针税） -- **shadow step 上的「纯 PyTorch」**——shadow 仍运行 NCCL/CPU/GPU 等其他采集器 +## 2. 控制路径为什么这样组织 -### 与 `probing bench` CLI 的区别 +![TorchProbe 组合确定性采样、GPU 延后读取、shadow 测量和可选自适应](../assets/architecture/probing-sampling-overhead.svg) -| 入口 | 测量对象 | -|------|----------| -| `make bench` → `examples/overhead/bench_instrumentation.py` | Python/Rust **instrumentation** 墙钟开销 | -| `probing bench write`（隐藏 CLI） | memtable **写路径吞吐/延迟**，不是训练 hook 税 | +step gate 与 layer gate 分别控制“多久深入一次”和“一次深入多少层”。step gate 只依赖 step 序号， +所有 rank 因而命中同一批 step；layer gate 使用 `(step, layer)` 的确定性哈希，在保留跨 rank 可比性的 +同时降低单步覆盖面。默认 `rate=0.05`、`layer_rate=1.0`，意味着少量 step 保留完整 module 关系， +而不是每步只得到彼此无法拼接的零散 layer。 -二者不可互换。 +GPU event 的记录与读取被拆开。采样 step 只提交 event，经过 settle 窗口后再读取 elapsed time；默认 +异步 worker 使用容量 4096 的有界队列。队列满时同步回退而不是无限占用内存，进程退出时 flush， +从而在资源有界和数据不静默丢失之间取得明确平衡。 ---- +shadow 默认按 `4:1` 交错插入，使 probed 与 baseline 经历相近的数据、collective 和系统噪声。自适应 +采样默认关闭；显式开启后，只有 `shadow_n ≥ 5` 且 `dispatch_n ≥ 16` 才允许调整，并且不能超过用户 +设置的初始 rate。控制器因此只能在有足够证据时降低成本，不能自行扩大观测强度。 -## 2. 术语表 +## 3. 计时边界与统计语义 -| 术语 | 定义 | -|------|------| -| **probed step** | `python.torch_step_timing.is_shadow = 0` 的正常训练步；TorchProbe 模块 hook 按配置执行（shadow 步除外） | -| **shadow step** | `is_shadow = 1` 的基线步；模块/optimizer hook **立即返回**，不写 `python.torch_trace`，但仍写 timing 行 | -| **shadow cadence** | 默认 `4:1`：每 5 个 optimizer step 中 4 个 probed、1 个 shadow（`shadow=4:1`） | -| **sampled step** | probed 且 `sampled = 1`：该步会 flush module 级 trace（可能含 GPU defer/sync） | -| **hook tax（混合）** | 中位数口径：全部 probed step（含采样+非采样）相对 shadow | 噪声较大，保留兼容 | -| **dispatch overhead** | 中位数口径：仅 `sampled=0` 的 probed step 相对 shadow | **推荐主指标**，抗采样步干扰 | -| **sampled overhead** | 中位数口径：仅 sampled probed step 相对 shadow | 重路径（含 flush/sync） | -| **total overhead（摊销）** | 加权口径：`(1−rate)×dispatch_overhead + rate×sampled_overhead`（Web UI） | -| **in-run** | 训练进程内，靠 `python.torch_step_timing` 持续对比 | -| **offline** | 独立基准脚本，A/B 或 paired 对比 | +![TorchProbe 的 Step 计时边界与 deferred drain 顺序](../assets/architecture/probing-overhead-timing-window.svg) ---- +`step_duration_sec` 从上一 optimizer `post_step_hook` 末尾的 `_mark_step_wall_start()` 开始，到当前 +`post_step_hook` 中的 `_record_step_timing()` 结束。随后才执行 `_drain_deferred()`，最后推进状态并 +开始下一步。这个顺序保证当前 step 包含本步训练计算和 hook 收尾，却不把前几步 GPU event 的回收 +成本错误归给当前 step。 -## 3. 通用公式 +`train.step` span 测量用户包裹的计算区间，不包含 hook 派发和持久化；`step_duration_sec` 则有意覆盖 +这些边界成本。两者回答的问题不同，不能直接相减或互相替代。 -### 3.1 相对 overhead（比值法） - -适用于 TorchProbe in-run 与多数离线场景： +运行时使用中位数抵抗数据加载、collective 和调度毛刺。设 $M_s$ 为 shadow step 时长中位数， +$M_d$ 为未采样 probed step 中位数，$M_p$ 为采样 probed step 中位数，则： $$ -\text{overhead\_pct} = \left(\frac{M_{\text{probed}}}{M_{\text{shadow}}} - 1\right) \times 100 +\text{dispatch} = \left(\frac{M_d}{M_s}-1\right)\times100\%, \qquad +\text{sampled} = \left(\frac{M_p}{M_s}-1\right)\times100\% $$ -其中 $M$ 为聚合函数（见下表）。当 $M_{\text{shadow}} = 0$ 时结果无定义，SQL 用 `nullif(..., 0)` 处理。 - -### 3.2 相对 baseline（离线 A/B） +采样率为 $r$ 时，摊销后的有效开销是： $$ -\text{vs\_baseline\_pct} = \left(\frac{T_{\text{measured}}}{T_{\text{beline}}} - 1\right) \times 100 +\text{effective}=(1-r)\times\text{dispatch}+r\times\text{sampled} $$ -### 3.3 绝对增量（paired 训练基准） - -$$ -\Delta T = T_{\text{instrumented}} - T_{\text{baseline}}, \quad T_{\text{inst\_med}} = \text{median}(T_{\text{baseline}}) + \text{median}(\Delta T) -$$ - -### 3.4 NCCL 延迟增量 - -$$ -\text{pct\_delta}(ref, x) = 100 \times \frac{x - ref}{ref} -$$ - -正值表示 profiled 模式更慢（延迟更高或吞吐更低）。 - ---- - -## 4. 运行时测量：TorchProbe Shadow Step - -### 4.1 机制 - -启用 `PROBING_TORCH_PROFILING=on`（默认含 `shadow=4:1`）后，每个 optimizer step 在 `python.torch_step_timing` 写一行。 - -**Shadow 判定**（`shadow_step_in_cycle`）： - -```text -cycle_len = shadow_normal + shadow_baseline # 默认 4 + 1 = 5 -is_shadow = (cycle_index % cycle_len) >= shadow_normal -``` - -默认下 index 0–3 为 probed，index 4 为 shadow。 - -### 4.2 计时窗口（`step_duration_sec`） - -使用 `time.perf_counter()` 墙钟，**非** GPU kernel 时间。 - -```mermaid -sequenceDiagram - participant Train as forward/backward/opt - participant Hook as post_step_hook - participant T as torch_step_timing - - Note over Hook: 上一步结束时 _mark_step_wall_start() - Train->>Hook: optimizer.step() 返回 - Hook->>Hook: drain / flush / save（记时之后、下一步起点之前） - Hook->>T: _record_step_timing() - Hook->>Hook: _mark_step_wall_start()（下一步起点） -``` - -**边界定义**： - -- **起点**：上一 optimizer step 的 `post_step_hook` 末尾调用 `_mark_step_wall_start()` -- **终点**：当前 step 的 `post_step_hook` 内 `_record_step_timing()` 调用时刻（**早于** `_drain_deferred()`） - -因此 `step_duration_sec` 覆盖：**整步训练计算 + 本步 hook 收尾**（含 sampled 步当步的 trace flush），**不含**前几步 deferred GPU event 的 `elapsed_time` 回收（回收在记时之后、下一步起点之前执行）。 - -**与 `train.step` span 的区别**：后者来自 `python.trace_event`，度量 span 包裹的计算区间，**不含** hook 派发与落盘；Web UI 同时展示两者，不可直接对比数值。 - -### 4.3 各 step 类型下 hook 行为 - -| 类型 | 模块 hook | 写 torch_trace | timing 行 | -|------|-----------|----------------|-----------| -| discovery（首步） | 注册模块 | 否 | 特殊路径 | -| probed + 非采样 | 短路（无 module 记录） | 否 | `is_shadow=0, sampled=0` | -| probed + 采样 | 全量执行 | 是 | `is_shadow=0, sampled=1` | -| shadow | 立即 return | 否 | `is_shadow=1` | - -**重要**：`hook_tax` 比较的是 **probed 步（含采样与非采样混合）** vs **shadow** 的中位数。`sampled_overhead` 仅取 `sampled=1` 的 probed 子集，反映「重路径」开销。 - ---- - -## 5. 指标目录 - -### 5.1 In-run 指标 - -| 指标 | 分子 $M_{\text{probed}}$ | 分母 $M_{\text{shadow}}$ | 聚合 | 窗口 | 消费方 | -|------|-------------------------|-------------------------|------|------|--------| -| **hook_tax_pct** | 全部 probed `step_duration_sec` | shadow | **median** | 滚动 **80** 步 | `health_overview`（混合，兼容） | -| **dispatch_overhead_pct** | probed 且 `sampled=0` | shadow | **median** | 滚动 **80** 步 | **推荐告警口径** | -| **sampled_overhead_pct** | probed 且 `sampled=1` | shadow | **median** | 滚动 **80** 步 | `health_overview`、Web | -| **Hook tax / Dispatch（Web）** | dispatch（`sampled=0`）优先展示 | shadow | **median** | 最近 **80** 步 | Web 侧栏 / 面板 | -| **Total overhead（Web 摊销）** | dispatch + sampled 按采样率加权 | — | **加权 %** | 最近 80 步 | Web 面板 | -| **Sampled overhead（Web）** | probed 且 `sampled=1` | shadow | **median** | 最近 80 步 | Web 面板 | - -**推荐 SQL（滚动窗口 + 分层指标）**： - -```sql -WITH bounds AS ( - SELECT GREATEST(COALESCE(MAX(local_step), 0) - 80, 1) AS win_start - FROM python.torch_step_timing -) -SELECT - round((median(CASE WHEN is_shadow = 0 AND sampled = 0 THEN step_duration_sec END) - / nullif(median(CASE WHEN is_shadow = 1 THEN step_duration_sec END), 0) - 1) * 100, 2) - AS dispatch_overhead_pct, - round((median(CASE WHEN is_shadow = 0 THEN step_duration_sec END) - / nullif(median(CASE WHEN is_shadow = 1 THEN step_duration_sec END), 0) - 1) * 100, 2) - AS hook_tax_pct, - round((median(CASE WHEN is_shadow = 0 AND sampled = 1 THEN step_duration_sec END) - / nullif(median(CASE WHEN is_shadow = 1 THEN step_duration_sec END), 0) - 1) * 100, 2) - AS sampled_overhead_pct, - sum(CASE WHEN is_shadow = 0 AND sampled = 0 THEN 1 ELSE 0 END) AS dispatch_n, - sum(CASE WHEN is_shadow = 0 THEN 1 ELSE 0 END) AS probed_n, - sum(CASE WHEN is_shadow = 1 THEN 1 ELSE 0 END) AS shadow_n -FROM python.torch_step_timing, bounds -WHERE local_step >= bounds.win_start AND local_step > 1; -``` - -**显示规则**（Web）：$|\text{pct}| < 0.5$ 显示为 `≈0%`；需 `shadow_n ≥ 5` 且 `dispatch_n ≥ 16` 才视为稳定估计（见 §5.2、§11）。 - -### 5.2 样本量与可信度 - -| 条件 | 行为 | -|------|------| -| `shadow_baseline = 0`（`shadow=off`） | 无法计算 overhead % | -| `shadow_n < 5` | UI / 技能提示「样本不足」；百分比仅供参考 | -| `dispatch_n < 16`（默认 rate=0.05 时约需 320 步窗口） | `dispatch_overhead` 方差大，优先看趋势勿读单点 | -| `probed_n = 0` 或 `shadow_n = 0` | 跳过比率断言（soak） | -| $\|hook\_tax - dispatch\_overhead\| > 10\text{pp}$ | 提示检查采样率 / GPU defer（混合指标被采样步拉高） | - -### 5.3 辅助指标 - -| 指标 | 公式 / 来源 | 用途 | -|------|-------------|------| -| `train_step_median_ms` | `median(span_end.time - span_start.time)`，`name='train.step'` | 计算区间参考（非 hook 税） | -| `nccl.profiler_counters` | 最新一行计数器 | 数据完整性（非 overhead %） | - ---- - -## 6. 离线基准：`make bench` - -脚本：`examples/overhead/bench_instrumentation.py`。须在 **probing 注入进程**内运行： - -```bash -PROBING=1 make bench # 完整 -PROBING=1 make bench-quick # --quick -``` - -### 6.1 参数 - -| 参数 | full（默认） | `--quick` | -|------|-------------|-----------| -| span_iters | 300 | 80 | -| probe_steps | 40 | 12 | -| train batches | 30 | 8 | -| warmup（外层轮次） | 2 | 1 | -| runs（外层轮次） | 5 | 3 | - -### 6.2 三层测量 - -#### 层 A — tracing - -对每个场景： - -1. 内层 warmup `warmup` 次 + 计时 `runs` 次 -2. 取 **median(总墙钟秒)** -3. `vs_baseline_pct` 相对 `span (no backend)`（`PROBING_SPAN_BACKENDS` 空） - -| 场景 | 迭代内容 | -|------|----------| -| empty loop | 空循环基线 | -| span (no backend) | `probing.span`，无持久化 | -| span (memtable) | span + memtable 落盘 | -| span + event | span + event 行 | -| train.step span triple | forward / backward / optimizer 三个 span + `probing.step()` | - -**每迭代微秒**： - -$$ -\text{per\_iter\_us} = \frac{\text{median\_sec}}{\text{iterations}} \times 10^6 -$$ - -#### 层 B — torch_probe（合成） - -- 单假模块 `_FakeMod`，`DelayedRecord.save` 打桩 -- **无** 外层 warmup/runs；一次跑 `probe_steps` 步取 timing **median** -- baseline：`hooks only (trace_spans=off, shadow=off)` -- `shadow=4:1` 场景额外输出 `hook_tax_pct`（与 §3.1 同公式，按 cycle index 分桶） +不能改成 `mean(probed)/mean(shadow)`：probed 集合混合了轻重路径，shadow 样本又更少，均值会同时受 +采样比例和长尾 step 支配。历史 `hook_tax` 使用全部 probed step 的中位数，只保留作兼容和保守上界。 -> **局限**：无真实 backward/GPU；数值用于 **回归对比** 与配置敏感度，不能等同于生产大模型 overhead。 +## 4. 什么时候测量可信 -#### 层 C — torch_train（TinyNet） +Web 与诊断 skill 使用最近 80 个 step，使窗口同时覆盖多个 shadow 周期又不被很早的冷启动污染。 +`shadow_n < 5` 或 `dispatch_n < 16` 时只展示正在收集或低置信提示，不用百分比触发稳定告警； +`shadow=off` 时分母不存在，运行中开销百分比本身无定义。绝对值小于 `0.5%` 显示为 `≈0%`，避免把 +计时分辨率和自然抖动包装成精确差异。 -- 设备：CUDA 可用则用 GPU，否则 CPU -- 每轮 **背靠背**：先无 hook baseline，再带 phase/TorchProbe -- 每场景内 2 步 warmup + `batches` 步计时 -- 报告 `paired_delta_ms` 与合成 `inst_med = median(base) + median(Δ)` +噪声处理服从来源分离。采样重路径不混入 dispatch；step 尖峰用滚动中位数抑制；discovery、JIT 和 +缓存预热从稳定窗口排除；deferred drain 在计时之后执行；跨 rank 先分别计算本 rank 的分层指标， +不能把不同负载的 step 直接混成一个总体均值。 -### 6.3 JSON 导出 +`nccl.profiler_counters`、队列满和写入失败描述数据完整性，不是 overhead 百分比。缺少事件时必须先 +排除采集缺口，再解释为“没有额外成本”。 -```bash -PROBING=1 python examples/overhead/bench_instrumentation.py --json-out /tmp/bench.json -``` - ---- - -## 7. NCCL Profiler Overhead（仅离线） - -**无 in-run shadow**。运行时只看 `nccl.profiler_counters` 健康度。 - -### 7.1 E2E AllReduce / AllGather - -脚本：`examples/overhead/nccl_profiler_overhead.py`；编排：`examples/overhead/run_nccl_bench.sh`。 - -| 模式 | 环境 | -|------|------| -| baseline | 无 `NCCL_PROFILER_PLUGIN`、无 `PROBING` | -| profiled | 插件 + `PROBING=2` + `PROBING_NCCL_INFLIGHT_THRESHOLD_SECS=0` | - -**流程**： - -1. `warmup_iters = 20` -2. `bench_iters = 200`（默认 msg 1 MiB） -3. 每次 collective 前后 `cuda.synchronize()` -4. 输出 `latency_us_{mean,p50,p99}`、`throughput_gbs` -5. `--compare` 用 §3.4 算 `latency_*_pct`、`throughput_pct` - -### 7.2 微基准（Rust Criterion） - -`probing/extensions/nccl-profiler/benches/callback_path.rs`：callback 路径、slot pool、`now_ns`——**组件级**，不与 E2E 数值直接对照。 - ---- - -## 8. 其他子系统 - -| 子系统 | 测量方法 | 公式 / 阈值 | -|--------|----------|-------------| -| **Span + memtable** | `tests/regression/profiling/test_span_overhead.py` | `T_on < T_off × 8 + 0.05s`（300 iter） | -| **TorchProbe module spans** | 同上 | `med_on < med_off × 6 + 0.02s`（20 steps） | -| **memtable 写路径** | `probing bench write` | 吞吐 / 可选延迟 reservoir | -| **pprof** | `probing.pprof.sample_freq` | 定性：SIGPROF 频率越高开销越大 | -| **PROBING_SPAN_LOCATION** | 无自动化公式 | 定性：`inspect.stack()` 显著增开销 | - -**隔离栈成本**：`PROBING_SPAN_BACKENDS=none` 或 bench 中 `configure_backends([])`。 - ---- - -## 9. 阈值与门禁（当前仓库） - -| 门禁 | 条件 | 位置 | -|------|------|------| -| 诊断 warning | `dispatch_overhead_pct > 5%`（稳定口径） | `health_overview` | -| 诊断 info（混合偏高） | `hook_tax_pct > 5%` 且 `dispatch_overhead_pct ≤ 5%` | 可选：采样步导致 | -| soak 失败 | `hook_tax_pct > 75%`（默认 `--max-hook-tax-pct`） | `examples/imagenet/soak_assert.py` | -| CI 回归 | span 倍数上界（§8） | `test_span_overhead.py` | - -> **注意**：5%（告警）、75%（soak）、8×（回归）服务于不同目的，**不是**统一 SLO。发布前应结合目标硬件与模型自行标定。 - ---- - -## 10. 已知偏差与解释 - -1. **Shadow 非真空基线**：NCCL lite hook、CPU/GPU 周期性采集、span 等仍在 shadow 步运行；`hook_tax` 仅隔离 **TorchProbe 模块 hook 路径**。 -2. **采样步膨胀**：sampled probed 步的 `step_duration_sec` 可能含 deferred GPU `elapsed_time` drain；用 `sampled_overhead` 看重路径，勿与非采样 probed 混读。 -3. **窗口已对齐**：Web / `health_overview` 默认均为最近 80 步；全历史 SQL 方差更大。 -4. **合成 bench ≠ 生产**：`bench_instrumentation` torch_probe 层无真实模型；torch_train 层仅 TinyNet。 -5. **多 rank**：各 rank 独立写 `local_step`；联邦查询需注意 `rank` 切片。 -6. **discovery 步**：`local_step = 1` 通常应过滤（`WHERE local_step > 1`）。 - ---- - -## 11. 抗噪声与稳定性 +## 5. 离线验证为什么仍然必要 -### 11.1 噪声来源 +运行中测量贴近生产负载，但只能看到 TorchProbe 相对 shadow 的增量。离线 benchmark 因而分三层： +tracing 层隔离 span 栈与持久化，合成 TorchProbe 层验证 hook 和采样状态机，TinyNet 层用背靠背 paired +delta 验证真实 forward/backward/optimizer 组合。三层不是三个产品指标，而是从组件到端到端逐步定位 +回归的证据链。 -| 来源 | 影响 | 缓解 | -|------|------|------| -| **采样步混入** | `sampled=1` 步含 trace flush、GPU defer/sync，墙钟远高于非采样步 | 用 **`dispatch_overhead`**（`sampled=0`）作 hook 税主指标 | -| **训练步本身抖动** | 数据加载、collective、checkpoint 等导致 step 时长尖峰 | **median** 优于 mean；滚动窗口 | -| **窗口过短** | 全表或 <20 步时 median 方差大 | 默认 **80 步**滚动窗口（Web / health 对齐） | -| **shadow 样本稀少** | 默认 4:1 下 80 步仅 ~16 个 shadow 点 | 要求 `shadow_n ≥ 5`；长跑 job 更可信 | -| **冷启动 / discovery** | 首步注册模块、JIT、缓存预热 | 过滤 `local_step > 1`；离线 bench 用 warmup | -| **混合 probed 聚合** | `hook_tax` 把轻/重路径平均 | 分层报告 dispatch / sampled / blended | -| **OS 调度 / 频率缩放** | 墙钟偶发毛刺 | 离线 bench 多轮 median；in-run 看趋势勿盯单步 | -| **Deferred 回收阻塞 hook** | GPU `elapsed_time` + memtable 写在 `post_step_hook` 内 | 记时后再 drain；默认 **`PROBING_TORCH_DEFER_ASYNC=1`** 后台线程 + 有界队列 | +NCCL 采用另一条验证链：相同消息大小、warmup 和同步边界下运行 baseline 与 profiled collective， +比较 latency 和 throughput；Criterion 只衡量 callback、slot pool 和时钟读取等组件成本，不能代替 +collective 端到端结果。 -### 11.2.1 异步 deferred 回收（中期） +仓库中的 5% 诊断 warning、75% soak 上界和组件倍数门禁服务于不同层次。它们不是统一性能 SLO， +也不能替代目标模型与硬件上的发布前标定。 -默认开启（``PROBING_TORCH_DEFER_ASYNC=1``）。主线程在 ``_record_step_timing`` 之后只做非阻塞 ``event.query()`` 分区，将 ready 的 ``DelayedRecord`` 入队；后台 daemon 线程执行 ``elapsed_time`` + ``save()``。队列满时回退到同步 ``save()``。 +## 6. 修改时必须保持的不变量 {#change-invariants} -| 环境变量 | 默认 | 含义 | -|----------|------|------| -| ``PROBING_TORCH_DEFER_ASYNC`` | ``1`` | ``0`` 关闭后台线程（测试 / 调试） | -| ``PROBING_TORCH_DEFER_QUEUE_SIZE`` | ``4096`` | 有界队列深度 | +下面的表是变更安全契约，而不是另一套开销模型。 -进程退出时 ``atexit`` 会 ``flush`` 队列。测试可调用 ``flush_deferred_drain()``。 +| 不变量 | 必须保持 | 守护位置 | +|--------|----------|----------| +| 主百分比 | `median(dispatch) / median(shadow) - 1`，不能换成 `mean(probed)/mean(shadow)` | `web/src/overhead/metrics.rs` | +| 摊销开销 | `(1 − rate) × dispatch + rate × sampled` | `amortized_blends_dispatch_and_sampled_by_rate` | +| hook 顺序 | `_record_step_timing()` → `_drain_deferred()` → advance → `_mark_step_wall_start()` | Python overhead/sampling regression tests | +| 异步回收 | `PROBING_TORCH_DEFER_ASYNC=1` 为默认；有界队列、满时同步回退、退出 flush | `test_deferred_drain_worker.py` | +| 稳定性门控 | `shadow_n ≥ 5` 且 `dispatch_n ≥ 16` 才能解释为稳定百分比 | Web metrics 与 `health_overview` | +| UI 语义 | Typical=dispatch，Effective=采样率加权；`abs(pct)<0.5%` 显示 `≈0%` | Web formatting/copy tests | -### 11.2 稳定性设计原则 - -1. **分层而非混合**：dispatch（轻路径）与 sampled（重路径）分开；`hook_tax` 仅作兼容汇总。 -2. **稳健聚合**：in-run 默认 **median**；摊销 overhead 用 `(1−rate)×dispatch + rate×sampled`，**不用** `mean(probed)/mean(shadow)`（步长抖动 + 小样本 shadow 会严重失真）。 -3. **滚动窗口**：消费方统一 `WINDOW = 80`（约 16 个 shadow 周期 @ 4:1），避免冷启动与远古 outlier 污染。 -4. **成对对比**：离线 torch_train 用 **背靠背 paired Δ**，抵消 run 间漂移。 -5. **周期对齐**：shadow 按固定 cadence 插入，probed/shadow 经历相同 NCCL/GPU 背景采集。 -6. **显式可信度门控**：样本不足时不报警，只提示 Collecting / 仅供参考。 - -### 11.3 指标选用指南 - -| 目的 | 推荐指标 | 原因 | -|------|----------|------| -| 日常告警 / 侧栏 | `dispatch_overhead_pct` | 与 shadow 路径最接近，抗采样噪声 | -| 评估采样配置代价 | `sampled_overhead_pct` | 隔离重路径 | -| 兼容旧面板 / soak | `hook_tax_pct`（混合） | 保守上界，易偏高 | -| 发布前回归 | `make bench` + CI 倍数上界 | 可控环境、可重复 | -| 长期趋势 | 滚动 80 步 + 每 5min 采样一点 | 避免逐步抖动 | - -### 11.4 调参建议（提高稳定性） - -```bash -# 更稳的 shadow 基线（更多 shadow 步，略增 cadence 开销） -PROBING_TORCH_PROFILING=on,shadow=8:2,rate=0.05 - -# 仅看轻路径时 temporarily 提高 rate 可增加 dispatch_n（更重） -# 诊断 SQL 时务必加窗口： -# WHERE local_step >= (SELECT max(local_step) - 80 FROM python.torch_step_timing) -``` - -### 11.5 未来增强（未实现） - -- **Trimmed mean / winsorize**：去掉 probed 时长 top/bottom 5% 后再算比值 -- **按 cadence 周期聚合**：每 5 步一组先算组内 median，再跨组 median -- **EWMA 平滑**：Web 侧栏显示指数滑动平均，降低逐步跳动 -- **置信区间**：bootstrap 或 shadow 周期自助法给出 ± 区间 -- **NCCL in-run shadow**：与 TorchProbe 对称的 collective 基线 - ---- - -## 12. 操作指南 - -### 启用 in-run overhead - -```bash -# 默认：rate=0.05, shadow=4:1 -PROBING=1 PROBING_TORCH_PROFILING=on python train.py - -# 或运行时配置 -# SET probing.torch.profiling = 'on,shadow=4:1,rate=0.05'; -``` - -### 降低开销（优先级） - -1. 降低 `rate` / `layer_rate` -2. 关闭 `trace_spans`、`sync=on`、`backward=on` -3. `shadow=off` 仅当不需要 in-run 估计 -4. 关闭 torch profiling，仅保留基础探针 - -### 快速冒烟 +修改公式、hook 顺序或异步回收默认值后运行： ```bash -PROBING=1 python examples/overhead/torch_probe_overhead_smoke.py # 无 GPU -PROBING=1 make bench-quick +cd web && cargo test overhead +PROBING=0 pytest tests/regression/profiling/test_overhead_invariants.py \ + tests/regression/profiling/test_torch_probe_sampling.py \ + tests/regression/profiling/test_deferred_drain_worker.py -q ``` ---- - -## 13. 实现索引 - -| 组件 | 路径 | -|------|------| -| Shadow 逻辑与计时 | `python/probing/profiling/torch_probe.py` | -| 离线基准 | `examples/overhead/bench_instrumentation.py` | -| Web SQL | `web/src/overhead/sql.rs` | -| Web 格式化 | `web/src/overhead/metrics.rs` | -| 技能 SQL | `skills/health_overview/steps.yaml` | -| soak 断言 | `examples/imagenet/soak_assert.py` | -| NCCL E2E | `examples/overhead/nccl_profiler_overhead.py` | - ---- - ## 相关文档 -- [性能分析实现](profiling.zh.md) — TorchProbe 采样、表结构 -- [Tracing Span](tracing-spans.zh.md) — span 分层与 bench 入口 -- [NCCL Profiler](nccl-profiler.zh.md) — 插件 ABI 与计数器 -- [常见问题 — High Overhead](../guide/troubleshooting.zh.md) +- [性能采集架构](profiling.zh.md) +- [数据层](data-layer.zh.md) +- [NCCL Profiler 架构](nccl-profiler.zh.md) +- [SQL 表参考](../reference/sql-tables.zh.md) diff --git a/docs/src/design/profiling.md b/docs/src/design/profiling.md index 3acd00a3..b5378b90 100644 --- a/docs/src/design/profiling.md +++ b/docs/src/design/profiling.md @@ -1,266 +1,200 @@ -# Profiling Implementation +# Profiling Architecture + +Performance analysis is first a cost-allocation problem. Long-running observation requires low, +predictable overhead and stable coordinates; deep diagnosis needs dense operator, kernel, and +stack events. A single collector cannot optimize for both. Probing therefore separates continuous +observation from short-window drill-down and composes their evidence at query time. -Probing provides profiling capabilities for AI workloads with minimal overhead and SQL-queryable storage. +## System decomposition -## Overview +![Independent collectors write local tables](../assets/architecture/probing-collector-landscape.svg) -The profiling system collects performance data through: +Collection runs inside the process that owns the data because module hooks, communication +callbacks, and runtime stacks are cheapest to observe there. Each collector owns only its state +and local tables; collectors do not call one another on hot paths. Version skew, contention, or a +failure in one collector therefore does not spread into the training path or another collector. + +Independence creates a correlation problem. Probing solves it without a synchronous event bus: +facts carry step, rank, time, and parallel-role coordinates, and the query engine reconstructs +context across tables and ranks. Coordination cost moves from collection time to query time. This +decision is what allows Torch, NCCL, HCCL, stack, and system collectors to evolve independently. + +## Two observation levels for PyTorch + +TorchProbe and Torch Profiler coexist because observation depth and sustainable cost conflict. +TorchProbe remains active through training and retains step, module, optimizer, and memory facts. +It gives up operator detail so its cost can be sampled, measured, and sustained. Torch Profiler +uses Kineto over a known anomalous window to collect CPU op, CUDA kernel, runtime, and memcpy +events. It yields deeper evidence, but is not a continuous telemetry path. -- Hook-based and periodic collectors -- Statistical sampling (long-running telemetry, not episodic trace windows) -- Columnar table storage (memtable / Arrow-backed tables) -- SQL query interface +The paths have incompatible lifetimes, buffers, and failure boundaries, so they are not merged +into one state machine. They meet only through shared step/rank coordinates in SQL. The resulting +diagnostic progression is deliberate: continuous data narrows the search to a rank, step, and +module; a targeted capture then pays for operator and kernel detail. If both run together their +costs add, and TorchProbe shadow measurements must not be interpreted as Kineto overhead. -## Two independent PyTorch collection paths +### Torch Profiler as a bounded capture transaction -Probing provides both **TorchProbe** and **Torch Profiler**. Their names are similar, but they -are not configurations of one collector and do not share sessions, event buffers, or lifecycle. +![On-demand Torch Profiler control, finalization, SQL views, and timeline output](../assets/architecture/probing-torch-profiler-integration.svg) -| Dimension | TorchProbe | Torch Profiler (`torch.profiler` / Kineto) | -|-----------|------------|---------------------------------------------| -| Role | Low-overhead telemetry throughout training | Short-window drill-down after an anomaly is located | -| Primary granularity | Step, `nn.Module`, optimizer, memory deltas | CPU op, CUDA kernel, runtime, memcpy | -| Start | `PROBING_TORCH_PROFILING` / `configure()`; training hooks | Explicit HTTP or REPL capture for N optimizer steps | -| Control | Step/layer sampling, shadow baseline, deferred GPU-event reads | Capture `steps`, then finalize Kineto output | -| Data | mmap memtables: `python.torch_trace`, `python.torch_step_timing` | Bounded in-process session store exposed as `python.profile_capture` and `python.profile_hotspot` | -| Typical overhead | Low and amortized for long-running observation | Higher, especially with CUDA, shapes, stacks, and FLOPs | -| Answers | Which step/rank/module is persistently slow? What is hook overhead? | Which op/kernel is slow in the target window? Where does GPU time go? | +`ProfilerController` permits one capture at a time because Kineto owns process-wide state. +Concurrent sessions would make event ownership, stop, and cleanup ambiguous. The capture advances +at optimizer-step boundaries so windows across ranks can be aligned by training coordinates rather +than by the instant at which a control request arrived. -**Independent** has three operational consequences: +Aggregation and format conversion happen only after the window closes. This keeps +`key_averages()`, raw-event traversal, and JSON generation off the training hot path. Finalization +prefers op/kernel aggregates; when aggregation is unavailable it preserves a bounded raw-event +fallback and records `truncated` explicitly instead of presenting partial data as complete. -1. Enabling TorchProbe does **not** start Kineto; starting Torch Profiler does **not** change - TorchProbe sampling or shadow scheduling. -2. They may observe the same optimizer step, but correlate only through step/capture coordinates - in SQL; neither collector calls the other. -3. If both run together, their overheads add. TorchProbe shadow steps estimate only the - TorchProbe module-hook path, not Torch Profiler/Kineto overhead. +One capture produces two representations because machine analysis and human inspection need +different shapes. `python.profile_capture` and `python.profile_hotspot` are virtual tables over a +bounded session store, suitable for filtering, aggregation, and cross-rank comparison through +`global.python.profile_hotspot`. The complete `traceEvents` structure remains a timeline for the +Web UI. It is not expanded into MEMT rows: copying every event would amplify hot-path writes, while +capture lifetime is fundamentally different from continuous telemetry. + +### TorchProbe as a long-running step state machine -The intended diagnostic ladder is TorchProbe first to locate the anomalous step, rank, and -module, followed by a targeted Torch Profiler window only when op/kernel detail is needed. +![TorchProbe step state and async queues](../assets/architecture/probing-torchprobe-state.svg) + +Optimizer hooks define step boundaries, while module hooks record facts within the current step. +The main thread advances the state machine, evaluates sampling gates, and records bounded events; +CUDA elapsed-time reads and batch preparation move to a deferred queue. Step wall time is fixed +before old events are drained, so drain cost is not charged to the step that just ended. -## TorchProbe Data Collection Architecture +Hook selection follows a minimum-intrusion rule. Forward timing uses module pre/post hooks. +Backward timing avoids module backward hooks, which interact poorly with inplace activations, and +instead registers tensor grad hooks on forward inputs and outputs. The interval from grad-output +ready to grad-input ready approximates the module backward window; when both boundaries cannot be +formed, the system does not claim a precise duration. -```mermaid -graph TB - subgraph "Data Sources" - TORCH[PyTorch module hooks] - PYTHON[Python / native stacks] - SYSTEM[System & GPU metrics] - end +Sampling has two stages because step density and per-step coverage are independent cost controls. +The step gate is a deterministic, evenly spaced function of the step number, so every rank selects +the same steps. Within a sampled step, a deterministic hash of `(step, layer)` selects modules. +The default `rate=0.05`, `layer_rate=1.0` retains full module relationships for a small fraction of +steps. Unsampled steps short-circuit at hook entry but still write step wall time, preserving a +continuous trend. - subgraph "Collection Layer" - SAMPLER[TorchProbe sampler] - PENDING[Per-step pending buffer] - end +Shadow steps are interleaved at `4:1` by default and bypass TorchProbe hooks. This places the +baseline inside the same training run and workload, reducing environment drift from offline A/B +measurement. The consequence is equally important: it measures only the TorchProbe path. Timing +boundaries, statistical semantics, and confidence gates are defined in the +[overhead model](overhead.md). - subgraph "Storage Layer" - TABLES[python.* tables] - QUERY[Query engine] - end +The continuous path publishes two stable contracts: `python.torch_trace` for module facts and +`python.torch_step_timing` for step class and wall time. Column definitions belong in the +[SQL table reference](../reference/sql-tables.md#python-torch_trace); distributed timeline +construction over these local facts is described in +[Distributed Profiler query and visualization](distributed-profiler.md). - TORCH --> SAMPLER - SAMPLER --> PENDING - PENDING --> TABLES - PYTHON --> TABLES - SYSTEM --> TABLES - TABLES --> QUERY -``` +## Megatron coordinate integration -## PyTorch Profiling (TorchProbe) +![Megatron import hooks align parallel roles and iterations with Probing coordinates](../assets/architecture/probing-megatron-integration.svg) -### Design +The Megatron adapter is a coordinate bridge, not another collector. Import hooks watch +`megatron.core.parallel_state` and `megatron.training.training`. Once their APIs are available, +the adapter reads TP/PP/DP/EP/CP ranks into `probing.set_role(...)` and wraps `train_step` on a +best-effort basis to align Megatron's iteration and micro-batch count with `probing.step(...)`. -TorchProbe targets **always-on, module-level training telemetry** after probe injection or -`PROBING_TORCH_PROFILING=on`. It is the long-running path described above, not a lightweight -configuration or frontend for `torch.profiler`. For on-demand Kineto capture exposed as -**virtual SQL tables** (not memtable), see **[Torch Profiler SQL](torch-profiler-sql.md)**. +This boundary keeps version-sensitive Megatron getters in one adapter. TorchProbe, collective, +stack, profiler, and system collectors continue to depend only on the common step/role state and +join through SQL. Missing modules or an incompatible Megatron API degrade without blocking the +training loop. Runtime controls are documented under +[Megatron autostart](../reference/env-vars.md#megatron-autostart). -There is **no warmup schedule API**. Skip cold-start steps in SQL when needed: +## HCCL collection through the MSProf boundary -```sql -SELECT * FROM python.torch_trace WHERE local_step > 10; -``` +![HCCL MSProf shim records local tables and forwards the original calls to CANN](../assets/architecture/probing-hccl-collector.svg) -### Hooks +On Ascend, HCCL already reports profiling events through `libprofapi.so`. Probing places an +ABI-compatible shim at that boundary: it exports the expected MSProf symbols, classifies and +decodes `ReportApi`, `ReportCompactInfo`, and `ReportAdditionalInfo` payloads, appends rows to +`hccl.host_ops`, `hccl.collectives`, `hccl.tasks`, `hccl.mc2_streams`, and +`hccl.context_ids`, then forwards the original arguments and return value to the real CANN +library. -By default, Probing installs: +The real library is resolved from `PROBING_HCCL_PROFAPI_REAL`, a sibling +`libprofapi.so.real`, or the configured Ascend installation. A table-open failure disables only +that table; it does not disable forwarding. Because MSProf structure layouts follow the deployed +CANN version, installation must preserve the matching real library and validate the ABI. The +shim never resolves bare `libprofapi.so`, which would recursively load itself. + +## Tracing and training phases {#tracing-training-phases} + +Tracing owns the coarse training timeline; TorchProbe owns module timing and memory facts. They may +correlate on a step, but must not both own forward/backward/optimizer phase spans. -- Forward pre/post hooks on every `nn.Module` in the model tree -- Optimizer pre/post step hooks +### State ownership and persistence {#span-api} -**Backward timing is off by default.** With `backward=on`, probing times each module's *own* backward as the interval between **grad_output ready** (a hook on the forward output tensor — fires just before the module's backward) and **grad_input ready** (a hook on the forward input tensor — fires just after). Plain tensor `register_hook` callbacks are inplace-safe, unlike module backward hooks, which crash when downstream layers use inplace activations (AlexNet/ResNet ReLU). Modules whose input does not require grad (e.g. the first layer) record ~0 backward since the interval cannot be measured. Enable via `PROBING_TORCH_PROFILING=on,backward=on` only when needed. +![The span stack owns phase state while the Recorder only commits to backends](../assets/architecture/probing-span-persistence.svg) -### Sampling +The span stack is the sole owner of phase state; persistence is only an output. `probing.span` +creates a nested scope, `probing.event` marks the current scope, and `record_span` submits an +already-closed interval. All three converge on `SpanRecorder`, keeping memtable, logger, and OTEL +as sinks that cannot feed state back into phase tracking. -The first complete training step is **discovery**: modules are registered, no rows are written. Sampling begins on subsequent steps. +`probing.span` uses deferred close. A span without events becomes one closed interval on exit; the +first event causes a lazy `span_start`, followed by `span_end` on exit. This reduces write +amplification for quiet scopes. The tradeoff is explicit: an active span without events is not yet +visible to SQL. That is commit semantics, not missing data. + +### Training-phase invariants {#training-phase-semantics} -`rate` sets the **step sampling density**: exactly one step out of every `round(1/rate)` is sampled, evenly spaced and starting at the first probed step. This is *stratified* (not i.i.d.), so a low rate never leaves long gaps — data appears immediately and at a steady cadence (e.g. `rate=0.01` samples every 100th step, `rate=0.05` every 20th). The schedule is derived from the step index (no host RNG, no per-process seed), so every rank samples the *same* steps — distributed traces stay aligned and training stays reproducible. Non-sampled steps short-circuit: module/optimizer hooks and the GPU flush are skipped (wall timing in `torch_step_timing` is still written). +Training has no second global phase variable. `phase` is always the innermost `forward`, +`backward`, or `optimizer` span; an empty training stack means `idle`. `train.step` is the closed +interval for one logical iteration, not a fourth phase. Optimizer exit advances `micro_step`, and +`micro_batches` then maps it to `local_step`, so gradient accumulation does not create false +complete steps. -On a sampled step, each layer is recorded independently with probability `layer_rate` (the optional second field, e.g. `0.1:0.3` = sample every 10th step, and within it hit each layer with 30% chance). Default `layer_rate=1.0` = full snapshot. The per-layer decision is a deterministic hash of *(step, layer)* — it looks random and varies per layer, yet is reproducible and identical across ranks. The offset-`0` anchor (first hook in the step) is always recorded so every sampled step has a time reference. +The resulting invariants are: -Grammar: `rate[:layer_rate]` — `rate` is the step density, `layer_rate` the per-layer hit probability. A leading `random:` / `ordered:` mode token is still accepted for back-compat and treated as `random` (the legacy per-step rotating-module `ordered` mode has been removed). +1. phase state comes from the span stack, not a second global variable; +2. `train.step` starts at the first forward and ends at optimizer exit across gradient accumulation; +3. an optimizer exit writes at most one `train.step`, and only after a forward; +4. manual spans, phase hooks, and TorchProbe never duplicate an already-active phase; +5. with `micro_batches=k`, each k micro steps advance one `local_step`. -Default when enabled (`PROBING_TORCH_PROFILING=on`): **`rate=0.05`, `layer_rate=1.0`** (full snapshot on 5% of steps). Use `1.0` for every step, `0.05:0.1` for 5% of steps sampling 10% of layers. - -**Shadow baseline steps (default `shadow=4:1`):** every 4 probed training steps are followed by 1 step where TorchProbe hooks are fully bypassed (no module-level `python.torch_trace` rows). NCCL, CPU/GPU sampling, and other collectors are unchanged. Each step writes one row to `python.torch_step_timing` (`is_shadow=1` for baseline steps). Disable with `shadow=off`. - -Estimate overhead (formulas and methodology: **[Overhead measurement](overhead.md)**): - -```sql -SELECT - round(median(CASE WHEN is_shadow = 0 THEN step_duration_sec END) - / nullif(median(CASE WHEN is_shadow = 1 THEN step_duration_sec END), 0) - 1, 4) * 100 - AS overhead_pct -FROM python.torch_step_timing -WHERE local_step > 1; -``` - -Hook overhead is reduced by sampling; forward hooks remain registered on all modules (except on shadow steps, where hooks return immediately). - -### NCCL profiler overhead - -TorchProbe shadow steps measure **module-hook** overhead only. The NCCL profiler plugin has no in-run shadow baseline today — it always records collective events when enabled. For NCCL AllReduce overhead vs probing, use the offline benchmark: - -```bash -./examples/overhead/run_nccl_bench.sh -# or: python examples/overhead/torch_probe_overhead_smoke.py # Torch-only smoke (no GPU) -``` - -Monitor runtime health via `nccl.profiler_counters` (`pool_exhausted`, `write_errors`, `rows_written`). The Web UI overhead panel links to the offline NCCL bench when the profiler is active. - -Records are flushed at the end of each optimizer step (after optional GPU `synchronize()`). Pre/post hook pairs produce two rows; **duration is set on the post row** (`post forward`, `post step`, etc.). - -### Collected Data (`python.torch_trace`) - -Full column list: [SQL Tables — torch_trace](../reference/sql-tables.md#python-torch_trace). - -| Field | Type | Description | -|-------|------|-------------| -| `local_step` | int | Local training step (per rank) | -| global_step | int | Global step (`step_snapshot`) | -| rank | int | `torch.distributed` rank | -| world_size | int | World size | -| role | string | Parallel role key, e.g. `dp=2,pp=1,tp=0` | -| seq | int | Hook sequence within step | -| module | string | Module name | -| stage | string | `pre forward`, `post forward`, `pre step`, `post step` (backward not collected by default) | -| allocated | float | GPU memory allocated (MB); CUDA only | -| max_allocated | float | Peak GPU memory (MB) | -| cached | float | GPU memory reserved (MB) | -| max_cached | float | Peak reserved (MB) | -| time_offset | float | Seconds since step anchor | -| duration | float | Stage duration (seconds); meaningful on post rows | - -Use `role` + `global_step` to join with `python.comm_collective` on the same rank. - -### Collective rows (`python.comm_collective`) - -Lite-mode hooks on `torch.distributed` write one row per collective with `duration_ms`, -`bytes`, `op`, and the same step/role coordinates. **Off by default** (including -multi-rank jobs); enable with `PROBING_TORCH_COLLECTIVE_ENABLE=1` or -`SET probing.torch.collective.enable=1`. See [SQL Tables](../reference/sql-tables.md#python-comm_collective) and [SQL Analytics](../guide/sql-analytics.md#python-comm_collective). - -### Enable PyTorch Profiling - -```bash -# Environment variable (synced to probing.torch.profiling) -PROBING_TORCH_PROFILING=on python train.py - -# Full snapshot on 50% of steps -PROBING_TORCH_PROFILING=0.5 python train.py - -# 10% of steps, and within each hit 30% of layers -PROBING_TORCH_PROFILING=0.1:0.3,tracepy=on python train.py -``` - -Programmatic configuration: - -```python -from probing.profiling.torch_probe import configure - -configure("on,rate=0.5,layer_rate=0.3") -``` - -Profiling starts on the first `optimizer.step()` after torch is imported (optimizer post hook). +Phase ownership belongs to `attach_training_phases`: it closes forward/backward/optimizer and +submits `train.step` wall time. TorchProbe detects an existing owner and does not duplicate phase +spans; it publishes only module timing and memory facts. This ownership rule prevents two hook +systems from assigning different meanings to the same training interval. + +`probing.tracing.SPANS_SQL` joins start/end rows into `duration_us` for querying. See +[Core model](../guide/concepts.md) and [Environment variables](../reference/env-vars.md#tracing-spans). ## Python Stack Profiling -### Backtrace Collection - -Feature layout under `probing/extensions/python/src/features/`: - -| Dir | Role | -|-----|------| -| `python/` | PyO3: `bridge` / `bindings` / `tracing` | -| `stacktrace/` | Stack capture, merge, tracers | -| `torch/` | Module profiling (`python.torch_trace`) | -| `flamegraph/` | Shared flamegraph render + distributed folded merge | -| `crash/` | Fatal-signal backtrace | - -`stacktrace/` pipeline: `StackSnapshot` → `ParsedStacks` → `FoldedStacks`. - -| Module | Role | -|--------|------| -| `snapshot` | Capture document + `StackSource` / flags (only signal-writable form) | -| `compact` | Heap-sized sampler bucket payload (used frame lengths only) | -| `fingerprint` | Aggregation key = `tid` + flags + PCs + py keys (no demangle); pprof also filters to main tid | -| `parse` | Snapshot → CallFrames (multi-slot `(tid,seq)` FIFO view cache) | -| `fold` | Parsed/Snapshot → flamegraph / distributed aggregation | -| `metrics` | JSON groups `sampler` (drop / fingerprint / export-fold) vs `view` (parse / cache) | -| `merge` | Python ⊕ native splice + canonicalize | -| `capture` | Thread registry, intern, signal fill (does not own parse/fold) | -| `spy` | CPython ABI / TLS (py-spy-derived) | -| `tracers/vm` | Eval-frame hook (sole Python frame source) | -| `tracers/pprof` | `SIGPROF` sampling (SQL / continuous profiling) | -| `tracers/dynamic` | `SIGUSR2` + command/HTTP on-demand (always via parse) | - -On-demand and CPU sampling share one pipeline: - -- **Python frames:** only from the **vm tracer** (`PYSTACKS`); symbols are interned under the GIL (full path for source view, basename in flamegraph labels); signal handlers copy pointer keys only. -- **Native frames (Linux):** `SIGPROF` / `SIGUSR2` fill a POD via `fill_raw_snapshot` on a per-thread `SA_ONSTACK` alt stack (in-place into the ring / slot). **macOS:** async `ITIMER_PROF` into Apple libc SIMD (`_platform_strlen`) has caused fixed-PC `SIGILL`; `sample_freq` defaults to rate-limited eval-frame cooperative capture of **`PYSTACKS` only** (no mid-hook SyncWalk — that used to paste `_PyInit__core` / vectorcall under every `[py]` frame). `PROBING_PPROF_SIGPROF=1` forces async SIGPROF. Merge drops CPython call-protocol / extension `PyInit_*` noise. Symbolize/merge run off-signal. -- **metrics JSON:** `sampler.*` (ring/publish drop / fingerprint / fold-on-**export**) vs `view.*` (parse / `(tid,seq)` cache)—`dropped_publish` counts snapshots skipped when both publication buffers are busy; do not read a burst of export `parse_calls` as per-sample demangle cost. -- **Fetch paths:** dynamic (command/HTTP) or pprof (SQL / `sample_freq`); both read Python frames recorded by the vm tracer. -- **Reuse:** when sampling is active, the main-thread HTTP/flamegraph path reuses the latest per-thread snapshot when available. -- **Main-thread HTTP path:** prefer the latest mixed snapshot. On macOS, when no native sample exists, briefly suspend the target through Mach, copy its PC/frame-pointer chain, resume it, and only then symbolize; this avoids delivering a signal to the training thread. **Never `SIGUSR2` the main tid while `sample_freq` is active** (Distributed included). Linux defaults to on-demand `SIGUSR2` with an alternate signal stack and bounded frame-pointer walk when sampling is off; `PROBING_STACK_SIGUSR2_MAIN=0` disables it. Cross-thread on-demand still uses `SIGUSR2`. -- **Distributed flamegraph:** with sampling on, export only aggregated sampler buckets per rank (empty buckets → empty graph; no on-demand fallback). - -TorchProbe module hooks are independent. Distributed CPU mixed-mode flamegraphs: `GET /apis/pprofextension/flamegraph/distributed/json` (Web: **Stacks → Distributed**). SPMD torch module flamegraph: `GET /apis/torchextension/flamegraph/distributed/json`. - -## System Metrics - -Host CPU, memory, GPU utilization, and related metrics are collected on configurable intervals via environment variables such as `PROBING_GPU_SAMPLE_MS`. - -**Variable / tensor watch (`probing.inspect.trace`):** logs go to the Python logger by default. Set `PROBING_TRACE_STDOUT=1` to emit updates on **stdout** instead (useful for quick local debugging; avoid in production training logs). - -## Data Storage - -Torch traces and other probe data are stored in **columnar probe tables** (e.g. `python.torch_trace`), queryable through the engine. Retention and federation follow memtable / server configuration—not a fixed-size in-process ring buffer. - -## Query Interface - -```sql --- Skip discovery / warm-up steps -SELECT module, stage, AVG(duration) AS avg_sec -FROM python.torch_trace -WHERE local_step > 1 AND duration > 0 -GROUP BY module, stage -ORDER BY avg_sec DESC; - --- Per-module flamegraph input uses median(duration) on post rows -SELECT module, stage, median(CAST(duration AS DOUBLE)) -FROM python.torch_trace -WHERE module <> 'None' AND stage LIKE 'post %' -GROUP BY module, stage; -``` - -## Performance Overhead - -Overhead depends on model size (all modules carry forward hooks), sampling mode/rate, and optional features (`sync`, `tracepy`, variable watch). Use lower `rate`, disable torch profiling when not needed, and filter early steps in SQL rather than adding a warmup schedule. - -| Scenario | Typical impact | -|----------|----------------| -| Torch profiling off | Baseline probe overhead only | -| `on` (default `0.05`, full snapshot) | Low; ~5% of steps sampled | -| `0.05:0.1` | Very low; 5% of steps, 10% of layers | -| `1.0` | Higher; full snapshot every step | -| `sync=on` | Higher; synchronizes GPU each hook | +The stack path is divided into `StackSnapshot → ParsedStacks → FoldedStacks` because asynchronous +signal context cannot allocate, symbolize, or take complex locks. Capture writes only thread/source +flags, native PCs, and pre-interned Python frame keys into fixed storage. Parse reconstructs symbols +and mixed stacks outside signal context; fold then performs fingerprint aggregation and flamegraph +output. On-demand capture and continuous sampling can share the latter stages without coupling their +trigger mechanisms to interpretation. + +The eval-frame VM tracer is the only source of Python frames. It interns symbols while holding the +GIL, leaving the signal path to copy keys. On Linux, native frames are filled in place by +`SIGPROF`/`SIGUSR2` handlers running on alternate stacks. On macOS, asynchronous SIGPROF can land in +system SIMD routines and cause `SIGILL`, so the default is cooperative, rate-limited Python capture +from eval-frame. When an on-demand native stack is required, Mach briefly suspends the target, +copies the PC/frame-pointer chain, resumes it immediately, and symbolizes later. Platform-specific +behavior is therefore confined to capture; parse, merge, and fold stay common. + +Continuous sampling uses a bounded ring and two publication buffers. Contention drops and counts a +snapshot instead of blocking the training thread. Query and Web paths reuse the latest snapshot +rather than signaling a main thread that is already being sampled. Across ranks, each process folds +duplicate paths before transfer; the coordinator merges equal paths and records rank coverage. This +changes network cost from proportional to raw samples to proportional to distinct call paths, while +still allowing a partial, explicitly incomplete result when some ranks fail. + +TorchProbe module flamegraphs use the same distributed aggregation idea but retain independent +collection state. Module timing and mixed CPU stacks meet only in the query and presentation layers. + +## Boundary with the other layers + +System metrics are periodic; Torch, communication, and stack collectors are event-driven. They +share coordinates but not scheduler threads. Long-running facts enter columnar probe tables, while +retention, hot/cold placement, and cross-rank querying belong to the data and query layers. A +collector does not grow a second storage policy. See the [table reference](../reference/sql-tables.md) +and [SQL analytics guide](../guide/sql-analytics.md) for fields and examples, and the +[overhead model](overhead.md) for measurement semantics and invariants. diff --git a/docs/src/design/profiling.zh.md b/docs/src/design/profiling.zh.md index 7240d864..bcfabb5b 100644 --- a/docs/src/design/profiling.zh.md +++ b/docs/src/design/profiling.zh.md @@ -1,269 +1,180 @@ -# 性能分析实现 +# 性能采集架构 -Probing 为 AI 工作负载提供低开销、可 SQL 查询的性能数据采集能力。 +性能分析首先是一个成本分配问题：长期运行需要低开销和稳定坐标，深入定位则需要算子、内核和 +调用栈等高密度事件。单一采集器无法同时满足这两个目标。Probing 因而不追求“一次采全”，而是把 +持续观测与短窗口深挖拆成独立路径，再在查询阶段组合证据。 -## 概览 +## 整体分解 -性能分析系统通过以下方式收集数据： +![不同采集器独立回调并写入本机表](../assets/architecture/probing-collector-landscape.svg) -- 基于钩子与周期性的采集器 -- 统计采样（长期遥测，而非短时 trace 窗口） -- 列式表存储（memtable / Arrow 表） -- SQL 查询接口 +采集发生在数据所在进程，因为 module hook、通信回调和运行时栈只有本进程能以最低成本取得。 +每个采集器只维护自己的状态并写本机表，彼此不在热路径调用。这样，一个采集器的版本差异、锁竞争 +或失败不会扩散到训练主路径和其他采集器。 -## 两条独立的 PyTorch 性能采集路径 +独立采集带来的问题是数据如何重新关联。Probing 没有再引入一条同步总线，而是让所有事实携带 +step、rank、时间和并行角色坐标。协调成本从采集时移动到查询时：采集路径保持短小，查询引擎负责 +跨表、跨 rank 恢复完整上下文。这是后续 Torch、NCCL、HCCL、堆栈和系统指标能够独立扩展的基础。 -Probing 同时提供 **TorchProbe** 和 **Torch Profiler**。两者名字相近，但不是同一个 -collector 的不同配置，也不共享采集 session、事件缓冲或生命周期。 +## PyTorch 的两级观察路径 -| 维度 | TorchProbe | Torch Profiler（`torch.profiler` / Kineto） | -|------|------------|----------------------------------------------| -| 定位 | 训练期间长期运行的低开销遥测 | 对已定位异常做短窗口深挖 | -| 主要粒度 | step、`nn.Module`、optimizer、显存变化 | CPU op、CUDA kernel、runtime、memcpy | -| 启动方式 | `PROBING_TORCH_PROFILING` / `configure()`，随训练 hook 运行 | HTTP 或 REPL 显式触发若干 optimizer step | -| 控制方式 | step 采样、layer 采样、shadow baseline、延迟 GPU event 读取 | 一次 capture 的 `steps`，结束后 finalize Kineto 结果 | -| 数据出口 | 持续写 mmap memtable：`python.torch_trace`、`python.torch_step_timing` | 有界进程内 session store，经虚拟表 `python.profile_capture`、`python.profile_hotspot` 查询 | -| 典型开销 | 低且可摊销，面向长期观测 | 较高，尤其启用 CUDA、shape、stack、FLOPs 时 | -| 典型问题 | 哪个 step/rank/module 持续变慢？开销是多少？ | 异常窗口里具体是哪个 op/kernel 慢？GPU 时间如何构成？ | +TorchProbe 与 Torch Profiler 并存，不是因为能力重复，而是因为观察尺度与成本不可兼得。 +TorchProbe 常驻训练过程，只保留 step、module、optimizer 和显存变化；它牺牲算子级细节，换取 +可采样、可测量并能长期运行的成本。Torch Profiler 依赖 Kineto，在一个已知异常窗口内采集 CPU op、 +CUDA kernel、runtime 和 memcpy；它提供更深证据，但不适合作为持续遥测。 -“独立”有三个直接含义： +两条路径拥有不同的生命周期、缓冲和失败边界，因此不能合并成一个状态机。它们只通过相同的 +step/rank 坐标在 SQL 中相遇。由此形成明确的诊断递进：持续路径先缩小到异常 rank、step 和 module， +短窗口路径再为该范围支付更高成本。两者同时开启时成本相加，TorchProbe 的 shadow 估计也只解释 +TorchProbe 自身，不能被误读为 Kineto 的成本。 -1. 开启 TorchProbe **不会**启动 Kineto；触发 Torch Profiler 也**不会**修改 TorchProbe - 的采样率或 shadow 调度。 -2. 两者可以观察到同一个 optimizer step，但只通过 step/capture 坐标在 SQL 层关联， - 不在采集器内部互相调用。 -3. 如果同时开启，两条路径的开销会叠加；TorchProbe 的 shadow baseline 只估算 - TorchProbe module-hook 路径，不能代表 Torch Profiler/Kineto 的开销。 +### Torch Profiler：一次有边界的采集事务 -推荐诊断顺序是：先用 TorchProbe 长期发现异常的 step、rank 和 module，再只对目标 -窗口启动 Torch Profiler。仅在需要 op/kernel 细节时支付 Kineto 的较高成本。 +![按需 Torch Profiler 的控制、结束处理、SQL 视图与时间线输出](../assets/architecture/probing-torch-profiler-integration.svg) -## TorchProbe 数据收集架构 +`ProfilerController` 同时只允许一个 capture，因为 Kineto 本身拥有进程级状态；并发 session 不仅 +难以归属事件，也会让停止和清理失去确定边界。capture 在 optimizer step 边界推进，使不同 rank 的 +窗口可以用训练坐标对齐，而不是依赖控制请求到达的瞬间。 -```mermaid -graph TB - subgraph "数据源" - TORCH[PyTorch 模块钩子] - PYTHON[Python / 原生栈] - SYSTEM[系统与 GPU 指标] - end +窗口关闭后才做聚合和格式转换，避免把 `key_averages()`、原始事件遍历和 JSON 生成放进训练热路径。 +结束处理优先产生 op/kernel 聚合；聚合不可用时保留有界原始事件，并显式记录 `truncated`，不把 +不完整结果伪装成完整结果。 - subgraph "收集层" - SAMPLER[TorchProbe 采样器] - PENDING[每步 pending 缓冲] - end +同一份 capture 派生出两种视图，是为了分离“机器分析”和“人工查看”。 +`python.profile_capture` 与 `python.profile_hotspot` 是有界 session store 上的虚拟表，供本机或 +`global.python.profile_hotspot` 做过滤、聚合和跨 rank 比较；完整 `traceEvents` 保持时间线结构， +交给 Web 可视化。原始时间线不展开写入 MEMT，因为逐事件复制会放大写入成本，而 session 生命周期 +也不同于长期遥测数据。 - subgraph "存储层" - TABLES[python.* 表] - QUERY[查询引擎] - end +### TorchProbe：可长期运行的 step 状态机 - TORCH --> SAMPLER - SAMPLER --> PENDING - PENDING --> TABLES - PYTHON --> TABLES - SYSTEM --> TABLES - TABLES --> QUERY -``` +![TorchProbe 的 step 状态机与异步队列](../assets/architecture/probing-torchprobe-state.svg) -## PyTorch 分析（TorchProbe） +optimizer hook 定义 step 边界，module hook 只记录本 step 内的局部事实。主线程推进状态机、执行采样 +判定并写入有界事件；CUDA elapsed 读取和批量整理进入延迟队列。关闭 step 时先固定本步墙钟，再排空 +旧事件，因此排空成本不会被记到刚结束的 step 上。 -### 设计定位 +hook 选择服从“最小侵入”原则。forward 使用 module pre/post hook；backward 不使用容易与 inplace +激活冲突的 module backward hook，而是在前向输入、输出张量上注册 grad hook，以 grad_output ready +到 grad_input ready 近似模块反向区间。无法形成这两个边界的模块不制造虚假的精确时长。 -TorchProbe 面向**注入后长期开启的 module 级训练遥测**（`PROBING_TORCH_PROFILING=on`）。 -它是上一节两条独立路径中的长期遥测路径，不是 `torch.profiler` 的轻量配置或前端。 -按需 Kineto 采集以**虚拟 SQL 表**暴露（不写 memtable）见 -**[Torch Profiler SQL](torch-profiler-sql.zh.md)**。 +采样分成两级，因为 step 密度和单步覆盖面是两个独立的成本旋钮。step gate 按 step 序号做等间距 +确定性采样，所有 rank 因而选择同一批 step；进入采样 step 后，再以 `(step, layer)` 的确定性哈希 +决定 module 命中。默认 `rate=0.05`、`layer_rate=1.0`，即用少量 step 的完整 module 快照保留层间关系。 +未采样 step 在 hook 入口短路，但仍写 step 墙钟，避免长期趋势出现空洞。 -**不提供 warmup schedule API**。跳过冷启动步请在 SQL 中过滤： +shadow step 默认按 `4:1` 交错插入并绕过 TorchProbe hook。这个设计把基线放在同一次训练、同一份 +负载中，减少离线 A/B 的环境漂移；代价是它只能测量 TorchProbe 路径。详细计时边界、统计口径和 +稳定性门槛见[开销模型](overhead.zh.md)。 -```sql -SELECT * FROM python.torch_trace WHERE local_step > 10; -``` +持续路径最终只发布两个稳定契约：`python.torch_trace` 保存 module 级事实， +`python.torch_step_timing` 保存 step 类型和墙钟。字段定义属于 +[SQL 表参考](../reference/sql-tables.zh.md#python-torch_trace)，而分布式时间线如何在这些本机事实之上 +构造，见[分布式 Profiler 查询与可视化](distributed-profiler.zh.md)。 -### 钩子 +## Megatron 坐标集成 -默认安装： +![Megatron import hook 将并行角色和 iteration 对齐到 Probing 公共坐标](../assets/architecture/probing-megatron-integration.svg) -- 模型树上每个 `nn.Module` 的 forward pre/post 钩子 -- Optimizer 的 pre/post step 钩子 +Megatron 适配器只负责坐标转换，不是新的采集器。import hook 观察 +`megatron.core.parallel_state` 与 `megatron.training.training`；相关 API 就绪后，适配器把 +TP/PP/DP/EP/CP rank 写入 `probing.set_role(...)`，并以 best-effort 方式包装 `train_step`， +将 Megatron iteration 与 micro-batch 数对齐到 `probing.step(...)`。 -**默认不启用 backward 计时。** 开启 `backward=on` 时,对每个模块的 backward 计时取「grad_output 就绪(前向输出张量的 grad hook,在该模块反向**开始前**触发)」到「grad_input 就绪(前向输入张量的 grad hook,在反向**结束后**触发)」的区间。使用普通 tensor `register_hook` 回调,`inplace` 安全(不使用 module backward hook,避免 AlexNet/ResNet 等 `inplace` 激活导致 autograd 崩溃)。输入不需要梯度的模块(如首层)因无法测量区间,记为 ~0。生产环境请谨慎开启。 +版本敏感的 Megatron getter 因而集中在一个适配器内。TorchProbe、通信、堆栈、Profiler 与系统 +采集器仍然只依赖公共 step/role 状态，并在 SQL 中关联。模块不存在或 API 不兼容时只降级集成， +不能阻塞训练循环。运行时开关见[环境变量 — Megatron 自动集成](../reference/env-vars.zh.md#megatron-autostart)。 -### 采样策略 +## 通过 MSProf 边界采集 HCCL -第一个完整训练 step 为 **discovery**：只注册模块,不写库。从后续 step 开始采样。 +![HCCL MSProf shim 写入本机表并把原调用转发给 CANN](../assets/architecture/probing-hccl-collector.svg) -`rate` 为 **step 级采样密度**：每 `round(1/rate)` 个 step 采样 1 个,等间距、从首个 probed step 开始(stratified 分层采样,非 i.i.d.,低采样率也不会长时间无数据)。调度仅由 step 序号决定(不使用宿主 RNG、无进程种子),因此各 rank 采样**相同**的 step——分布式 trace 对齐、训练可复现。未采样的 step 短路:跳过 module/optimizer hook 与 GPU flush(仍写 `torch_step_timing` 的墙钟时间)。 +在昇腾环境中，HCCL 已经通过 `libprofapi.so` 上报 profiling 事件。Probing 在该边界放置 ABI +兼容的 shim：导出 HCCL 所需的 MSProf 符号，分类并解码 `ReportApi`、 +`ReportCompactInfo` 和 `ReportAdditionalInfo`，分别追加到 `hccl.host_ops`、 +`hccl.collectives`、`hccl.tasks`、`hccl.mc2_streams` 与 `hccl.context_ids`，随后把原参数和 +返回值转发给真实 CANN 库。 -被采样的 step 内,每个 layer 以概率 `layer_rate`(可选第二段,默认 `1.0` = 全量快照)独立命中,判定为 (step, layer) 的确定性哈希——看起来随机、逐层变化,但可复现且跨 rank 一致。offset `0` 锚点(本 step 第一个钩子)始终记录,保证每个采样 step 都有时间基准。 +真实库按 `PROBING_HCCL_PROFAPI_REAL`、shim 同目录的 `libprofapi.so.real`、Ascend 安装目录 +依次解析。某张表打开失败只停用该表，不影响原调用转发。MSProf 结构体布局跟随部署的 CANN 版本， +因此安装时必须保存匹配版本的真实库并验证 ABI。shim 不按裸名称再次加载 `libprofapi.so`，避免递归 +加载自身。 -文法:`rate[:layer_rate]`。仍接受前缀 `random:`/`ordered:` 以兼容旧配置(一律按 `random` 处理;旧的逐 step 轮转模块的 `ordered` 模式已移除)。 +## Tracing 与训练阶段 {#tracing-training-phases} -启用时的默认值(`PROBING_TORCH_PROFILING=on`)：**`rate=0.05`,`layer_rate=1.0`**(约 5% 的 step 做全量快照)。`1.0` 表示每步都采样,`0.05:0.1` 表示 5% 的 step、每步采 10% 的 layer。 +Tracing 负责粗粒度的训练时间线，TorchProbe 负责 module 级 timing 与显存事实。二者可以在 +同一 step 上关联，但不能重复拥有 forward/backward/optimizer 阶段。 -**Shadow 基线 step（默认 `shadow=4:1`）**：每 4 个正常训练 step 之后插入 1 个完全跳过 TorchProbe hook 的 step（不写 module 级 `python.torch_trace`）。NCCL、CPU/GPU 等其它采集不变。每个 step 写一行 `python.torch_step_timing`（shadow step 的 `is_shadow=1`）。可用 `shadow=off` 关闭。 +### 状态所有权与持久化 {#span-api} -估算开销（公式与测量方法见 **[开销测量](overhead.zh.md)**）： +![Span 栈拥有阶段状态，Recorder 只负责向不同后端提交](../assets/architecture/probing-span-persistence.svg) -```sql -SELECT - round(median(CASE WHEN is_shadow = 0 THEN step_duration_sec END) - / nullif(median(CASE WHEN is_shadow = 1 THEN step_duration_sec END), 0) - 1, 4) * 100 - AS overhead_pct -FROM python.torch_step_timing -WHERE local_step > 1; -``` +Span 栈是阶段状态的唯一来源，持久化只是它的一个出口。`probing.span` 创建嵌套作用域， +`probing.event` 在当前作用域内打点，`record_span` 则直接提交已经闭合的区间。三种入口最终都经过 +`SpanRecorder`，从而让 memtable、logger 和 OTEL 只承担输出职责，不反向影响阶段状态。 -采样降低记录开销；forward 钩子仍挂在全部子模块上（shadow step 上 hook 立即返回，零开销）。 +`probing.span` 默认延迟关闭：没有 event 的 span 退出时只写一个闭区间；出现 event 后才按需写 +`span_start`，退出再写 `span_end`。这个选择减少无事件作用域的写放大，代价是运行中的无 event span +暂时不能被 SQL 看见。这是提交语义，不是数据丢失。 -### NCCL profiler 开销 +`PROBING_SPAN_BACKENDS` 默认为 `memtable`，也可选 `logger`、`otel` 或 `none`。`none` 仍维护 +线程内 span 栈，但跳过 attributes、JSON 和落盘，用于 benchmark 或只需 `phase()` 的场景。 +`PROBING_SPAN_LOCATION=1` 会调用 `inspect.stack()`，不应在生产训练热路径默认开启。 -Shadow step 仅衡量 **TorchProbe 模块 hook** 开销。NCCL profiler 插件目前没有 in-run shadow 基线，启用后会持续记录 collective 事件。NCCL AllReduce 与 probing 的开销对比请用离线 benchmark： +### 训练阶段不变量 {#training-phase-semantics} -```bash -./examples/overhead/run_nccl_bench.sh -# 或：python examples/overhead/torch_probe_overhead_smoke.py # 仅 Torch 冒烟（无需 GPU） -``` +训练阶段没有第二份全局状态。`phase` 始终从 span 栈中最内层的 +`forward`、`backward` 或 `optimizer` 得到；栈中没有训练阶段时就是 `idle`。`train.step` 是一次 +logical iteration 的闭区间，不是第四种 phase。optimizer 退出时才推进 `micro_step`，再根据 +`micro_batches` 折算 `local_step`，因此梯度累积不会制造假的完整 step。 -运行时健康度见 `nccl.profiler_counters`（`pool_exhausted`、`write_errors`、`rows_written`）。Web UI 开销面板在 NCCL 活跃时会提示离线 bench 入口。 +必须保持以下不变量： -记录在每个 optimizer step 结束时批量落盘（可选 GPU `synchronize()`）。pre/post 成对产生两行；**时长在 post 行**（`post forward`、`post step` 等）上有效。 +1. `phase()` 从 span 栈派生，不维护第二份全局阶段状态；一个 batch 之外显示 `idle` 是正常的。 +2. `train.step` 从本 logical iteration 的第一次 forward 开始，到 optimizer hook 退出结束； + 梯度累积中的中间 forward/backward 不重置它。 +3. 每次 optimizer 退出最多写一条 `train.step`，且此前必须观察到 forward。 +4. 同一 phase 同时只允许一个拥有者；手动 span、phase hook 或 TorchProbe 已打开该 phase 时， + 其他 hook 不重复创建。 +5. `micro_batches=k` 时，每 k 次 micro step 才推进一个 `local_step`。 -### 采集字段（`python.torch_trace`） +阶段所有权只交给 `attach_training_phases`：它关闭 forward/backward/optimizer，并提交 +`train.step` 墙钟。TorchProbe 发现已有 owner 时不重复创建阶段，只发布 module timing 与显存事实。 +这一所有权规则避免同一训练区间被两套 hook 重复解释。 -完整列说明：[SQL 表 — torch_trace](../reference/sql-tables.zh.md#python-torch_trace)。 +查询层通过 `probing.tracing.SPANS_SQL` 把 `python.trace_event` 的 start/end 行恢复成闭区间， +采集器不额外维护第二份 duration 表。坐标与表语义见[核心模型](../guide/concepts.zh.md)，backend 环境变量见 +[环境变量](../reference/env-vars.zh.md#tracing-spans)。 -| 字段 | 类型 | 描述 | -|------|------|------| -| step | int | 本地训练步（每 rank） | -| global_step | int | 全局步（`step_snapshot`） | -| rank | int | `torch.distributed` rank | -| world_size | int | world size | -| role | string | 并行角色 key，如 `dp=2,pp=1,tp=0` | -| seq | int | step 内钩子序号 | -| module | string | 模块名 | -| stage | string | `pre forward`、`post forward`、`pre step`、`post step`（默认不采 backward） | -| allocated | float | GPU 已分配内存 (MB)，仅 CUDA | -| max_allocated | float | GPU 峰值内存 (MB) | -| cached | float | GPU 预留内存 (MB) | -| max_cached | float | 峰值预留 (MB) | -| time_offset | float | 相对本 step 锚点的秒数 | -| duration | float | 阶段耗时（秒）；post 行有意义 | - -可用 `role` + `global_step` 与同 rank 的 `python.comm_collective` JOIN。 - -### 集合通信（`python.comm_collective`） - -对 `torch.distributed` 的 lite 模式钩子每条 collective 写一行，含 `duration_ms`、`bytes`、`op` -及相同 step/role 坐标。**默认关闭**（含多 rank 作业）；需显式开启： -`PROBING_TORCH_COLLECTIVE_ENABLE=1` 或 `SET probing.torch.collective.enable=1`。 -见 [SQL 表](../reference/sql-tables.zh.md#python-comm_collective) 与 -[SQL 分析](../guide/sql-analytics.zh.md#python-comm_collective)。 - -### 启用 PyTorch 分析 - -```bash -# 环境变量（同步为 probing.torch.profiling） -PROBING_TORCH_PROFILING=on python train.py - -# 50% step，全量快照 -PROBING_TORCH_PROFILING=0.5 python train.py +## Python 堆栈分析 -# 10% step，每步采 30% 的 layer -PROBING_TORCH_PROFILING=0.1:0.3,tracepy=on python train.py +![各 rank 折叠堆栈后合并为分布式火焰图](../assets/architecture/probing-distributed-stacks.svg) -# 开启 backward 模块计时（默认关闭，可能影响 autograd） -PROBING_TORCH_PROFILING=1.0,backward=on python train.py -``` +跨 rank 时先在数据所在进程生成 folded lines，再由查询入口归一化并合并相同调用路径。 +结果同时保留路径权重、完成/失败节点以及路径覆盖的 rank，避免上传全部原始 stack sample。 -编程配置： +堆栈路径被切成 `StackSnapshot → ParsedStacks → FoldedStacks` 三个阶段，核心原因是异步信号环境 +不能做分配、符号化或复杂锁操作。capture 只把线程、来源标记、native PC 和已 intern 的 Python +frame key 写入固定结构；parse 离开信号上下文后恢复符号与混合栈；fold 再做指纹聚合和火焰图输出。 +按需抓栈与连续采样因此可以共享后两段，而不会把各自的触发机制耦合进数据解释。 -```python -from probing.profiling.torch_probe import configure +Python frame 的唯一来源是 eval-frame VM tracer。符号在持有 GIL 时 intern，信号路径只复制 key。 +native frame 在 Linux 由运行在备用信号栈上的 `SIGPROF`/`SIGUSR2` handler 原地填充。macOS 的异步 +SIGPROF 可能落入系统 SIMD 例程并导致 `SIGILL`，所以默认改用 eval-frame 节流的协作式 Python +采样；需要 native 栈的按需路径通过 Mach 短暂停线程、复制 PC/帧指针后立即恢复，再异步符号化。 +平台差异被限制在 capture 阶段，后续 parse、merge 和 fold 保持一致。 -configure("on,rate=0.5,layer_rate=0.3") -``` +连续采样使用有界 ring 和双发布缓冲；缓冲繁忙时丢弃快照并计数，而不是阻塞训练线程。查询或 Web +请求优先复用最近快照，不再向已被持续采样的主线程追加一次信号。跨 rank 时，各进程先把重复调用路径 +折叠成带权 folded lines，再合并相同路径并附带 rank 覆盖范围。这把网络传输量从“原始样本数”降为 +“不同调用路径数”，同时允许部分 rank 失败时返回可解释的不完整结果。 -在 torch 导入后首次 `optimizer.step()` 时通过 optimizer post hook 启动。 +TorchProbe 的 module 火焰图与 CPU 混合栈使用相同的跨 rank 聚合思想，但采集状态彼此独立：前者 +来源于 module timing，后者来源于 VM/native snapshot，二者只在查询和展示层组合。 -## Python 堆栈分析 +## 与其他层的边界 -`probing/extensions/python/src/features/` 按关注点分组： - -| 目录 | 职责 | -|------|------| -| `python/` | PyO3：`bridge` / `bindings` / `tracing` | -| `stacktrace/` | 堆栈捕获、merge、tracers | -| `torch/` | 模块级 profiling（`python.torch_trace`） | -| `flamegraph/` | 共享火焰图渲染 + 分布式 folded merge | -| `crash/` | 致命信号回溯 | - -`stacktrace/` 数据管线：`StackSnapshot` → `ParsedStacks` → `FoldedStacks`。 - -| 模块 | 职责 | -|------|------| -| `snapshot` | 采集单据 + `StackSource` / flags（信号路径唯一可写） | -| `compact` | 采样桶堆上紧凑 payload（按实际帧长，避免整份 POD） | -| `fingerprint` | 聚合键 = `tid` + flags + PCs + py keys（不 demangle）；pprof 另过滤仅主线程 | -| `parse` | Snapshot → CallFrame（`(tid,seq)` 多槽 FIFO 视图缓存） | -| `fold` | Parsed/Snapshot → 火焰图 / 分布式聚合 | -| `metrics` | JSON 分 `sampler`（drop / fingerprint / export-fold）与 `view`（parse / cache） | -| `merge` | Python ⊕ native splice + canonicalize | -| `capture` | 线程注册、intern、信号 fill（不拥有 parse/fold） | -| `spy` | CPython ABI / TLS（py-spy 衍生） | -| `tracers/vm` | 解释器 eval-frame 钩子（Python 帧唯一来源） | -| `tracers/pprof` | `SIGPROF` 采样（SQL / 连续 profiling） | -| `tracers/dynamic` | `SIGUSR2` + 命令/HTTP 按需采集（一律走 parse） | - -按需栈与 CPU 采样共用同一套基础设施： - -- **Python 帧**：仅来自 **vm tracer**（`PYSTACKS`）；符号在 GIL 下 intern（完整路径供源码查看，火焰图展示 basename），signal 路径只拷贝指针。 -- **C++/Rust 帧**：Linux 上 `SIGPROF` / `SIGUSR2` 共用 `SA_ONSTACK` 安全 handler（每 Python 线程 `sigaltstack`，原地 `fill_raw_snapshot`）。**macOS 默认不用 `ITIMER_PROF`**：异步 SIGPROF 打进 `_platform_strlen` 等 SIMD 例程后会固定 PC `SIGILL`；`sample_freq` 改为 eval-frame 节流协作采样（**只记 `PYSTACKS`，不做 mid-hook SyncWalk**，避免 `_PyInit__core` / vectorcall 把分布式火焰图打散；`PROBING_PPROF_SIGPROF=1` 可强制异步）。merge 丢弃 CPython call-protocol / 扩展 `PyInit_*` 噪声。symbolize / merge 在 signal 外完成。 -- **metrics JSON**：`sampler.*`（ring/publish drop / fingerprint / **导出时** fold）与 `view.*`（parse / `(tid,seq)` cache）分栏；`dropped_publish` 表示两个发布缓冲区都繁忙时跳过的快照数；不要把导出批次的 `parse_calls` 读成「每样本都在 demangle」。 -- **获取路径**：dynamic（命令/HTTP）或 pprof（SQL/`sample_freq`）；二者取 Python 信息时都读 vm tracer 已记录的帧。 -- **复用**：采样开启时，主线程 HTTP/火焰图优先复用该线程最近一次采样快照。 -- **主线程 HTTP 路径**：优先复用最新采样混合快照。macOS 没有 native 样本时，通过 Mach 短暂暂停目标线程并复制 PC/帧指针链，立即恢复后再做符号化，避免向训练线程投递信号。**`sample_freq` 开启时禁止对主线程 `SIGUSR2`**（含 Distributed）。Linux 在采样关闭时默认使用带备用信号栈和有边界帧指针遍历的按需 `SIGUSR2`；可用 `PROBING_STACK_SIGUSR2_MAIN=0` 禁用。跨线程按需仍可走 `SIGUSR2`。 -- **分布式火焰图**：`sample_freq` 开启后只聚合各 rank 采样桶（为空则空图，不回退 on-demand）；跨 rank merge 后每个 frame 携带 `ranks`。 -- **canonicalize**：剥 `_Py_RunMain` / importlib / `platform.py` 等 bootstrap；SIGPROF 仅统计已注册 Python 主线程样本。 - -TorchProbe 模块钩子与上述栈采集相互独立。分布式 CPU 混合栈火焰图见 `GET /apis/pprofextension/flamegraph/distributed/json`（Web：**Stacks → Distributed**）。SPMD torch 模块级火焰图：`GET /apis/torchextension/flamegraph/distributed/json`。 - -## 系统指标 - -通过 `PROBING_GPU_SAMPLE_MS` 等环境变量配置间隔，采集主机 CPU、内存、GPU 利用率等。 - -**变量/张量 watch（`probing.inspect.trace`）：** 默认写入 Python logger；设置 `PROBING_TRACE_STDOUT=1` 可改为 **stdout** 输出（本地调试方便，生产训练日志慎用）。 - -## 数据存储 - -探针数据存入**列式探针表**（如 `python.torch_trace`），由查询引擎访问。保留与联邦策略由 memtable / server 配置决定，而非进程内固定大小环形缓冲区。 - -## 查询示例 - -```sql --- 跳过 discovery / 冷启动 -SELECT module, stage, AVG(duration) AS avg_sec -FROM python.torch_trace -WHERE local_step > 1 AND duration > 0 -GROUP BY module, stage -ORDER BY avg_sec DESC; - --- 火焰图聚合（post 行上的 median duration） -SELECT module, stage, median(CAST(duration AS DOUBLE)) -FROM python.torch_trace -WHERE module <> 'None' AND stage LIKE 'post %' -GROUP BY module, stage; -``` - -## 性能开销 - -开销取决于模型规模（全树 forward 钩子）、采样模式/rate、以及 `sync`、`tracepy`、变量监视等选项。降低 `rate`、关闭 torch profiling、在 SQL 中过滤早期 step，而不是引入 warmup schedule。 - -| 场景 | 典型影响 | -|------|----------| -| 关闭 torch profiling | 仅基础探针开销 | -| `on`（默认 `0.05`，全量快照） | 较低；约 5% step 被采样 | -| `0.05:0.1` | 极低；5% step、每步 10% 的 layer | -| `1.0` | 较高；每步全量快照 | -| `sync=on` | 较高；每个钩子同步 GPU | +系统指标按周期采样，Torch、通信和堆栈按各自事件触发；它们只共享坐标，不共享调度线程。 +所有长期事实进入列式探针表，保留、冷热分层和跨 rank 查询由数据层与查询引擎决定，采集器不自行 +实现第二套存储策略。字段和 SQL 示例见[表参考](../reference/sql-tables.zh.md)与 +[SQL 分析指南](../guide/sql-analytics.zh.md)；生产开销的测量方法和不变量见[开销模型](overhead.zh.md)。 diff --git a/docs/src/design/torch-profiler-sql.md b/docs/src/design/torch-profiler-sql.md deleted file mode 100644 index bcd3791c..00000000 --- a/docs/src/design/torch-profiler-sql.md +++ /dev/null @@ -1,304 +0,0 @@ -# Torch Profiler → SQL (conclusion-driven virtual tables) - -Turn on-demand **`torch.profiler`** capture into **SQL that answers diagnostic questions** — not a -mirror of Kineto events. Virtual table schemas are derived from **conclusions we need**; the -Adaptor compiles timeline data into those conclusion slots. Full Chrome timelines stay on HTTP/UI. - -> **Naming boundary:** Torch Profiler here means short-window `torch.profiler` / Kineto capture, -> controlled by `python/probing/profiling/torch_profiler/`. It is independent from the -> long-running module telemetry in `python/probing/profiling/torch_probe.py`: lifecycle and -> storage are separate, correlation happens only through step/capture coordinates in SQL, and -> overheads add if both run together. See the -> [full comparison](profiling.md#two-independent-pytorch-collection-paths). - -Read with [Profiling](profiling.md), [Federated query engine](federation.md), [NCCL Profiler](nccl-profiler.md). - -中文: [中文版](/zh/design/torch-profiler-sql/) - ---- - -## 1. Design principles - -### 1.1 Needs-first, not profiler-first - -| ❌ Wrong starting point | ✅ Right starting point | -|------------------------|-------------------------| -| Mirror Kineto fields as SQL columns | Define **conclusions** diagnostics require | -| One `traceEvents[]` row per SQL row | Pre-aggregate **time buckets** for those conclusions | -| Single-node timeline viewer | **Cross-rank comparable** facts at the same `local_step` + `global.*` | -| Table names reflect implementation | Table names reflect **analytical questions** | - -**Adaptor role:** at finalize, **compile** Kineto / EventList into conclusion fact rows; SQL never -parses Chrome JSON. - -### 1.2 Fit in the existing diagnostic stack - -```mermaid -flowchart LR - subgraph always_on["Always-on (MEMT)"] - TT[torch_trace → which Module] - CC[comm_collective / nccl.* → comm] - SR[slow_rank skill] - end - subgraph on_demand["On-demand (virtual tables)"] - PH[profile_hotspot → kernels inside Module] - PC[profile_capture → capture coordinates] - end - SR -->|"lock step + rank"| PC - TT -->|"lock module"| PH - CC -->|"align step"| PH -``` - -| Existing conclusion | Source | Profiler SQL adds | -|--------------------|--------|-------------------| -| Which **module** is slow | `python.torch_trace` | Which **kernels/ops** under that module | -| Which **rank** is slow | `slow_rank` / `global.python.comm_collective` | Whether the slow rank has a **different kernel profile** | -| NCCL **culprit/victim** | `nccl.proxy_ops` | Whether **compute** is also abnormal on that rank | -| Overall **step slowdown** | `python.torch_step_timing` | **GPU time composition** for a step (compute/mem/sync/…) | - ---- - -## 2. Diagnostic conclusions catalog (SSOT) - -**Q1–Q8** below are the acceptance criteria for virtual tables, skills, and the Adaptor. - -### Q1 — Where does GPU time go on this step? - -**Conclusion:** Top kernels/ops by time for a given `local_step` or capture. - -**Typical trigger:** drill-down after `module_bottleneck`. - -```sql -SELECT bucket_name, bucket_kind, self_us, pct_of_capture, calls -FROM python.profile_hotspot -WHERE capture_id = @capture_id AND bucket_kind IN ('kernel', 'cpu_op') -ORDER BY self_us DESC -LIMIT 20; -``` - -### Q2 — Time composition: compute / memcpy / runtime / other? - -```sql -SELECT bucket_kind, sum(self_us) AS us, sum(pct_of_capture) AS pct -FROM python.profile_hotspot -WHERE capture_id = @capture_id -GROUP BY bucket_kind -ORDER BY us DESC; -``` - -`bucket_kind` is an Adaptor mapping (§7), not raw Kineto `cat`. - -### Q3 — How much slower is a rank on the **same kernel** vs cluster median? - -**Core distributed conclusion** — turning single-node timeline into cluster analysis. - -```sql -WITH per_rank AS ( - SELECT _rank, bucket_name, sum(self_us) AS us - FROM global.python.profile_hotspot - WHERE local_step = @step AND bucket_kind = 'kernel' - GROUP BY _rank, bucket_name -), -median AS ( - SELECT bucket_name, median(us) AS med_us - FROM per_rank - GROUP BY bucket_name -) -SELECT p._rank, p.bucket_name, p.us, - m.med_us, p.us - m.med_us AS delta_us, - (p.us - m.med_us) / nullif(m.med_us, 0) AS delta_ratio -FROM per_rank p -JOIN median m ON p.bucket_name = m.bucket_name -WHERE p.us > m.med_us * 1.2 -ORDER BY delta_us DESC -LIMIT 30; -``` - -- High `delta_ratio` on few ranks → **straggler** kernel slowness -- All ranks high → **shared** algorithmic slowness - -### Q4 — Does the slow rank have a **different hotspot set**? - -Compare slow rank vs a reference rank on the same step (see §2 Q4 in zh doc for full query). - -### Q5 — Can module hotspots be tied to kernels? - -Join `python.torch_trace` with `profile_capture` / `profile_hotspot` on `local_step`. -v2: `module_hint` column when `with_stack=true`. - -### Q6 — Shared slowness vs single-rank anomaly? - -Cross-rank spread (`max - min`) per `bucket_name` on `global.python.profile_hotspot`. - -### Q7 — Is the capture aligned with the right training step / collectives? - -Query `profile_capture` against latest `torch_trace` step and `comm_collective` row counts. - -### Q8 — Is this capture trustworthy (truncated / failed)? - -```sql -SELECT capture_id, status, truncated, event_count, error -FROM python.profile_capture -WHERE capture_id = @capture_id; -``` - ---- - -## 3. Conclusions → virtual tables (minimal schema) - -Two core virtual tables (+ federated mirrors). **No raw event table** in v1. - -### 3.1 `python.profile_capture` - -One row per profiler window. **Federation anchor.** - -| Column | Serves | -|--------|--------| -| `capture_id` | Primary filter | -| `local_step`, `global_step` | Align with torch_trace / comm / nccl | -| `rank`, `world_size`, `role` | Coordinates | -| `trigger` | Audit (skill / manual) | -| `steps_profiled`, `wall_us` | Window metadata | -| `status`, `truncated`, `event_count`, `error` | Q8 data quality | - -Federated: `global.python.profile_capture` + `_host`, `_addr`, `_rank`, `_role`. - -### 3.2 `python.profile_hotspot` - -**One row = one time bucket aggregated for one capture on one rank.** - -| Column | Serves | -|--------|--------| -| `capture_id`, `local_step`, `global_step`, `rank` | Keys | -| `bucket_kind` | Q2 — `kernel`, `cpu_op`, `cuda_runtime`, `memcpy`, `collective`, `other` | -| `bucket_name` | Q1, Q3–Q6 | -| `self_us` | **Primary sort metric** (exclusive time when available) | -| `wall_us`, `calls` | Secondary | -| `pct_of_capture` | Q1, Q2 | -| `module_hint` | Q5 (v2) | - -Federated: `global.python.profile_hotspot` — primary surface for **Q3–Q6**. - -### 3.3 Intentionally omitted (v1) - -| Omitted | Why | Alternative | -|---------|-----|-------------| -| Raw `traceEvents` table | Millions of rows, no direct conclusion | HTTP timeline | -| Kineto `pid`/`tid` mirror | Not operator-meaningful | `bucket_kind` | -| Precomputed `rank_delta` table | Keep flexibility in SQL/skills | §2 query patterns | - ---- - -## 4. Distributed analysis patterns - -### 4.1 Recommended workflow - -1. `slow_rank` or `module_bottleneck` locks `@step` (and optionally `@slow_rank`) -2. Coordinated `profile/start` at that step (v2: all ranks) -3. Adaptor fills `profile_capture` + `profile_hotspot` on each rank -4. Run Q3–Q6 on `global.python.profile_hotspot` - -### 4.2 Federation contract - -| Rule | Detail | -|------|--------| -| Alignment keys | `local_step` + `capture_id`; same `trigger` / `global_step` for one diagnostic action | -| Missing ranks | Contribute zero rows; existing `PROBING_FANOUT_STRICT` semantics | -| Labels | `_rank`, `_host`, `_role` for Q3–Q6 | -| NCCL join | Same `global_step` on `global.nccl.proxy_ops` (time-window join v2) | - -### 4.3 Planned skills - -| Skill | Prerequisite | Conclusions | -|-------|--------------|-------------| -| `kernel_bottleneck` (new) | `module_bottleneck` | Q1, Q2, Q5 | -| `kernel_straggler` (new) | `slow_rank` | Q3, Q4, Q6 | - ---- - -## 5. Trigger and memory (summary) - -| Item | Design | -|------|--------| -| Trigger | HTTP, REPL, MCP, skill chain | -| Window | Default `steps=1`; `local_step` written to `profile_capture` | -| Store | In-process session; Adaptor outputs **only §3 rows** | -| Limits | Env caps; `truncated=true` retains hotspot aggregates | -| Chrome | UI only | - ---- - -## 6. Implementation layering - -```text -Questions / SQL patterns (§2) - ↓ -Virtual schemas (§3) ← TableProvider / global.* - ↓ -KinetoSqlAdaptor: timeline → profile_capture + profile_hotspot rows - ↓ -ProfilerController + SessionStore + raw torch.profiler -``` - ---- - -## 7. Adaptor mapping (appendix) - -Kineto is **input**, not **output schema**. - -| Source | → `profile_hotspot` | -|--------|---------------------| -| CUDA kernel name | `bucket_kind=kernel`, `bucket_name` | -| ATen op | `bucket_kind=cpu_op` | -| memcpy APIs | `bucket_kind=memcpy` | -| sync APIs | `bucket_kind=cuda_runtime` | -| `nccl*` kernels | `bucket_kind=collective` | -| self cuda/cpu time | `self_us` | -| total cuda/cpu time | `wall_us` | -| count | `calls` | - ---- - -## 8. Control API (draft) - -| Method | Path | -|--------|------| -| `POST` | `/apis/pythonext/pytorch/profile/start` | -| `POST` | `/apis/pythonext/pytorch/profile/stop` | -| `GET` | `/apis/pythonext/pytorch/profile/status` | -| `GET` | `/apis/pythonext/pytorch/timeline` | - ---- - -## 9. Implementation phases (by conclusion priority) - -| Phase | Deliverable | Validates | -|-------|-------------|-----------| -| **P0** | Controller + Adaptor → hotspot rows | Q1, Q2, Q8 local | -| **P1** | Virtual tables + local SQL | Q5, Q7 | -| **P2** | `global.*` + `kernel_bottleneck` skill | Q1–Q2 federated | -| **P3** | `kernel_straggler` + coordinated trigger | Q3, Q4, Q6 | -| **P4** | `module_hint` + nccl time join | Q5 refined | - ---- - -## 10. Risks - -| Risk | Mitigation | -|------|------------| -| Wrong-step conclusions | `profile_capture` step fields; Q7 templates | -| Misaligned federation | v2 coordinated capture; check `ranks_seen` in skills | -| Over-reliance on `self_us` | Document vs `wall_us`; Chrome for drill-down | -| Profiler perturbation | Short windows; skills label as intrusive measurement | - ---- - -## 11. Related code (today) - -| Component | Path | -|-----------|------| -| REPL profiler | `python/probing/repl/torch_magic.py` | -| HTTP timeline | `python/probing/handlers/pythonext.py` | -| Module conclusions | `skills/module_bottleneck/` | -| Rank conclusions | `skills/slow_rank/` | -| Virtual table infra | `probing/core/src/core/data_source.rs` | diff --git a/docs/src/design/torch-profiler-sql.zh.md b/docs/src/design/torch-profiler-sql.zh.md deleted file mode 100644 index 29a01150..00000000 --- a/docs/src/design/torch-profiler-sql.zh.md +++ /dev/null @@ -1,423 +0,0 @@ -# Torch Profiler → SQL（结论驱动的虚拟表） - -将按需 **`torch.profiler`** 采集变成 **可回答诊断问题的 SQL 能力**，而不是 Kineto 事件的镜像。 -虚拟表从 **「我们想得出什么结论」** 反推 schema；Adaptor 负责把 timeline 填进这些结论槽位。 -完整 Chrome timeline 仍走 HTTP/UI。 - -> **名称边界：**本文的 Torch Profiler 指 `torch.profiler` / Kineto 短窗口采集，由 -> `python/probing/profiling/torch_profiler/` 控制。它与 -> `python/probing/profiling/torch_probe.py` 的长期 module 遥测是两条独立路径： -> 启停和存储互不依赖，只通过 step/capture 坐标在 SQL 层关联；同时运行时开销叠加。 -> 完整对照见 [性能分析实现](profiling.zh.md) 开头的两条路径对照表。 - -阅读：[性能分析](profiling.zh.md)、[联邦查询引擎](federation.zh.md)、[NCCL Profiler](nccl-profiler.zh.md)。 - -英文版：[torch-profiler-sql.md](torch-profiler-sql.md) - ---- - -## 1. 设计原则 - -### 1.1 需求驱动，而非 Profiler 驱动 - -| ❌ 错误起点 | ✅ 正确起点 | -|------------|------------| -| Kineto 有哪些字段就建哪些列 | 诊断需要哪些 **结论** | -| `traceEvents[]` 一行进 SQL 一行 | 为结论预聚合 **时间桶（time bucket）** | -| 单节点 timeline 浏览器 | **同一 `local_step` 跨 rank 可比较** 的事实表 + `global.*` | -| 表名反映实现（`profiler_kernel`） | 表名反映分析问题（`profile_hotspot`） | - -**Adaptor 的角色**：在 finalize 时把 Kineto / EventList **编译**成结论事实行；SQL 层不再理解 Chrome JSON。 - -### 1.2 与现有诊断栈的分工 - -```mermaid -flowchart LR - subgraph always_on["常开（MEMT）"] - TT[torch_trace → 哪个 Module] - CC[comm_collective / nccl.* → 通信] - SR[slow_rank skill] - end - subgraph on_demand["按需（虚拟表）"] - PH[profile_hotspot → Module 里什么 kernel / 哪类时间] - PC[profile_capture → 采集坐标] - end - SR -->|"锁定 step + rank"| PC - TT -->|"锁定 module"| PH - CC -->|"对齐 step"| PH -``` - -| 已有结论 | 数据源 | Profiler SQL 补什么 | -|----------|--------|---------------------| -| 哪个 **Module** 慢 | `python.torch_trace` | 该 Module 对应哪些 **kernel/op** | -| 哪个 **rank** 慢 | `slow_rank` / `global.python.comm_collective` | 慢 rank 的 **kernel 画像是否不同** | -| NCCL **culprit/victim** | `nccl.proxy_ops` | 慢 rank 上 **compute 是否也异常**（排除/确认 compute-bound） | -| 训练是否 **整体变慢** | `torch_step_timing` | 某步 **GPU 时间构成**（compute/mem/sync/其他） | - ---- - -## 2. 诊断结论清单（SSOT） - -下列 **Q1–Q8** 是虚拟表必须支撑的结论。实现与 skill 以本表为验收标准。 - -### Q1 — 本步 GPU 时间主要花在哪？ - -**结论**：给定 `local_step`（或最近一次 capture），按 kernel/op **排序的耗时 Top-K**。 - -**典型触发**：`module_bottleneck` 发现热点 module 后的钻取。 - -**本地 SQL 范式**： - -```sql -SELECT bucket_name, bucket_kind, self_us, pct_of_capture, calls -FROM python.profile_hotspot -WHERE capture_id = @capture_id AND bucket_kind IN ('kernel', 'cpu_op') -ORDER BY self_us DESC -LIMIT 20; -``` - ---- - -### Q2 — 时间构成：compute / memcpy / runtime / 其他？ - -**结论**：一步 profile 窗口内的 **类别占比**（用于判断是算力、访存还是调度）。 - -**本地 SQL 范式**： - -```sql -SELECT bucket_kind, sum(self_us) AS us, sum(pct_of_capture) AS pct -FROM python.profile_hotspot -WHERE capture_id = @capture_id -GROUP BY bucket_kind -ORDER BY us DESC; -``` - -`bucket_kind` 由 Adaptor 映射（见 §7），不是 Kineto 原始 `cat` 字段。 - ---- - -### Q3 — 慢 rank 在 **相同 kernel** 上比集群中位 rank 慢多少？ - -**结论**：跨节点 **同 step、同 kernel 名** 的 rank 间差异 — 这是「单节点 timeline → 分布式分析」的核心。 - -**联邦 SQL 范式**： - -```sql -WITH per_rank AS ( - SELECT _rank, bucket_name, sum(self_us) AS us - FROM global.python.profile_hotspot - WHERE local_step = @step AND bucket_kind = 'kernel' - GROUP BY _rank, bucket_name -), -median AS ( - SELECT bucket_name, median(us) AS med_us - FROM per_rank - GROUP BY bucket_name -) -SELECT p._rank, p.bucket_name, p.us, - m.med_us, p.us - m.med_us AS delta_us, - (p.us - m.med_us) / nullif(m.med_us, 0) AS delta_ratio -FROM per_rank p -JOIN median m ON p.bucket_name = m.bucket_name -WHERE p.us > m.med_us * 1.2 -ORDER BY delta_us DESC -LIMIT 30; -``` - -**解读**： - -- `delta_ratio` 大且仅个别 rank → **straggler 型** kernel 慢（数据/卡/调度） -- 所有 rank 都高 → **算法/输入** 共性慢（非 straggler） - ---- - -### Q4 — 慢 rank 的 **热点集合** 是否与其他 rank 不同？ - -**结论**：不仅「同一个 kernel 更慢」，还有「慢 rank 多了哪些 kernel / 少了哪些 kernel」。 - -**联邦 SQL 范式**（慢 rank 来自 `slow_rank` 或人工指定 `@slow_rank`）： - -```sql -SELECT h.bucket_name, h.self_us AS slow_us, - (SELECT sum(self_us) FROM global.python.profile_hotspot g - WHERE g.local_step = @step AND g.bucket_name = h.bucket_name - AND g._rank = @median_rank) AS median_rank_us -FROM global.python.profile_hotspot h -WHERE h.local_step = @step AND h._rank = @slow_rank -ORDER BY slow_us DESC -LIMIT 20; -``` - ---- - -### Q5 — Module 热点能否对应到底层 kernel？ - -**结论**：把 `torch_trace` 的 module 级慢点与 profile 窗口内的 op/kernel **关联**（v1 以 **同一步、时间邻域** 为主；精确栈映射为 v2）。 - -**本地 SQL 范式**： - -```sql -SELECT t.module, t.duration AS module_ms, h.bucket_name, h.self_us / 1e3 AS kernel_ms -FROM python.torch_trace t -JOIN python.profile_capture c ON t.local_step = c.local_step -JOIN python.profile_hotspot h ON h.capture_id = c.capture_id -WHERE t.local_step = @step AND t.stage = 'post forward' -ORDER BY t.duration DESC, h.self_us DESC -LIMIT 50; -``` - -v2：Adaptor 在 `with_stack=true` 时填充 `module_hint` 列，支持等值 JOIN。 - ---- - -### Q6 — 这是 **全 rank 共性慢** 还是 **单 rank 异常**？ - -**结论**：对同一 `bucket_name`，看跨 rank 的 **离散度**（std / max-min）。 - -**联邦 SQL 范式**： - -```sql -SELECT bucket_name, - min(self_us) AS min_us, max(self_us) AS max_us, - max(self_us) - min(self_us) AS spread_us, - count(DISTINCT _rank) AS ranks_seen -FROM global.python.profile_hotspot -WHERE local_step = @step AND bucket_kind = 'kernel' -GROUP BY bucket_name -HAVING spread_us > @threshold -ORDER BY spread_us DESC; -``` - ---- - -### Q7 — Profile 窗口是否与训练 step / collective **对齐**？ - -**结论**：确认 capture 的 `local_step` / `global_step` 与通信、module trace **同一坐标系**，避免「拿错步」下结论。 - -**SQL 范式**： - -```sql -SELECT c.capture_id, c.local_step, c.global_step, c.rank, c.trigger, - (SELECT max(local_step) FROM python.torch_trace) AS latest_torch_step, - (SELECT count(*) FROM python.comm_collective - WHERE global_step = c.global_step) AS coll_rows -FROM python.profile_capture c -ORDER BY c.ended_at_us DESC -LIMIT 5; -``` - ---- - -### Q8 — 这次 profile 是否可信（截断 / 失败）？ - -**结论**：Agent 在出结论前必须检查 **数据质量**。 - -```sql -SELECT capture_id, status, truncated, event_count, error -FROM python.profile_capture -WHERE capture_id = @capture_id; -``` - -`truncated = true` 时 Q1–Q6 仅作 **方向性** 参考，skill 应降级表述。 - ---- - -## 3. 结论 → 虚拟表（最小 schema） - -只为支撑 §2 的查询，定义 **两张核心虚拟表**（+ 联邦镜像）。**不**暴露原始 event 表为 v1 默认面。 - -### 3.1 `python.profile_capture`（采集锚点） - -一次 `torch.profiler` 窗口 = 一行。联邦 JOIN 的 **主键锚点**。 - -| 列 | 类型 | 服务于 | -|----|------|--------| -| `capture_id` | text | Q1–Q8 过滤 | -| `local_step`, `global_step` | int | 与 torch_trace / comm / nccl 对齐（Q5、Q7） | -| `rank`, `world_size`, `role` | | 分布式坐标 | -| `trigger` | text | 审计（skill / manual） | -| `steps_profiled` | int | 窗口长度 | -| `wall_us` | bigint | Q2 分母 | -| `started_at_us`, `ended_at_us` | bigint | 时间 | -| `status` | text | Q8 | -| `truncated` | bool | Q8 | -| `event_count` | int | Q8 | -| `error` | text | Q8 | - -联邦：`global.python.profile_capture` + `_host`, `_addr`, `_rank`, `_role`。 - -### 3.2 `python.profile_hotspot`（结论事实表） - -**一行 = 在一个 capture、一个 rank 上，一个时间桶的聚合结果。** - -| 列 | 类型 | 服务于 | -|----|------|--------| -| `capture_id` | text | 关联 capture | -| `local_step`, `global_step` | int | 无 capture_id 时按 step 查（Q3、Q6） | -| `rank` | int | 本地；联邦用 `_rank` | -| `bucket_kind` | text | Q2 — 见下表 | -| `bucket_name` | text | Q1、Q3、Q4、Q6 | -| `self_us` | bigint | **主排序指标** — 不含子节点重叠（优先 Kineto self） | -| `wall_us` | bigint | 含子树 wall time（可选对比） | -| `calls` | int | 调用次数 | -| `pct_of_capture` | double | Q1、Q2 — `self_us / capture.wall_us` | -| `module_hint` | text | Q5（v2，stack 开启时） | - -**`bucket_kind` 枚举（Adaptor 映射，非 Kineto 原文）**： - -| `bucket_kind` | 含义 | 典型结论 | -|---------------|------|----------| -| `kernel` | CUDA kernel | Q1、Q3、Q6 | -| `cpu_op` | ATen CPU op | Q1 | -| `cuda_runtime` | CUDA API / sync | Q2 — 调度/同步瓶颈 | -| `memcpy` | D2D/H2D 等 | Q2 — 访存 | -| `collective` | 若 Kineto 标出 NCCL kernel | 与 `nccl.*` 交叉验证 | -| `other` | 未分类 | 兜底 | - -联邦：`global.python.profile_hotspot` — **Q3–Q6 的主战场**。 - -### 3.3 故意不做的表（v1） - -| 不做 | 原因 | 替代 | -|------|------|------| -| 原始 `traceEvents` 表 | 百万行、无结论 | HTTP timeline | -| 按 Kineto `pid/tid` 镜像 | 对运维无意义 | `bucket_kind` | -| 预计算 `rank_delta` 表 | 结论应留在 SQL/skill 层灵活组合 | §2 范式查询 | - ---- - -## 4. 分布式分析模式 - -### 4.1 标准工作流（推荐） - -```mermaid -sequenceDiagram - participant S as slow_rank / module_bottleneck - participant T as profile/start（协调 step） - participant R as 各 rank 训练进程 - participant Q as global.* 查询 - - S->>T: 锁定 local_step=@step, slow_rank=@r - T->>R: 在 step=@step 采集 1 步（v2 协调） - R->>R: Adaptor → profile_capture + profile_hotspot - Q->>Q: Q3/Q4/Q6 联邦 SQL - Q->>S: 结论：kernel X 在 rank R 上 delta_ratio=1.8 -``` - -### 4.2 联邦契约 - -| 规则 | 说明 | -|------|------| -| **对齐键** | 优先 `local_step` + `capture_id`；跨 rank 同一「诊断动作」应共享 `trigger` 与 `global_step` | -| **缺失 rank** | 无 capture 的 rank 不参与聚合；`PROBING_FANOUT_STRICT` 行为与现网一致 | -| **列标签** | `_rank`, `_host`, `_role` 用于 Q3–Q6，不写入本地表 | -| **与 NCCL 联立** | 同一 `global_step` 上 JOIN `global.nccl.proxy_ops`（时间窗口 JOIN 为 v2） | - -### 4.3 与 skills 的映射（拟议） - -| Skill | 先决 skill | 支撑的结论 | -|-------|-----------|------------| -| `kernel_bottleneck`（新） | `module_bottleneck` 或人工 | Q1、Q2、Q5 | -| `kernel_straggler`（新） | `slow_rank` | Q3、Q4、Q6 | -| `health_overview`（扩展） | — | Q7、Q8 元数据行 | - ---- - -## 5. 触发与内存（简述） - -结论能力的前提是一次 **短窗口、可对齐** 的采集。 - -| 项 | 设计 | -|----|------| -| 触发 | HTTP `profile/start`、REPL、MCP、skill 链式调用 | -| 窗口 | 默认 `steps=1`，绑定 `local_step` 写入 `profile_capture` | -| 存储 | 进程内 Session；Adaptor **只产出 §3 两张表** 的行缓存 | -| 上限 | `PROBING_TORCH_PROFILER_MAX_SESSIONS` 等；截断时 `truncated=true`，保留 hotspot 聚合 | -| Chrome | 仅 UI；不进 SQL | - -v2：**协调式触发** — 所有 rank 在同一 `local_step` 采集，使 Q3–Q6 语义严格成立。 - ---- - -## 6. 实现分层 - -```text -需求 / SQL 范式（本文 §2） - ↓ -虚拟表 schema（§3）← TableProvider / global.* - ↓ -KinetoSqlAdaptor：timeline → profile_capture + profile_hotspot 行 - ↓ -ProfilerController + SessionStore + torch.profiler 原始产物 -``` - -| 层 | 路径（拟议） | -|----|-------------| -| L2 控制 + 适配 | `python/probing/profiling/torch_profiler/` | -| L1 注册 | `ProbeDataSource` / `TableProvider` | -| L4 skill | `skills/kernel_bottleneck/`, `skills/kernel_straggler/` | - ---- - -## 7. Adaptor 映射（实现附录） - -Kineto 是 **输入格式**，不是 **输出 schema**。 - -| Kineto / profiler 来源 | → `profile_hotspot` | -|------------------------|---------------------| -| CUDA kernel 名 | `bucket_kind=kernel`, `bucket_name=name` | -| ATen op | `bucket_kind=cpu_op` | -| `cudaMemcpy*` 等 | `bucket_kind=memcpy` | -| `cudaDeviceSynchronize` 等 | `bucket_kind=cuda_runtime` | -| `nccl*` kernel 名 | `bucket_kind=collective` | -| `self_cuda_time` / `self_cpu_time` | `self_us` | -| `cuda_time_total` | `wall_us` | -| `count` | `calls` | - -Chrome `traceEvents` 仅在 EventList 不可用时作 fallback 解析，产出 **相同 hotspot 行**。 - ---- - -## 8. 控制面 API（草案） - -| 方法 | 路径 | -|------|------| -| `POST` | `/apis/pythonext/pytorch/profile/start` — body 含 `trigger`, `steps`, `align_step` | -| `POST` | `/apis/pythonext/pytorch/profile/stop` | -| `GET` | `/apis/pythonext/pytorch/profile/status` | -| `GET` | `/apis/pythonext/pytorch/timeline` — Chrome，服务 UI | - ---- - -## 9. 实施阶段（按结论优先级） - -| 阶段 | 交付 | 验收结论 | -|------|------|----------| -| **P0** | Controller + SessionStore + Adaptor 产出 hotspot 行 | Q1、Q2、Q8 本地 SQL | -| **P1** | `profile_capture` / `profile_hotspot` 虚拟表 + 本地 SQL | Q5、Q7 | -| **P2** | `global.*` + skill `kernel_bottleneck` | Q1–Q2 联邦可选 | -| **P3** | skill `kernel_straggler` + 协调 step 触发 | Q3、Q4、Q6 | -| **P4** | `module_hint` + 与 nccl 时间联立 | Q5 精确化 | - ---- - -## 10. 风险 - -| 风险 | 缓解 | -|------|------| -| 结论误导（错 step） | `profile_capture` 强制写 step；Q7 查询模板 | -| 联邦不对齐 | v2 协调触发；skill 检查 `ranks_seen` | -| 过度依赖 self_us | 文档说明与 `wall_us` 差异；Chrome 钻取 | -| Profiler 扰动 | 短窗口、skill 明示「干预性测量」 | - ---- - -## 11. 相关代码（当前） - -| 组件 | 路径 | -|------|------| -| REPL profiler | `python/probing/repl/torch_magic.py` | -| HTTP timeline | `python/probing/handlers/pythonext.py` | -| 模块级结论 | `skills/module_bottleneck/` | -| Rank 级结论 | `skills/slow_rank/` | -| 虚拟表基础设施 | `probing/core/src/core/data_source.rs` | diff --git a/docs/src/design/torchrun-cluster.md b/docs/src/design/torchrun-cluster.md deleted file mode 100644 index 9389d89b..00000000 --- a/docs/src/design/torchrun-cluster.md +++ /dev/null @@ -1,66 +0,0 @@ -# Torchrun hierarchical cluster heartbeat - -Multi-process `torchrun` jobs **auto-register** cluster nodes when probing is injected, powering `probing cluster nodes`, the Web cluster page, and `global.*` federation queries. Implementation lives in L3 `probing/server` (Rust). It does **not** block `init_process_group` and does **not** write torch rendezvous keys. - -## When it starts - -| Condition | Notes | -|-----------|--------| -| `PROBING=1/2` | Probing injected | -| `WORLD_SIZE > 1` | Single-process jobs skip cluster | -| `PROBING_TORCHRUN_CLUSTER≠0` | Default **on** | -| `PROBING_CLUSTER_REPORT≠0` | Default **on** | -| Not elastic supervisor | torchrun parent process skips HTTP bind | - -The Rust ctor (`import probing`) calls `maybe_start_torchrun_cluster()`: bind HTTP, publish master/local0 on TCPStore, start the Tokio heartbeat worker. - -## Hierarchy - -```text -leaf (local_rank>0) ──PUT──► local0 (local_rank=0 on same node) -local0 (not global0) ──PUT──► master (global rank 0) -global rank 0 ──PUT──► local master view -``` - -Discovery keys: `probing/torchrun//master` and `.../node//local0` on the job TCPStore (same endpoint as rendezvous, separate key namespace). - -## Environment variables - -See [Environment variables](../reference/env-vars.md) for the full list. Highlights: - -| Variable | Default | Purpose | -|----------|---------|---------| -| `PROBING_TORCHRUN_CLUSTER` | `1` | Enable torchrun cluster | -| `PROBING_ADVERTISE_ADDR` | `MASTER_ADDR`, then hostname | Peer-reachable address published for the current rank; set explicitly when the rendezvous host is not this node's reachable address | -| `PROBING_NODE_HOST` | OS hostname | Explicit host label reported by heartbeat; affects node identity and UI grouping, not the peer network address | -| `PROBING_CLUSTER_REPORT` | `1` | Periodic heartbeat | -| `PROBING_CLUSTER_REPORT_INTERVAL_SEC` | `10` | Base interval (seconds) | -| `PROBING_CLUSTER_STALE_SEC` | `25` | Mark node `dead` after one TTL; remove after a second TTL | -| `PROBING_CLUSTER_REPORT_MAX_INTERVAL_SEC` | `120` | Backoff cap (clamped below stale) | - -**Stale vs backoff:** effective max interval = `min(configured_max, STALE_SEC - STALE_SEC/4 - 1)`. With default stale=25, max ≈ **18s**. For ~60s stable heartbeats, raise `PROBING_CLUSTER_STALE_SEC` (≥90 recommended). - -## Presets (`PROBING_CLUSTER_PRESET`) - -`examples/cluster/run_multinode.sh` supports: - -| Preset | Use case | -|--------|----------| -| `demo` (default) | Local multinode demo | -| `fast` | Faster convergence visibility (5s interval) | -| `steady` | Long runs, lower CPU (90s stale) | - -```bash -PROBING_CLUSTER_PRESET=fast ./examples/cluster/run_multinode.sh 2 2 -``` - -## Demo - -```bash -./examples/cluster/run_multinode.sh -probing -t rank0-host:18080 cluster nodes -``` - -Cluster heartbeat starts from the **Rust ctor only**; `init_process_group` is not patched. Python `probing.torchrun_cluster.setup_torchrun_cluster()` remains for explicit calls and tests. - -See also [Distributed architecture](distributed.md). diff --git a/docs/src/design/torchrun-cluster.zh.md b/docs/src/design/torchrun-cluster.zh.md deleted file mode 100644 index bf58c0d6..00000000 --- a/docs/src/design/torchrun-cluster.zh.md +++ /dev/null @@ -1,102 +0,0 @@ -# torchrun 分层集群心跳 - -多进程 `torchrun` 作业在 probing 注入后**默认自动**注册集群节点，供 `probing cluster nodes`、Web 集群页与 `global.*` 联邦查询使用。实现位于 L3 `probing/server`（Rust），**不阻塞** `init_process_group`，**不写入** torch rendezvous key。 - -## 启动条件 - -| 条件 | 说明 | -|------|------| -| `PROBING=1/2` | 进程已注入 probing | -| `WORLD_SIZE > 1` | 单进程不启动 | -| `PROBING_TORCHRUN_CLUSTER≠0` | 默认 **开** | -| `PROBING_CLUSTER_REPORT≠0` | 默认 **开** | -| 非 elastic supervisor | torchrun 主进程不绑 HTTP | - -满足条件时，Rust ctor（`import probing`）调用 `maybe_start_torchrun_cluster()`：绑 HTTP、在 TCPStore 发布 master/local0 地址、启动 Tokio 心跳 worker。 - -## 分层拓扑 - -```text -leaf (local_rank>0) ──PUT──► local0 (本机 local_rank=0) -local0 (非 global0) ──PUT──► master (global rank 0) -global rank 0 ──PUT──► 本机 master 视图 -``` - -- **发现**：`probing/torchrun//master` 与 `.../node//local0` 写在 job TCPStore（与 torch rendezvous 同 endpoint，独立 key 前缀）。 -- **合并**：`PUT /apis/nodes` 合并心跳；`GET /apis/nodes` 返回按 rank 排序的快照。 -- **收敛**：未凑齐全员时保持 **base 间隔**（默认 10s）；全员 alive 后指数退避（默认 ×2，有上限）。 - -## 环境变量 - -### 开关 - -| 变量 | 默认 | 说明 | -|------|------|------| -| `PROBING_TORCHRUN_CLUSTER` | `1` | `0` 关闭 torchrun 集群 | -| `PROBING_CLUSTER_REPORT` | `1` | `0` 只绑 HTTP，不跑周期心跳 | -| `PROBING_CLUSTER_REPORT_BACKOFF` | `1` | `0` 关闭稳定后退避 | - -### 心跳与 TTL - -| 变量 | 默认 | 说明 | -|------|------|------| -| `PROBING_CLUSTER_REPORT_INTERVAL_SEC` | `10` | 基础心跳间隔（秒） | -| `PROBING_CLUSTER_REPORT_MAX_INTERVAL_SEC` | `120` | 退避上限（会被 stale 安全钳制） | -| `PROBING_CLUSTER_REPORT_BACKOFF_FACTOR` | `2` | 每次稳定 tick 的乘数 | -| `PROBING_CLUSTER_STALE_SEC` | `25` | 一个 TTL 未心跳则标 `dead`，第二个 TTL 后删除 | -| `PROBING_CLUSTER_DISCOVER_TIMEOUT_SEC` | `2` | 发现 master/local0 单次超时 | -| `PROBING_CLUSTER_REPORT_TIMEOUT_SEC` | `5` | PUT 超时 | - -**stale 与退避**：实际 max 间隔 = `min(配置上限, STALE_SEC - STALE_SEC/4 - 1)`。默认 stale=25 时 max ≈ **18s**。若希望稳定后 60s 心跳，需同时提高 `PROBING_CLUSTER_STALE_SEC`（建议 ≥90）。 - -### 网络 - -| 变量 | 说明 | -|------|------| -| `PROBING_PORT` | 仅 **global rank 0** 绑定；其他 rank 用 `0.0.0.0:0` | -| `PROBING_ADVERTISE_ADDR` | 当前 rank 向 peer 发布的可达地址；默认使用 `MASTER_ADDR`，再回退 hostname；rendezvous 地址不是当前节点可达地址时必须显式设置 | -| `PROBING_NODE_HOST` | heartbeat 中显式上报的 host 标签；仅影响节点身份与 UI 分组，不改变 peer 网络地址 | -| `MASTER_ADDR` / `MASTER_PORT` | TCPStore endpoint（torchrun 已设置） | -| `RDZV_ID` | 多 `torchrun` 并行时必须共享（见 demo 脚本） | - -## 预设（`PROBING_CLUSTER_PRESET`） - -`examples/cluster/run_multinode.sh` 支持预设，便于 demo / 调试 / 长跑： - -| 预设 | 用途 | 主要效果 | -|------|------|----------| -| `demo`（默认） | 本地多机 demo | 默认 env，10s 收敛 + 短 stale | -| `fast` | 快速看清 cluster 收敛 | interval=5s，stale=30s | -| `steady` | 长跑省 CPU | stale=90s，max interval≈67s | - -```bash -PROBING_CLUSTER_PRESET=fast ./examples/cluster/run_multinode.sh 2 2 -PROBING_CLUSTER_PRESET=steady ./examples/cluster/run_multinode.sh 4 8 -``` - -也可手动覆盖任意变量；预设只设置未显式 export 的项。 - -## 查询与 role - -```bash -# 连 global rank 0（PROBING_PORT，默认 18080） -probing -t rank0-host:18080 cluster nodes -``` - -训练脚本中 `probing.set_role(dp=…)` 后调用 `refresh_node_role()`（Python facade）立即补一条心跳。 - -## 演示 - -```bash -./examples/cluster/run_multinode.sh # 2 机 × 2 卡 -./examples/cluster/run_multinode.sh 3 4 60 # 3 机 × 4 卡，sleep 60s -``` - -## 与 Python hook 的关系 - -集群心跳**仅由 Rust ctor 启动**；不再 patch `torch.distributed.init_process_group`。Python 模块 `probing.torchrun_cluster` 保留 `setup_torchrun_cluster()` / `master_info()` 供显式调用与测试。 - -## 相关文档 - -- [分布式架构](distributed.zh.md) — 联邦查询与 cluster API -- [环境变量](../reference/env-vars.md) — 完整 env 列表 diff --git a/docs/src/design/tracing-spans.md b/docs/src/design/tracing-spans.md deleted file mode 100644 index 4aa80ecc..00000000 --- a/docs/src/design/tracing-spans.md +++ /dev/null @@ -1,46 +0,0 @@ -# Span API - -!!! note "Language" - The full design is in **[中文 / Chinese](/zh/design/tracing-spans/)**. - This page summarizes the public surface for English readers. - -Training timelines use the **Span API**; module-level profiling uses **TorchProbe** -(`python.torch_trace`). See [Training Phases](/zh/design/training-phase/) for phase -invariants. - -## Which API to use - -| API | Stack | Persist | Use when | -|-----|-------|---------|----------| -| `attach_training_phases(model, optimizer)` | indirect | yes | **Default** — automatic forward/backward/optimizer + `train.step` | -| `with probing.span(...)` | yes | on exit (deferred) | Custom phases, nesting, `probing.event()` | -| `probing.record_span(name, duration_ns=...)` | no | immediately | Closed intervals with known duration | -| Raw `Span(...)` | yes | **no** | Internal only — use `probing.span` | - -## Disabling persistence - -- `PROBING_SPAN_BACKENDS=none` — stack only, no mmap writes -- `probing.tracing.configure_backends([])` — same until `reset_backends()` - -Unset or empty env still defaults to `memtable`. - -## Deferred close - -`with probing.span` does not write rows on enter. Rows appear on exit, or when the -first `event()` forces a lazy `span_start`. In-flight spans are not visible in SQL -until then. - -## Environment - -| Variable | Default | Notes | -|----------|---------|-------| -| `PROBING_SPAN_BACKENDS` | `memtable` | `memtable`, `logger`, `otel`, `none` | -| `PROBING_SPAN_LOCATION` | off | `inspect.stack()` per span — expensive | - -See [Environment Variables](../reference/env-vars.md#tracing--spans). - -## Related - -- [Training Phases](/zh/design/training-phase/) *(中文)* -- [Profiling](profiling.md) -- [Core Concepts](../guide/concepts.md) diff --git a/docs/src/design/tracing-spans.zh.md b/docs/src/design/tracing-spans.zh.md deleted file mode 100644 index 0c0ada6a..00000000 --- a/docs/src/design/tracing-spans.zh.md +++ /dev/null @@ -1,118 +0,0 @@ -# Span API 设计 - -训练时间线（粗粒度阶段）与 TorchProbe（细粒度模块 profiling）的分工、公共 API 选型与性能特性。Phase 协作细节见 **[训练阶段](training-phase.zh.md)**。 - -## 分层 - -``` -probing.span / event / record_span ← 用户 API - ↓ -_RecordedSpan（Python） ← 持久化编排、deferred close - ↓ -Rust Span 栈（thread-local） ← trace_id / span_id / phase / 时间戳 - ↓ -SpanRecorder → backends ← memtable / logger / otel / none - ↓ -python.trace_event（mmap） ← span_start / span_end / event 行 -``` - -| 层 | 职责 | -|----|------| -| **Rust `Span`** | 线程内嵌套栈、`probing.phase()`、`current_span()` | -| **`probing.span`** | 开栈 + 退出时写 backend（或仅栈，无 backend 时） | -| **`record_span`** | 已知 duration 的闭区间，**不进栈** | -| **`attach_training_phases`** | 自动 forward/backward/optimizer + `train.step` | - -## 何时用哪个 API - -```mermaid -flowchart TD - Q[需要记录什么？] - Q -->|训练阶段边界| A["attach_training_phases(model, opt)"] - Q -->|手动阶段 / 嵌套 / event| B["with probing.span(...)"] - Q -->|已知起止时间的闭区间| C["probing.record_span(name, duration_ns=...)"] - Q -->|仅查询当前阶段| D["probing.phase()"] - A --> E["python.trace_event"] - B --> E - C --> E -``` - -| API | 进栈 | 写盘 | 典型场景 | -|-----|------|------|----------| -| `with probing.span(...)` | 是 | exit 时（deferred） | 自定义阶段、`probing.event()` 打点 | -| `probing.record_span(...)` | 否 | 立即 | `train.step`、collective 闭区间 | -| `probing.event(...)` | — | 首次 event 时 lazy 写 `span_start` | span 内里程碑 | -| 裸 `Span(...)`（Rust） | 是 | **否** | 内部/测试；请用 `probing.span` | -| `attach_training_phases` | 间接 | 是 | **推荐**：零侵入 iteration 时间线 | - -### 与 TorchProbe 的分工 - -| 能力 | phase hook | TorchProbe | -|------|------------|------------| -| iteration phase（forward/backward/optimizer） | **拥有** | `owns_training_phases` 时跳过 | -| `train.step` 墙钟 | **拥有**（`record_span`） | 不写 | -| 模块 timing / 显存 | — | `python.torch_trace` | -| 模块级 trace span | — | `trace_spans=on`（默认关或采样） | - -**推荐组合**：`attach_training_phases` + TorchProbe `on`（`trace_spans` 保持 off）——时间线靠 span，瓶颈靠 `torch_trace`。 - -## 持久化语义（deferred close） - -`with probing.span` 在 `__enter__` **不写盘**；`__exit__` 时： - -1. 若 span 生命周期内有过 `event` → 已 lazy 写 `span_start`，再写 `span_end` -2. 否则 → 单次 `record_closed_span`（batch 写 start+end） - -因此：**进行中的 span 在 SQL 里不可见**，直到关闭（或有 event）。适合降低热路径开销；live span 查询需接受这一语义。 - -`record_span` 始终写 closed 记录，适合 `train.step` 等事后已知 duration 的路径。 - -## 关闭持久化（benchmark / 纯栈） - -| 方式 | 效果 | -|------|------| -| `PROBING_SPAN_BACKENDS=none` | 仅 Rust 栈，零写盘 | -| `probing.tracing.configure_backends([])` | 同上（覆盖 env，直到 `reset_backends()`） | -| 未设置 / 空字符串 env | 仍 fallback 到 `memtable` | - -无 backend 时：`span_attrs`、`json.dumps`、memtable 构建 **全部跳过**，仅保留栈操作（bench 中 no-backend ≈ 栈成本）。 - -## 环境变量 - -| 变量 | 默认 | 说明 | -|------|------|------| -| `PROBING_SPAN_BACKENDS` | `memtable` | `memtable`, `logger`, `otel`, `none`（逗号分隔） | -| `PROBING_SPAN_LOG_LEVEL` | `INFO` | `logger` backend 级别 | -| `PROBING_SPAN_LOCATION` | 关 | `1` 时对每个 span 做 `inspect.stack()`（高开销） | - -完整列表见 [环境变量](../reference/env-vars.md#tracing--spans)。 - -## 性能要点 - -1. **坐标缓存**：同一 `micro_step` 内 `span_attrs` 复用 step + parallel 字段，不重复 `snapshot()`。 -2. **无 backend 快路径**：`persistence_enabled()` 为 false 时跳过 attrs 与 recorder。 -3. **Rust 栈 LIFO**：正常 `with` 嵌套用 O(1) pop；乱序退出仍 fallback 全栈搜索。 -4. **避免** `PROBING_SPAN_LOCATION=1` 于训练热路径；TorchProbe 变量追踪有独立 stack walk。 - -本地基准：`make bench-quick` 或 `python examples/overhead/bench_instrumentation.py --quick`。 - -## 查询 - -物化视图 `SPANS_SQL`（`probing.tracing.SPANS_SQL`）在 `span_start` / `span_end` 上 join 出 `duration_us`。示例： - -```sql -SELECT name, phase, local_step, duration_us -FROM ({SPANS_SQL}) AS spans -WHERE name IN ('forward', 'backward', 'optimizer', 'train.step') -ORDER BY start_time DESC -LIMIT 20; -``` - -（将 `{SPANS_SQL}` 替换为 `probing.tracing.SPANS_SQL` 字符串。） - -## 相关文档 - -- [训练阶段](training-phase.zh.md) — phase 不变量、`train.step`、梯度累积 -- [性能分析](profiling.zh.md) — TorchProbe hook 与落表 -- [核心模型 — trace_event](../guide/concepts.zh.md) — 表语义 -- [环境变量](../reference/env-vars.md) — `PROBING_SPAN_*` diff --git a/docs/src/design/training-phase.md b/docs/src/design/training-phase.md deleted file mode 100644 index faeeda2d..00000000 --- a/docs/src/design/training-phase.md +++ /dev/null @@ -1,13 +0,0 @@ -# Training Phases - -!!! note "Language" - The full design is available in **[中文 / Chinese](/zh/design/training-phase/)** only. - An English translation is planned. - -Training phase transitions (`forward`, `backward`, `optimizer`, collective boundaries) and -how they map to `python.trace_event` spans. - -See also: - -- [Profiling](profiling.md) — hook and sampling pipeline -- [Core Concepts — steps & phases](../guide/concepts.md) diff --git a/docs/src/design/training-phase.zh.md b/docs/src/design/training-phase.zh.md deleted file mode 100644 index 50143d42..00000000 --- a/docs/src/design/training-phase.zh.md +++ /dev/null @@ -1,96 +0,0 @@ -# 训练 Phase 语义（Tracing） - -本文定义 `probing.phase()`、`train.step` 与 hook/span 协作的 **不变量**。实现见 `python/probing/tracing/phases.py`。 - -## 核心对象 - -| 概念 | 含义 | -|------|------| -| **phase** | 训练阶段枚举：`forward` / `backward` / `optimizer`（span 字段） | -| **`probing.phase()`** | 当前 span 栈上**最内层**带 training phase 的 span；无则为 `idle` | -| **`train.step`** | 分析用 span **名称**（不是 phase）；表示一次 logical iteration 的 wall time | -| **`probing.step()`** | 坐标计数器；在 **OPTIMIZER span 退出** 时 +1 `micro_step` | - -## Span 命名（API spec） - -```python -# 规范形式：phase 给定则 name 默认为 phase -with probing.span(phase=probing.FORWARD): - ... - -# 分析用名称（非 training phase） -with probing.span("epoch"): - ... - -# 显式 display name + phase -with probing.span("compute", phase=probing.BACKWARD): - ... -``` - -`resolve_span(name, phase)` 规则： - -1. 仅 `phase` → `(name=phase, phase=phase)` -2. 仅 `name` → `(name, infer(name))` -3. 两者皆有 → `(name, resolve(name, phase))`，至少其一必填 - -## 不变量 - -1. **`phase()` 来自 span 栈**，不是独立全局变量；batch 结束后显示 `idle` 是预期行为。 -2. **`train.step` 起止**：从本 logical iteration 的**第一次 forward**（hook 进入）到 **optimizer hook 退出**；中间梯度累积的 forward/backward **不重置**计时器。 -3. **每个 optimizer 退出**最多写一条 `train.step`（需先出现过 forward）；无 forward 的 optimizer 不写。 -4. **同一 phase 同时只有一个活跃 span**：`phase_hook` 在已有同 phase span（manual / torch_probe）时不重复开 span。 -5. **`micro_step`**：每次 OPTIMIZER span 退出 +1；**`local_step = micro_step // micro_batches`**（设置 `probing.step(micro_batches=k)` 对应梯度累积因子）。 - -## TorchProbe × phase hook（ownership） - -| 能力 | phase hook | TorchProbe | -|------|------------|------------| -| iteration phase span（forward/backward/optimizer） | **拥有** | 当 `owns_training_phases(module=…)` 为真时**跳过** | -| `train.step` closed span | **拥有** | 不写 | -| 模块级 `torch_trace` 表（timing / mem） | — | **拥有** | -| 非 training 模块 span（如 init） | — | **拥有** | - -检测 API：`probing.owns_training_phases(model=…)` / `optimizer=…` / `module=…`。 - -典型组合： - -```python -probing.attach_training_phases(model, optimizer) # iteration phase + train.step -configure("on") # TorchProbe 仅写 torch_trace，不再开 training phase span -``` - -仅 TorchProbe、未 attach phase hook 时：TorchProbe 仍会开 training phase span（legacy 路径）。 - -## 组合规则（source） - -| source | 用途 | -|--------|------| -| `manual` | 用户 `probing.span(..., phase=...)` | -| `phase_hook` | `attach_training_phases` hook | -| `torch_probe` | 模块级 TorchProbe span（training phase 可被 hook 抑制） | - -同 phase 已存在活跃 span 时，hook **不再**开同名 phase span。 - -## 梯度累积示例 - -```python -probing.step(micro_batches=4) -probing.attach_training_phases(model, optimizer) - -for i, batch in enumerate(loader): - loss = model(batch) / 4 - loss.backward() - if (i + 1) % 4 == 0: - optimizer.step() - optimizer.zero_grad() -``` - -- 每个 micro-batch：forward/backward phase span 各一对。 -- 仅第 4、8、… 次 micro-batch 触发 optimizer 与 `train.step`。 -- `train.step` attrs 含 `accum_index`、`micro_step`、`local_step`。 - -## 性能：`inspect.stack()` - -自动 `location` **默认关闭**。仅在 `PROBING_SPAN_LOCATION=1` 或显式 `location=` 时，`span.py` 的 `_caller_location()` 会遍历 `inspect.stack()`。TorchProbe 变量追踪在 `torch_probe.py` 另有独立 stack walk。 - -Span API 分层、backend、`none` 与热路径优化见 **[Span API 设计](tracing-spans.zh.md)**。 diff --git a/docs/src/guide/concepts.md b/docs/src/guide/concepts.md index 513fcf95..e8bdf577 100644 --- a/docs/src/guide/concepts.md +++ b/docs/src/guide/concepts.md @@ -177,7 +177,7 @@ ORDER BY avg_ms DESC; set with `_rank` telling you which row came from where. Nodes register via torchrun (Rust ctor starts cluster heartbeat by default — see -[torchrun cluster heartbeat](../design/torchrun-cluster.md)) or by PUTting to +[distributed membership](../design/distributed.md#cluster-membership)) or by PUTting to `/apis/nodes`. Check current registration with `probing -t cluster nodes`. The `_role` tag uses the value from the **node registry**, which is kept in sync @@ -245,7 +245,7 @@ Probing has many configuration points. The most important ones: | `PROBING_AUTH_TOKEN` | Authentication token for remote mode | | `PROBING_CPU_SAMPLE_MS` | CPU sampling interval in milliseconds (0=off) | | `PROBING_GPU_SAMPLE_MS` | GPU sampling interval in milliseconds | -| `PROBING_SPAN_BACKENDS` | Comma-separated: `memtable`, `logger`, `otel`, `none` (stack only). See [Span API](../design/tracing-spans.md). | +| `PROBING_SPAN_BACKENDS` | Comma-separated: `memtable`, `logger`, `otel`, `none` (stack only). See [Tracing and training phases](../design/profiling.md#span-api). | | `PROBING_LOGLEVEL` | `trace`, `debug`, `info`, `warn`, `error` | The complete reference is at [Environment Variables](../reference/env-vars.md). diff --git a/docs/src/guide/concepts.zh.md b/docs/src/guide/concepts.zh.md index 78a9a7bb..06e223c9 100644 --- a/docs/src/guide/concepts.zh.md +++ b/docs/src/guide/concepts.zh.md @@ -25,15 +25,10 @@ CLI 从不直接与引擎交互。它通过 Unix socket（本地）或 TCP（远 启动时设置 `PROBING=1` 通过 `.pth` 钩子激活进程内服务器——无需 import、无需 修改代码。在 Linux 上，`probing inject` 也可以通过 ptrace 附着到已运行的进程。 -概念上： +CLI 与目标进程之间的边界如下图所示。CLI 只负责选择目标和发送 HTTP 请求；查询、配置与扩展状态 +都由目标进程内的 Server 通过公开接口交给 Engine 或 Extension 处理。 -``` -CLI ──(HTTP over Unix socket/TCP)──▶ probing server（目标进程内） - │ - ├── Engine（DataFusion） - ├── Config - └── Extensions（CPU、GPU、Python、NCCL...） -``` +![CLI 通过 Unix socket 或 TCP 访问目标进程内的 Server、Engine 与 Extensions](../assets/architecture/probing-cli-control-surface.svg) ## 数据表：只追加、持续写入 @@ -135,8 +130,8 @@ probing.clear_role() # 恢复为环境变量默认值 关闭写盘（仅保留栈，用于 benchmark）：`PROBING_SPAN_BACKENDS=none` 或 `probing.tracing.configure_backends([])`。 -完整选型、deferred close 语义与性能说明见 **[Span API 设计](../design/tracing-spans.zh.md)**； -phase 不变量见 **[训练阶段](../design/training-phase.zh.md)**。 +完整选型、deferred close 和 phase 不变量见 +**[性能分析 — Tracing 与训练阶段](../design/profiling.zh.md#tracing-training-phases)**。 ## 联邦查询：跨节点查询 @@ -167,7 +162,7 @@ GROUP BY _role, _rank, op ORDER BY avg_ms DESC; ``` -通过 torchrun 注入（Rust ctor 默认启动集群心跳，见 [torchrun 集群心跳](../design/torchrun-cluster.zh.md)）或 `PUT /apis/nodes` 注册节点。 +通过 torchrun 注入（Rust ctor 默认启动集群心跳，见 [分布式成员](../design/distributed.zh.md#cluster-membership)）或 `PUT /apis/nodes` 注册节点。 用 `probing -t cluster nodes` 验证。详见 [分布式](../design/distributed.zh.md)。 ## 表插件 vs 诊断 skill @@ -192,5 +187,5 @@ culprit/victim 等待分解。它是 Rust 扩展，而非 Python @table。见 [N | 每张表的列定义 | [SQL 表目录](../reference/sql-tables.zh.md) | | 多节点配置和 torchrun | [分布式](../design/distributed.zh.md) | | 编写自定义表或 skill | [扩展机制](../design/extensibility.zh.md) | -| Span 与训练时间线 | [Span API](../design/tracing-spans.zh.md) | +| Span 与训练时间线 | [性能分析 — Tracing 与训练阶段](../design/profiling.zh.md#tracing-training-phases) | | CLI 命令和 Python API | [API 参考](../api-reference.zh.md) | diff --git a/docs/src/guide/debugging.md b/docs/src/guide/debugging.md index 554b32b7..c6c77ffa 100644 --- a/docs/src/guide/debugging.md +++ b/docs/src/guide/debugging.md @@ -227,4 +227,4 @@ if dist.is_initialized(): - [Memory Analysis](memory-analysis.md) - Debug memory issues - [Troubleshooting](troubleshooting.md) - Common problems and solutions -- [Design Architecture](../design/architecture.md) - Understand internals +- [Modularity & Boundaries](../design/modularity.md) - Understand internals and ownership diff --git a/docs/src/guide/debugging.zh.md b/docs/src/guide/debugging.zh.md index 153f29a6..8df5623f 100644 --- a/docs/src/guide/debugging.zh.md +++ b/docs/src/guide/debugging.zh.md @@ -173,4 +173,4 @@ LIMIT 10; - [内存分析](memory-analysis.zh.md) - 调试内存问题 - [常见问题](troubleshooting.zh.md) - 常见问题和解决方案 -- [系统架构](../design/architecture.md) - 了解内部实现 +- [模块化与边界](../design/modularity.zh.md) - 了解内部实现与模块归属 diff --git a/docs/src/guide/web-ui.zh.md b/docs/src/guide/web-ui.zh.md index 0ed93f67..a8998746 100644 --- a/docs/src/guide/web-ui.zh.md +++ b/docs/src/guide/web-ui.zh.md @@ -4,6 +4,14 @@ Next Web UI 用于浏览运行中 Python 进程或分布式任务的诊断证据 指标替用户下结论；每个页面都应说明数据的范围、新鲜度、覆盖率和来源，让用户决定 下一步检查什么。 +## 技术架构 + +![Rust 单页面应用通过公开 HTTP 接口连接 Probing 服务端](../assets/architecture/probing-web-spa-architecture.svg) + +Web 使用 Rust/Dioxus 构建并编译为 WASM，浏览器只在首次访问时加载应用壳，后续通过路由 +切换工作区。它通过公开 HTTP/proto 接口读取查询、采集和 Skill 结果，不链接 +`probing-core`，也不在浏览器中复制查询引擎语义。 + ## 启动与连接 启用 Probing 启动任务，然后在浏览器中打开 HTTP 地址： diff --git a/docs/src/reference/env-vars.md b/docs/src/reference/env-vars.md index cc5d898a..8905a3ff 100644 --- a/docs/src/reference/env-vars.md +++ b/docs/src/reference/env-vars.md @@ -59,7 +59,7 @@ Prefix syntax: `init:SCRIPT+` runs `exec(open(SCRIPT).read())` after activ | Variable | Default | Description | |----------|---------|-------------| -| `PROBING_SPAN_BACKENDS` | `memtable` | Comma-separated span backends. Built-in: `memtable` (`python.trace_event`), `logger` (stderr), `otel` (OpenTelemetry), `none` (stack only, no persistence). `configure_backends([])` also disables until `reset_backends()`. Unknown names fall back to `memtable` only. Custom backends: `probing.span_backends` entry point. See [Span API](../design/tracing-spans.md). | +| `PROBING_SPAN_BACKENDS` | `memtable` | Comma-separated span backends. Built-in: `memtable` (`python.trace_event`), `logger` (stderr), `otel` (OpenTelemetry), `none` (stack only, no persistence). `configure_backends([])` also disables until `reset_backends()`. Unknown names fall back to `memtable` only. Custom backends: `probing.span_backends` entry point. See [Tracing and training phases](../design/profiling.md#span-api). | | `PROBING_SPAN_LOG_LEVEL` | `INFO` | Log level for the `logger` span backend. | | `PROBING_SPAN_LOCATION` | unset | Enable automatic location capture via `inspect.stack()` for every span. Adds overhead; use sparingly. | | `PROBING_TRACE_STDOUT` | unset | When `1`/`true`, `probing.inspect.trace` emits variable/tensor updates to **stdout** instead of the Python logger. | @@ -197,9 +197,9 @@ These variables are read by PyTorch, not Probing, but should be set before | `TORCH_NCCL_ENABLE_TIMING` | `false` | Add CUDA timing events for collectives; may add overhead. | | `TORCH_SYMBOLIZE_MODE` | PyTorch default | C++ stack symbolization mode (`dladdr`, `addr2line`, `fast`). | -## Cluster heartbeat (torchrun) +## Cluster heartbeat (torchrun) {#cluster} -Hierarchical side-channel registration when `WORLD_SIZE > 1`. See [torchrun cluster heartbeat](../design/torchrun-cluster.md). +Hierarchical side-channel registration when `WORLD_SIZE > 1`. See [Distributed membership](../design/distributed.md#cluster-membership). | Variable | Default | Description | |----------|---------|-------------| @@ -212,7 +212,7 @@ Hierarchical side-channel registration when `WORLD_SIZE > 1`. See [torchrun clus | `PROBING_CLUSTER_DISCOVER_TIMEOUT_SEC` | `2` | Timeout per master/local0 discovery attempt. | | `PROBING_CLUSTER_REPORT_TIMEOUT_SEC` | `5` | HTTP PUT timeout for cluster report. | | `PROBING_CLUSTER_PRESET` | — | Used by `examples/cluster/run_multinode.sh`: `demo`, `fast`, or `steady`. | -| `PROBING_CLUSTER_FANOUT_HIERARCHICAL` | `1` | Hierarchical cluster query fan-out (coordinator → local0 → leaves). `0` = flat fan-out to every peer. See [Hierarchical fan-out](../design/hierarchical-fanout.md). | +| `PROBING_CLUSTER_FANOUT_HIERARCHICAL` | `1` | Hierarchical cluster query fan-out (coordinator → local0 → leaves). `0` = flat fan-out to every peer. See [Federation — hierarchical fan-out](../design/federation.md#hierarchical-fan-out). | | `PROBING_REMOTE_QUERY_TIMEOUT_SECS` | `30` | Per-peer timeout for remote federated / cluster queries (seconds). | | `PROBING_FANOUT_CONCURRENCY` | `128` | Max concurrent in-flight remote fan-out HTTP requests per query. | | `PROBING_FANOUT_WORKER_THREADS` | `4` | Worker threads in the isolated async fan-out runtime shared by distributed SQL, extension capture, discovery, and heartbeat requests. | diff --git a/docs/src/reference/env-vars.zh.md b/docs/src/reference/env-vars.zh.md index c327d62d..49c688e8 100644 --- a/docs/src/reference/env-vars.zh.md +++ b/docs/src/reference/env-vars.zh.md @@ -9,9 +9,32 @@ Probing 读取的全部 `PROBING_*` 环境变量参考（按子系统分组） | `PROBING` | `0`, `1`/`followed`, `2`/`nested`, `regex:PATTERN`, `SCRIPT.py` | 未设置（禁用） | 是否启用 probing。`1` 仅当前进程；`2` 当前及子进程；`regex:` 脚本名匹配时启用。 | | `PROBING_ORIGINAL` | （自动设置） | — | 备份原始 `PROBING` 值；由 site_hook 设置，勿手动设置。 | +## 数据存储 {#data-storage} + +| 变量 | 默认 | 说明 | +|------|------|------| +| `PROBING_DATA_DIR` | 平台相关 | MEMT mmap 文件根目录；每个进程使用 PID 子目录。 | +| `PROBING_TABLE_DEFAULT_MB` | `20` | Python `@table` 与未指定容量的 `ExternalTable` 默认 mmap 环形容量。 | +| `PROBING_COLD` | 未设置 | 设为 `on` 启用 MEMT 到 MEMC 的后台整理。 | +| `PROBING_COLD_TARGET_MB` | — | 冷段目标滚动大小。 | +| `PROBING_COLD_MAX_TOTAL_MB` | — | 冷层总字节预算。 | +| `PROBING_COLD_TTL_SECS` | — | 冷段保留时间。 | +| `PROBING_COLD_POLL_MS` | — | Compactor 两轮扫描之间的间隔。 | +| `PROBING_COLD_MAX_AGE_SECS` | — | 打开段达到该年龄后强制封存。 | +| `PROBING_COLD_DIR` | `PROBING_DATA_DIR` 下 | 冷段目录。 | + +## Tracing 与 Span {#tracing-spans} + +| 变量 | 默认 | 说明 | +|------|------|------| +| `PROBING_SPAN_BACKENDS` | `memtable` | 逗号分隔的 backend：`memtable`、`logger`、`otel`、`none`。`none` 仅保留栈，不持久化。详见 [Tracing 与训练阶段](../design/profiling.zh.md#span-api)。 | +| `PROBING_SPAN_LOG_LEVEL` | `INFO` | `logger` backend 的日志级别。 | +| `PROBING_SPAN_LOCATION` | 未设置 | 为每个 span 通过 `inspect.stack()` 采集位置，开销较高。 | +| `PROBING_TRACE_STDOUT` | 未设置 | `1`/`true` 时让 `probing.inspect.trace` 输出到 stdout，而不是 Python logger。 | + ## 集群 {#集群} -`WORLD_SIZE > 1` 时的分层 side-channel 注册。详见 [torchrun 集群心跳](../design/torchrun-cluster.zh.md) 与 [分层 fan-out](../design/hierarchical-fanout.zh.md)。 +`WORLD_SIZE > 1` 时的分层 side-channel 注册。详见 [分布式成员](../design/distributed.zh.md#cluster-membership) 与 [联邦查询 — 分层 fan-out](../design/federation.zh.md#hierarchical-fan-out)。 | 变量 | 默认 | 说明 | |----------|---------|-------------| @@ -41,6 +64,21 @@ Probing 读取的全部 `PROBING_*` 环境变量参考（按子系统分组） | `PROBING_NCCL_POOL_SHARDS` | `8` | 按 comm hash 分片 slot pool（1–64）；总 slot 上限均分到各 shard。 | | `PROBING_NCCL_MIN_MSG_BYTES` | `0` | 低于此消息大小（字节）的事件不记录；`0` = 全记录。 | +## Megatron 自动集成 {#megatron-autostart} + +检测到 Megatron 环境变量或模块后，集成以 best-effort 方式自动启用；除 `PROBING=2` 外， +无需修改训练脚本。 + +| 变量 | 默认 | 说明 | +|------|------|------| +| `PROBING_MEGATRON` | `auto` | `auto` 表示检测到 Megatron 环境或模块时启用；也可用 `on`/`off` 强制开关。 | +| `PROBING_MEGATRON_STEP_SYNC` | `auto` | 包装 `train_step`，将 `probing.step` 与 Megatron iteration 对齐。 | +| `probing.megatron.enable` | — | 通过 `probing.config.set` 覆盖自动集成开关。 | +| `probing.megatron.step_sync` | — | 通过 `probing.config.set` 覆盖 iteration 同步开关。 | + +当 `megatron.core.parallel_state` 和 `megatron.training.training` 加载时，import hook +把并行 rank 写入 `probing.set_role`，并让 `train_step` 产生可供 SQL 关联的统一 step 坐标。 + ## 其余变量 激活、存储、Server、认证、Tracing、采样、NCCL、RDMA、PyTorch、调试等章节与 [英文 env-vars](env-vars.md) 同步；尚未单独翻译。 diff --git a/docs/src/reference/sql-tables.md b/docs/src/reference/sql-tables.md index 973297e9..5490ab87 100644 --- a/docs/src/reference/sql-tables.md +++ b/docs/src/reference/sql-tables.md @@ -242,7 +242,7 @@ Linux kernel ring buffer (dmesg) — OOM killer, GPU Xid, IB errors. **Linux onl --- -## Cluster +## Cluster {#cluster-nodes} ### `cluster.nodes` @@ -299,6 +299,52 @@ Per-proxy-op wait decomposition (culprit vs victim). --- +### `nccl.coll_perf` + +One row per completed collective or P2P operation. `exec_time_ns` is reconstructed from child +events and must be interpreted together with `timing_source`. + +| Column | Description | +|--------|-------------| +| `ts`, `rank`, `comm_hash`, `seq` | Completion time and operation identity | +| `coll_func`, `is_p2p`, `peer` | Collective/P2P kind and peer | +| `count`, `msg_size_bytes`, `dtype` | Message payload | +| `algo`, `proto`, `n_channels`, `n_ranks` | NCCL execution choice and communicator size | +| `exec_time_ns`, `enqueue_time_ns` | Reconstructed execution time and host enqueue time | +| `timing_source` | `kernel_gpu`, `kernel_ch`, `proxy`, or `enqueue` | +| `algobw_gbps` | Algorithm bandwidth based on `exec_time_ns` | +| `pool_events_dropped` | Missing child events due to pool pressure; nonzero weakens timing evidence | + +**Global:** `global.nccl.coll_perf` + +--- + +### `nccl.inflight_ops` + +Periodic watchdog snapshots of operations that started but have not stopped. They cover hangs that +cannot produce a completed row. + +| Column | Description | +|--------|-------------| +| `ts`, `rank`, `comm_hash`, `seq` | Snapshot time and operation identity | +| `coll_func`, `kind` | Operation and `coll` / `p2p` / `proxy_op` kind | +| `channel_id`, `peer`, `is_send` | Proxy direction fields; sentinel values when not applicable | +| `start_ns`, `age_ns` | Start time and age at the snapshot | + +**Global:** `global.nccl.inflight_ops` + +--- + +### `nccl.profiler_counters` + +Evidence-integrity snapshots. `rows_written`, `pool_exhausted`, `write_errors`, `filtered`, live/ +capacity fields, and ring-overwrite counters determine whether an absence of events is trustworthy; +they are not an overhead percentage. + +**Global:** `global.nccl.profiler_counters` + +--- + ### `nccl.net_qp` NCCL NetPlugin IB QP completion timing (optional mask bit 128). diff --git a/docs/src/reference/sql-tables.zh.md b/docs/src/reference/sql-tables.zh.md index 1a325b91..26fa9944 100644 --- a/docs/src/reference/sql-tables.zh.md +++ b/docs/src/reference/sql-tables.zh.md @@ -230,7 +230,7 @@ Linux 内核环缓冲（dmesg）。**仅 Linux。** --- -## 集群 +## 集群 {#cluster-nodes} ### `cluster.nodes` @@ -274,6 +274,44 @@ Proxy-op 等待分解（culprit / victim）。 **Global：** `global.nccl.proxy_ops` **联邦列：** `_host`、`_addr`、`_rank`、`_role` +### `nccl.coll_perf` + +每个已完成 collective/P2P 一行。`exec_time_ns` 由子事件窗口重建，必须结合 +`timing_source` 判断计时证据质量。 + +| 列 | 说明 | +|----|------| +| `ts`, `rank`, `comm_hash`, `seq` | 完成时间与操作身份 | +| `coll_func`, `is_p2p`, `peer` | collective/P2P 类型与对端 | +| `count`, `msg_size_bytes`, `dtype` | 消息负载 | +| `algo`, `proto`, `n_channels`, `n_ranks` | NCCL 执行选择与通信组规模 | +| `exec_time_ns`, `enqueue_time_ns` | 重建执行时间与 host enqueue 时间 | +| `timing_source` | `kernel_gpu`、`kernel_ch`、`proxy` 或 `enqueue` | +| `algobw_gbps` | 基于 `exec_time_ns` 的算法带宽 | +| `pool_events_dropped` | 子事件池耗尽造成的缺口；非零时谨慎解释执行时间 | + +**Global：** `global.nccl.coll_perf` + +### `nccl.inflight_ops` + +watchdog 对“已经 start、尚未 stop”的操作生成周期快照，用于补足挂死事件没有完成行的盲区。 + +| 列 | 说明 | +|----|------| +| `ts`, `rank`, `comm_hash`, `seq` | 快照时间与操作身份 | +| `coll_func`, `kind` | 操作名以及 `coll` / `p2p` / `proxy_op` 类型 | +| `channel_id`, `peer`, `is_send` | proxy 方向信息；不适用时使用哨兵值 | +| `start_ns`, `age_ns` | 开始时间与快照时的持续时间 | + +**Global：** `global.nccl.inflight_ops` + +### `nccl.profiler_counters` + +采集完整性快照。`rows_written`、`pool_exhausted`、`write_errors`、`filtered`、各类 live/capacity +以及 ring 覆写计数用于判断“没有事件”是否可信，不属于性能开销百分比。 + +**Global：** `global.nccl.profiler_counters` + ### `nccl.net_qp` NetPlugin IB QP 完成耗时（可选）。 diff --git a/examples/cluster/README.md b/examples/cluster/README.md index 8d6a7ebb..9f3b470d 100644 --- a/examples/cluster/README.md +++ b/examples/cluster/README.md @@ -10,7 +10,7 @@ PROBING_CLUSTER_PRESET=fast ./examples/cluster/run_multinode.sh 2 2 ``` -预设说明：`docs/src/design/torchrun-cluster.md`。 +预设说明：`docs/src/design/distributed.zh.md#cluster-membership`。 ## 文件 diff --git a/examples/cluster/run_multinode.sh b/examples/cluster/run_multinode.sh index 3335b7fe..708eca74 100755 --- a/examples/cluster/run_multinode.sh +++ b/examples/cluster/run_multinode.sh @@ -7,7 +7,7 @@ # # 也可用环境变量：NNODES NPROC SLEEP_SEC MASTER_PORT # probing 高级开关（一般不用改）：PROBING_PORT PROBING_CLUSTER_REPORT=0 -# 心跳预设：PROBING_CLUSTER_PRESET=demo|fast|steady（见 docs/src/design/torchrun-cluster.zh.md） +# 心跳预设：PROBING_CLUSTER_PRESET=demo|fast|steady（见 docs/src/design/distributed.zh.md#cluster-membership） set -euo pipefail cd "$(dirname "$0")/../.." diff --git a/examples/getting-started/tracing.py b/examples/getting-started/tracing.py index 2d9aa225..7c3cd227 100644 --- a/examples/getting-started/tracing.py +++ b/examples/getting-started/tracing.py @@ -11,7 +11,8 @@ 训练循环里 **不需要** ``with probing.span("forward")``。 梯度累积：先 ``probing.step(micro_batches=N)``，``train.step`` 覆盖 - N 个 micro-batch 的 wall time；详见 ``docs/src/design/training-phase.zh.md``。 + N 个 micro-batch 的 wall time；详见 + ``docs/src/design/profiling.zh.md#training-phase-semantics``。 ② ``probing.span`` / ``probing.event`` — **可选，粗粒度时间线** 包住模型初始化、epoch 等；与 ① 的 phase span 互不冲突。 diff --git a/probing/cli/src/cli/help.rs b/probing/cli/src/cli/help.rs index 2af766fa..c69e142a 100644 --- a/probing/cli/src/cli/help.rs +++ b/probing/cli/src/cli/help.rs @@ -21,7 +21,8 @@ struct HelpSection { commands: &'static [&'static str], } -/// Help grouping — not argv paths. See `docs/src/design/cli.md` § Help sections. +/// Help grouping — not argv paths. See `docs/src/design/activation-injection.md` +/// § Control entry points and CLI structure. #[cfg(target_os = "linux")] const SECTIONS: &[HelpSection] = &[ HelpSection { diff --git a/tests/regression/profiling/test_overhead_invariants.py b/tests/regression/profiling/test_overhead_invariants.py index 889a0cef..976274c7 100644 --- a/tests/regression/profiling/test_overhead_invariants.py +++ b/tests/regression/profiling/test_overhead_invariants.py @@ -1,4 +1,4 @@ -"""Regression guards for TorchProbe overhead semantics (see docs/design/overhead-invariants.*).""" +"""Regression guards for TorchProbe overhead semantics (see design/overhead.*).""" from __future__ import annotations @@ -20,7 +20,7 @@ def test_close_step_wall_source_order(): mark_pos = src.index("_mark_step_wall_start") assert timing_pos < drain_pos < mark_pos, ( "_close_step_wall must be: record → drain → advance → mark " - "(see docs/src/design/overhead-invariants.zh.md §I3)" + "(see docs/src/design/overhead.zh.md#change-invariants)" ) @@ -38,11 +38,11 @@ def test_deferred_drain_async_default_on(monkeypatch): assert dd.deferred_drain_async_enabled() is True -def test_overhead_invariants_doc_exists(): +def test_overhead_change_contract_exists(): """Design doc SSOT is present for agents.""" repo = Path(__file__).resolve().parents[3] - zh = repo / "docs/src/design/overhead-invariants.zh.md" - en = repo / "docs/src/design/overhead-invariants.md" + zh = repo / "docs/src/design/overhead.zh.md" + en = repo / "docs/src/design/overhead.md" assert zh.is_file(), f"missing {zh}" assert en.is_file(), f"missing {en}" text = zh.read_text(encoding="utf-8") @@ -52,7 +52,7 @@ def test_overhead_invariants_doc_exists(): "_record_step_timing", "PROBING_TORCH_DEFER_ASYNC", ): - assert needle in text, f"overhead-invariants doc missing: {needle!r}" + assert needle in text, f"overhead change contract missing: {needle!r}" def test_defer_settle_constants_documented_range(): diff --git a/web/DESIGN.md b/web/DESIGN.md index 6e27973b..5e7d8c2c 100644 --- a/web/DESIGN.md +++ b/web/DESIGN.md @@ -372,7 +372,7 @@ web/src/ |------|------| | Skills 格式与 catalog | `skills/README.md`、`skills/catalog.yaml` | | Agent / MCP 集成 | 仓库根 `AGENTS.md` | -| TorchProbe / overhead 不变量 | `docs/src/design/overhead-invariants.zh.md` | +| TorchProbe / overhead 不变量 | `docs/src/design/overhead.zh.md#change-invariants` | | Profiling / TorchProbe | `docs/src/design/profiling.zh.md` | | 扩展与自定义表 | `docs/src/design/extensibility.zh.md` | | 训练调试示例 | `docs/src/examples/training-debugging.zh.md` |