Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/soak.yml
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ jobs:
env:
SKIP_FRONTEND_CLEAN: "1"
run: |
if [ ! -f python/probing/bundled_web/public/index.html ]; then
if [ ! -f probing/server/web-assets/public/index.html ]; then
make frontend
fi
make wheel-ci
Expand Down Expand Up @@ -108,7 +108,7 @@ jobs:
env:
SKIP_FRONTEND_CLEAN: "1"
run: |
if [ ! -f python/probing/bundled_web/public/index.html ]; then
if [ ! -f probing/server/web-assets/public/index.html ]; then
make frontend
fi
make wheel-ci
Expand Down
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -36,4 +36,6 @@ docs/site/
web/target/
probing/server/web-assets/
/web/dist
# Legacy Python-side Web copy (assets embed into probing._core via server/web-assets)
python/probing/bundled_web/
frontend/
4 changes: 2 additions & 2 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -180,8 +180,8 @@ Keep the propagation chain clean — don't reintroduce scattered `map_err`/`insp

## Skills

All diagnostic skills live under **`skills/`** (authoring SSOT; wheel copy in
`python/probing/bundled_skills/`). Each subdirectory contains:
All diagnostic skills live under **`python/probing/bundled_skills/`** (content SSOT;
repo-root `skills/` is a symlink alias). Each subdirectory contains:

- **`SKILL.md`** — when to use the skill and how to interpret results (read this for routing)
- **`steps.yaml`** — executable probe steps (used by `probing skill run` and the Web Investigate agent)
Expand Down
3 changes: 3 additions & 0 deletions docs/archive/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# Archived reviews

One-off design/implementation review notes (2025-07). Not part of the living design docs; see `docs/src/design/modularity.md` for current boundaries.
File renamed without changes.
File renamed without changes.
File renamed without changes.
10 changes: 5 additions & 5 deletions docs/src/contributing.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ Preview docs while editing: `make docs-install && make docs-serve` → http://12
| **Rust** | SQL engine, server, collectors, CLI | `probing/` (Rust workspace) | [Modularity](design/modularity.md) | Issues in `probing/core`, `probing/server`, extensions |
| **Web UI** | Investigate agent, dashboards | `web/` | [web/DESIGN.md](https://github.com/DeepLink-org/probing/blob/main/web/DESIGN.md) | Agent UX, page polish (needs `dx` for full wheel build) |

**Skills vs Python package:** edit skill **data** in repo-root `skills/` (symlink to `python/probing/bundled_skills/`); edit skill **loader / install code** in `python/probing/skills/`.
**Skills vs Python package:** skill **data** SSOT is `python/probing/bundled_skills/` (repo-root `skills/` is a symlink to it); edit skill **loader / install code** in `python/probing/skills/`.

**Two folders named `probing/`:** `probing/` at the repo root is **Rust**; `python/probing/` is the **Python package**. `src/lib.rs` at the root is the PyO3 entry for `probing._core`.

Expand Down Expand Up @@ -178,9 +178,9 @@ See [examples/README.md](https://github.com/DeepLink-org/probing/blob/main/examp

## Skills & agents {#skills-agents}

- **Authoring**: repo root `skills/` (`SKILL.md`, `steps.yaml`, `catalog.yaml`)
- **Authoring**: `python/probing/bundled_skills/` (repo-root `skills/` symlink; `SKILL.md`, `steps.yaml`, `catalog.yaml`)
- **Install to IDE agents**: `./skills/install.sh` or `probing skill install`
- **Bundled in wheel**: `make wheel` copies skills into `python/probing/bundled_skills/`; `make frontend` builds ignored UI artifacts under `probing/server/web-assets/`, and the server build script embeds them into `probing._core` at compile time (plain Rust builds use a tracked fallback page)
- **Bundled in wheel**: maturin packages `python/probing/bundled_skills/` directly (no separate copy step); `make frontend` builds ignored UI artifacts under `probing/server/web-assets/`, and the server build script embeds them into `probing._core` at compile time (plain Rust builds use a tracked fallback page)
- **Docs**: `skills/README.md`, [Extensibility — Diagnostic skill](design/extensibility.md#path-2-diagnostic-skill)

## Development workflow
Expand Down Expand Up @@ -246,11 +246,11 @@ Advanced: `cd docs && make deploy` for GitHub Pages.

```
probing/ # repo root
├── skills/ # skill DATA (authoring) — see skills/README.md
├── skills/ # symlink → python/probing/bundled_skills/ (authoring alias)
├── python/
│ ├── probing/ # Python PACKAGE (not Rust)
│ │ ├── skills/ # skill loader/install CODE — see python/probing/skills/README.md
│ │ └── bundled_skills/ # skill DATA bundled in wheel (author in repo-root skills/)
│ │ └── bundled_skills/ # skill DATA SSOT (packaged in wheel)
│ ├── probing_hook.py # .pth → site hook
│ └── probing.pth
├── src/lib.rs # PyO3 entry → probing._core (maturin)
Expand Down
10 changes: 5 additions & 5 deletions docs/src/contributing.zh.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ make test-python-regression # Python 冒烟;完整套件:make test
| **Rust** | 引擎、服务、采集、CLI | `probing/`(Rust workspace) | [模块化](design/modularity.zh.md) | `probing/core`、`probing/server` 相关 issue |
| **Web UI** | Investigate、各页面 | `web/` | [web/DESIGN.md](https://github.com/DeepLink-org/probing/blob/main/web/DESIGN.md) | Agent 体验(完整 wheel 构建需 `dx`) |

**Skill 数据 vs Python 包:** skill **内容**改仓库根 `skills/`(wheel 复制到 `python/probing/bundled_skills/`);**加载/安装代码**改 `python/probing/skills/`。**不要**手改 `python/probing/bundled_skills/`(由 `make wheel` 自动生成)
**Skill 数据 vs Python 包:** skill **内容** SSOT 是 `python/probing/bundled_skills/`(仓库根 `skills/` 为其符号链接别名);**加载/安装代码**改 `python/probing/skills/`。

**两个 `probing/` 目录:** 仓库根 `probing/` 是 **Rust**;`python/probing/` 是 **Python 包**。根目录 `src/lib.rs` 是 PyO3 入口,构建为 `probing._core`。

Expand Down Expand Up @@ -174,9 +174,9 @@ PROBING=1 python examples/getting-started/tracing.py

## Skills 与 Agent {#skills-agents}

- **编写**:仓库根 `skills/``SKILL.md`、`steps.yaml`、`catalog.yaml`)
- **编写**:`python/probing/bundled_skills/`(仓库根 `skills/` 为符号链接;`SKILL.md`、`steps.yaml`、`catalog.yaml`)
- **安装到 IDE**:`./skills/install.sh` 或 `probing skill install`
- **打进 wheel**:`make wheel` 复制 Skill 到 `python/probing/bundled_skills/`;`make frontend` 将被忽略的 UI 产物构建到 `probing/server/web-assets/`,server build script 在编译时把它嵌入 `probing._core`(普通 Rust 构建使用受版本控制的 fallback 页面)
- **打进 wheel**:maturin 直接打包 `python/probing/bundled_skills/`(无额外复制步骤);`make frontend` 将被忽略的 UI 产物构建到 `probing/server/web-assets/`,server build script 在编译时把它嵌入 `probing._core`(普通 Rust 构建使用受版本控制的 fallback 页面)
- **说明**:`skills/README.md`、[扩展机制 — 诊断 skill](design/extensibility.zh.md#path-2-diagnostic-skill)

## 开发流程
Expand Down Expand Up @@ -241,11 +241,11 @@ make docs # 静态构建 → docs/site/

```
probing/ # 仓库根
├── skills/ # skill 数据(编写)— 见 skills/README.md
├── skills/ # 符号链接 → python/probing/bundled_skills/(编写别名)
├── python/
│ ├── probing/ # Python 包(不是 Rust)
│ │ ├── skills/ # skill 加载/安装代码 — 见 python/probing/skills/README.md
│ │ └── bundled_skills/ # wheel 打包的 skill 数据(编写在 repo-root skills/
│ │ └── bundled_skills/ # skill 数据 SSOT(直接打进 wheel
│ ├── probing_hook.py
│ └── probing.pth
├── src/lib.rs # PyO3 → probing._core
Expand Down
174 changes: 3 additions & 171 deletions docs/src/design/architecture.zh.md
Original file line number Diff line number Diff line change
@@ -1,176 +1,8 @@
# 系统架构

!!! warning "旧版文档"
本页的两层概览已被 **[模块化与边界](modularity.zh.md)**(四层模型、crate 地图、依赖规则与归属)**取代**。仅保留历史示意图。**新贡献者请从 [模块化](modularity.zh.md)** 入手,再读 [数据层](data-layer.zh.md) 与 [分布式概览](distributed.zh.md)。
本页的两层概览已被 **[模块化与边界](modularity.zh.md)**(四层模型、crate 地图、依赖规则与归属)**取代**。

Probing 采用简洁的两层架构设计,以最小化复杂性和部署难度
**新贡献者:** 从 [模块化与边界](modularity.zh.md) 入手,再读 [数据层](data-layer.zh.md) 与 [分布式概览](distributed.zh.md)

## 高层架构

```mermaid
graph TB
subgraph "用户界面"
CLI[CLI 客户端]
HTTP[HTTP API]
WEB[Web UI]
end

subgraph "目标进程"
PROBE[探针]
subgraph "探针组件"
ENGINE[引擎]
SERVER[服务器]
EXT[扩展]
end
end

CLI --> |Unix Socket / TCP| PROBE
HTTP --> |HTTP/REST| PROBE
WEB --> |WebSocket| PROBE

PROBE --> ENGINE
PROBE --> SERVER
PROBE --> EXT
```

## 组件

### 1. 探针 (Probe)

注入到目标进程中,获得对所有资源的完全访问权限:

- Python 解释器访问
- 文件系统访问
- 内存检查
- 网络能力

探针运行一个嵌入式 HTTP 服务器,监听在:

- **Unix 域套接字** - 用于本地连接(默认)
- **TCP 端口** - 用于远程连接

### 2. CLI

用于控制探针的命令行界面:

- 进程发现和列表
- 探针注入和管理
- 查询执行
- 代码执行

通过 HTTP 协议在 Unix 域套接字(本地)或 TCP(远程)上通信。

### 3. HTTP API

用于编程访问的 RESTful API:

- 所有 CLI 命令都可作为端点使用
- WebSocket 支持实时数据
- 与监控工具集成

## 探针内部架构

```mermaid
graph LR
subgraph "探针"
SERVER[HTTP 服务器]
ENGINE[查询引擎]
CONFIG[配置管理]

subgraph "扩展"
PYTHON[Python 扩展]
TORCH[PyTorch 扩展]
CUSTOM[自定义表]
end
end

SERVER --> ENGINE
SERVER --> CONFIG
ENGINE --> PYTHON
ENGINE --> TORCH
ENGINE --> CUSTOM
```

### 引擎

核心数据存储和处理:

- **DataFusion** - SQL 查询引擎
- **Arrow** - 列式数据格式
- 时间序列数据存储
- 配置管理

### 服务器

HTTP 服务器处理:

- 请求路由
- 认证(可选)
- WebSocket 连接
- 响应格式化

### 扩展

可插拔的数据提供者:

- **Python 扩展** - 堆栈跟踪、变量
- **PyTorch 扩展** - Torch 跟踪、内存
- **自定义表** - 用户定义的数据源

## 数据流

```mermaid
sequenceDiagram
participant CLI
participant 探针
participant 引擎
participant 扩展

CLI->>探针: query "SELECT * FROM python.torch_trace"
探针->>引擎: 解析 & 计划查询
引擎->>扩展: 请求数据
扩展-->>引擎: Arrow RecordBatch
引擎-->>探针: 查询结果
探针-->>CLI: JSON 响应
```

## 通信协议

### 本地连接

```
probing -t <pid> query "..."
|
v
Unix Socket: /tmp/probing-<pid>.sock
|
v
HTTP 请求: POST /query
```

### 远程连接

```
probing -t host:port query "..."
|
v
TCP 连接: host:port
|
v
HTTP 请求: POST /query
```

## 安全考虑

- **本地模式**: Unix 套接字权限(仅进程所有者)
- **远程模式**: 可选认证
- **网络**: 支持 TLS 加密

## 性能特征

| 方面 | 目标 |
|------|------|
| 开销 | 典型工作负载 < 5% |
| 内存 | 额外 < 50MB |
| 延迟 | 查询 < 10ms |
| 吞吐量 | 1000+ 查询/秒 |
为避免与当前 crate 地图漂移,旧版两层 Mermaid 示意图已移除。
21 changes: 11 additions & 10 deletions docs/src/design/modularity.md
Original file line number Diff line number Diff line change
Expand Up @@ -128,7 +128,7 @@ component state, logs, and readiness responses.
| Unit | Path | Responsibility |
|------|------|----------------|
| **web/** | Dioxus WASM | Pages, visualization, Investigate agent |
| **skills/** | YAML + SKILL.md | Skill content SSOT |
| **skills/** | symlink → `python/probing/bundled_skills/` | Authoring alias for skill content SSOT |
| **probing-skills** | `probing/crates/skills/` | Shared loader, interpreter, runner (CLI / Web / MCP) |
| **python/probing/extensions/** | entry points | Skill + magic + vendor package discovery |
| **python/probing/** | Python package | Hooks, `query()`, agent install helpers |
Expand Down Expand Up @@ -192,7 +192,7 @@ class CommCollective: ...

### 3.4 Skill contract — diagnostic workflows

**Where:** `skills/<id>/SKILL.md` + `steps.yaml`, catalog `skills/catalog.yaml`
**Where:** `python/probing/bundled_skills/<id>/SKILL.md` + `steps.yaml` (alias: `skills/`), catalog `catalog.yaml`

| Field | Purpose |
|-------|---------|
Expand All @@ -205,7 +205,7 @@ class CommCollective: ...

- Skills **only** talk to the engine via SQL (`probing query`) or documented HTTP APIs.
- No Rust/Python code in skills — YAML + markdown only.
- **Content SSOT:** `skills/` (wheel: `python/probing/bundled_skills/`).
- **Content SSOT:** `python/probing/bundled_skills/` (repo-root `skills/` is a symlink alias).
- **Discovery:** `python/probing/extensions/` entry points + `GET /apis/pythonext/skills/*`.
- **Execution SSOT:** `probing-skills` — CLI (`probing/cli/skill/`), Web WASM
(`web/src/agent/runner.rs`), MCP (`run_skill` / `plan_skill` in server).
Expand Down Expand Up @@ -259,7 +259,7 @@ Forbidden (fix if found):
L1 → L2/L3/L4
L2 → L2 (collector cross-deps)
L2 → L3 (extensions must not import server)
L2 → probing-cli — **packaging only** (see §4.1; not a runtime collector→control violation)
L2 → probing-cli — collectors must not depend on CLI (wheel `cli_main` is wired in root `src/lib.rs`; see §4.1)
skills → Rust internals
web → probing-core / pyo3
```
Expand All @@ -276,7 +276,7 @@ Forbidden (fix if found):
| **cli** | ✓ | opt | | | | — |
| **web** | ✓ | | | | | |

### 4.1 PyPI packaging coupling (`probing-python` → `probing-cli`)
### 4.1 PyPI packaging coupling (cdylib → `probing-cli`)

Maturin builds **one native artifact** (`probing._core` cdylib from root `Cargo.toml`). The
`probing` console script is **not** a separate Rust binary on PyPI:
Expand All @@ -293,15 +293,16 @@ control plane at runtime for data paths.

**Contract (keep the edge thin):**

- `probing-python` may depend on `probing-cli` **only** to re-export `cli_main` in
`features/python_api.rs` for the PyO3 entrypoint.
- Root `src/lib.rs` (the maturin cdylib composition root) may call `probing_cli::pyo3::cli_main`
and re-export it as `_core.cli_main`. `probing-python` itself does **not** depend on
`probing-cli`.
- Do **not** import other `probing-cli` modules (inject, skill runner internals, ctrl) from
collectors or server.
- Standalone Rust binary (`probing/cli/src/main.rs`) remains optional for non-PyPI installs;
PyPI users always go through the Python script entry.

If CLI logic grows, split **`probing-cli-lib`** (shared `cli_main` + HTTP client) from CLI-only
commands, rather than letting `probing-python` spread imports across the cli crate.
commands, rather than letting collectors or the cdylib spread imports across the cli crate.

---

Expand Down Expand Up @@ -387,11 +388,11 @@ Track and fix incrementally:

| Issue | Current | Target |
|-------|---------|--------|
| Python ext → CLI | `probing-python` → `probing-cli` | **Accepted** for maturin wheel (`cli_main` only); keep import surface minimal |
| Python ext → CLI | ~~`probing-python` → `probing-cli`~~ | **Done** — no crate dep; wheel `cli_main` wired only in root `src/lib.rs` (§4.1) |
| Python ext → CC | ~~`probing-python` → `probing-cc`~~ | **Done** — `send_sigusr2_to_thread_id` moved to `probing-core::signal` |
| Core → NCCL/HCCL | ~~`probing-core` → `probing-nccl-profiler` / `probing-hccl-shim` (`builtin-schema-docs` feature) for `semantic_catalog`~~ | **Done** — `register_docs()` called from `server/engine.rs` composition root; `probing-core` default features empty |
| Core → skills YAML | ~~`semantic_catalog.rs` `include_str!(skills/semantic/tables.yaml)`~~ | **Done** — overlay at `probing/core/resources/tables.yaml`; descriptions SSOT in `docs` registry |
| Server → python `features/*` | ~~`server/profiling.rs`~~ removed | Flamegraphs via `torchextension` / `pprofextension` `ProbeExtensionCall` |
| Server → python `features/*` | ~~`server/profiling.rs` / training flamegraph handlers~~ | **Done** — flamegraphs via `torchextension` / `pprofextension` `ProbeExtensionCall` only |
| Server → python REPL internals | ~~`PythonRepl` in server~~ | `/ws` uses `ReplSession` facade only |
| Composition sprawl | All wiring in `server/engine.rs` | Optional: manifest TOML listing enabled extensions |
| Skills triple loader | ~~Rust + Python + Web compile-time embed~~ | **Done** — `probing-skills` is loader/interpret/runner SSOT; Python keeps discovery entry-points + PyO3 serialize bridge; Web deserializes API into shared types |
Expand Down
Loading
Loading