Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 19 additions & 10 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -159,21 +159,30 @@ Inspect a completed or interrupted run without manually scanning JSONL:

```bash
uv run yada-trace \
.yada/runs/fix-parser-edge-case__2026-08-02_12-26-26.123456Z.jsonl
.yada/runs/fix-parser-edge-case__2026-08-02_20-26.jsonl
uv run yada-trace \
eval-results/pytest-dev__pytest-10051__2026-08-02_12-26-26.123456Z.artifacts/yada-trace.jsonl \
eval-results/pytest-dev__pytest-10051__2026-08-02_20-26.artifacts/yada-trace.jsonl \
--step 8
uv run yada-trace eval-results/<task>__<UTC-time>.artifacts/yada-trace.jsonl \
uv run yada-trace eval-results/<task>__<system-local-time>.artifacts/yada-trace.jsonl \
--verbose
uv run yada-trace TRACE.jsonl --events
```

The report correlates model requests, tool-call IDs, errors, reminders, and the
final verification state into a compact timeline. `--step` and `--verbose`
expand sanitized model messages, tool arguments, patches, stdout, and stderr.
The source JSONL remains the durable, streaming-friendly record. Debug traces can
contain source code and test output even after secret redaction, so handle them as
sensitive artifacts. See [docs/tracing.md](docs/tracing.md) for the event
reference, field-presence semantics, lifecycle, and `jq` recipes.
The default report groups each model request, response, planning decision, and
ordered tool executions into one agent step. Every summary includes physical
JSONL line references so the source evidence is immediately reachable with tools
such as `sed`. `--step` and `--verbose` expand sanitized model messages, tool
arguments, patches, stdout, and stderr inside grouped steps; `--events` retains a
line-prefixed flat timeline. The source JSONL remains the durable,
streaming-friendly record. Debug traces can contain source code and test output
even after secret redaction, so handle them as sensitive artifacts. See
[docs/tracing.md](docs/tracing.md) for the event reference, field-presence
semantics, lifecycle, and `jq` recipes.

Default trace and evaluation paths use the system-local time at minute
precision. If a name already exists, Yada appends `(1)`, `(2)`, and so on before
the file or artifacts suffix, keeping the result JSON and artifacts directory on
the same number.

## Safety model

Expand Down
13 changes: 10 additions & 3 deletions README.zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -119,11 +119,18 @@ uv run yada eval \
uv run yada-trace .yada/runs/20260801T120000.000000Z.jsonl
uv run yada-trace eval-results/<run>.artifacts/yada-trace.jsonl --step 8
uv run yada-trace eval-results/<run>.artifacts/yada-trace.jsonl --verbose
uv run yada-trace TRACE.jsonl --events
```

报告会汇总模型轮次、工具调用 ID、失败、协议提醒和最终验证状态。
`--step` 和 `--verbose` 会展开脱敏后的模型消息、工具参数、Patch、stdout 和
stderr。Debug trace 脱敏后仍可能包含源码和测试输出,应当作敏感 artifact 处理。
默认报告会按 Agent step 归组模型请求、响应、规划决定和有序工具执行,并为
step、模型调用、工具执行和协议事件显示真实 JSONL 行号。`--step` 和
`--verbose` 会在分组内展开脱敏后的模型消息、工具参数、Patch、stdout 和
stderr;`--events` 可切回带物理行号的平铺时间线。Debug trace 脱敏后仍可能
包含源码和测试输出,应当作敏感 artifact 处理。

默认 trace 和评测路径使用精确到分钟的系统本地时间,不包含秒和小数秒。
若名称已存在,Yada 会在文件或 `.artifacts` 后缀前依次添加 `(1)`、`(2)`;
同一次评测的结果 JSON 与 artifacts 目录始终使用相同编号。

## 安全边界

Expand Down
13 changes: 7 additions & 6 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -96,12 +96,13 @@ results remain correlated by step, request ID, and tool-call ID. `run_start`
records Yada version/commit, workspace base commit, case ID when available, and
the model configuration.

`yada-trace PATH` renders the compact summary. `--step N` expands one request →
response → tools slice, while `--verbose` expands the full timeline. Reasoning
is length/hash-redacted in summary traces and automatically retained in debug
traces. Common secret keys and bearer/API-key-like text are redacted in both
modes. A debug trace can still contain reasoning, source code, and test output
and must be handled as a sensitive artifact.
`yada-trace PATH` renders one source-located section per agent step. `--step N`
expands one complete request → response → tools step, while `--verbose` expands
every grouped step and `--events` provides the line-prefixed flat timeline.
Reasoning is length/hash-redacted in summary traces and automatically retained in
debug traces. Common secret keys and bearer/API-key-like text are redacted in
both modes. A debug trace can still contain reasoning, source code, and test
output and must be handled as a sensitive artifact.
The complete event and field reference lives in [tracing.md](tracing.md).

The MVP stores a full sanitized request snapshot per turn. This deliberately
Expand Down
38 changes: 32 additions & 6 deletions docs/tracing.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,13 @@ run_end
- A transport exception or process interruption can leave a trace without
`run_end`. Readers should report this as interrupted, not successful.

The reporting layer normalizes these records into `TraceRun`, `TraceStep`,
`TraceToolExecution`, and `LocatedTraceEvent` objects. A located event pairs the
unchanged persisted record with its physical JSONL line number. The public
`read_trace()` API still returns the original event dictionaries; source location
metadata is available through `read_located_trace()` and is never written back to
JSONL.

## Core events

| Event | Cardinality and meaning | Important `data` fields |
Expand Down Expand Up @@ -97,19 +104,38 @@ output.

## Inspection recipes

Default paths include a sanitized task name and a readable UTC timestamp. A
direct run resembles
`.yada/runs/fix-parser-boundary-issue__2026-08-02_12-26-26.123456Z.jsonl`.
An evaluation stores its trace under a directory such as
`eval-results/pytest-dev__pytest-10051__2026-08-02_12-26-26.123456Z.artifacts/`.
Explicit `--trace`, `--output`, and `--artifact-dir` values are never renamed.
Default paths include a sanitized task name and the system-local time at minute
precision. A direct run resembles
`.yada/runs/fix-parser-boundary-issue__2026-08-02_20-26.jsonl`. An evaluation
stores its trace under a directory such as
`eval-results/pytest-dev__pytest-10051__2026-08-02_20-26.artifacts/`. When a
default name already exists, Yada adds `(1)`, `(2)`, and so on before the output
suffix. The result JSON and artifacts directory share the same number. Explicit
`--trace`, `--output`, and `--artifact-dir` values are never renamed.

Render a run summary or expand one step:

```bash
uv run yada-trace TRACE.jsonl
uv run yada-trace TRACE.jsonl --step 12
uv run yada-trace TRACE.jsonl --verbose
uv run yada-trace TRACE.jsonl --events
```

The default view groups events by agent step. Step headings show the complete
physical line range, model calls identify their request and response lines, and
tool executions identify their call and result lines. Protocol reminders and
violations also carry line references. Blank JSONL lines still count as physical
lines, while `sequence` remains the deterministic event-order field; the two are
not interchangeable. `--events` keeps the previous flat timeline shape and
prefixes every event with its physical line.

Once a report identifies a suspicious step or tool execution, inspect the exact
source records directly:

```bash
sed -n '31p' TRACE.jsonl
sed -n '33,34p' TRACE.jsonl
```

List event counts:
Expand Down
12 changes: 9 additions & 3 deletions src/yada/evals/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@
from yada.evals.base import RunBudget
from yada.evals.benchmarks import LocalBenchmark, SWEbenchBenchmark
from yada.evals.runner import EvalRunner
from yada.utils.naming import readable_run_name
from yada.utils.naming import next_available_run_name, readable_run_name


def build_parser() -> argparse.ArgumentParser:
Expand Down Expand Up @@ -45,7 +45,7 @@ def build_parser() -> argparse.ArgumentParser:
"--output",
type=Path,
help=(
"Result JSON path (default: eval-results/<task>__<readable-UTC-time>.json)."
"Result JSON path (default: eval-results/<task>__<system-local-time>.json)."
),
)
parser.add_argument(
Expand Down Expand Up @@ -242,7 +242,13 @@ def run_cli(argv: list[str] | None = None) -> int:


def _default_output_path(task_name: str) -> Path:
return Path("eval-results") / f"{readable_run_name(task_name)}.json"
directory = Path("eval-results")
run_name = next_available_run_name(
directory,
readable_run_name(task_name),
suffixes=(".json", ".artifacts"),
)
return directory / f"{run_name}.json"


def _manifest_instance_id(benchmark: LocalBenchmark) -> str:
Expand Down
12 changes: 9 additions & 3 deletions src/yada/run/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@
from yada.models import DeepSeekAPIError, DeepSeekClient
from yada.tools import ToolRunner
from yada.traces import TraceWriter
from yada.utils.naming import readable_run_name
from yada.utils.naming import next_available_run_name, readable_run_name


def build_parser() -> argparse.ArgumentParser:
Expand Down Expand Up @@ -67,7 +67,7 @@ def build_parser() -> argparse.ArgumentParser:
type=Path,
help=(
"JSONL trace path (default: "
"WORKSPACE/.yada/runs/<task>__<readable-UTC-time>.jsonl)."
"WORKSPACE/.yada/runs/<task>__<system-local-time>.jsonl)."
),
)
parser.add_argument(
Expand Down Expand Up @@ -183,7 +183,13 @@ def run_cli(argv: list[str] | None = None) -> int:


def _default_trace_path(workspace: Path, task: str) -> Path:
return workspace / ".yada" / "runs" / f"{readable_run_name(task)}.jsonl"
directory = workspace / ".yada" / "runs"
run_name = next_available_run_name(
directory,
readable_run_name(task),
suffixes=(".jsonl",),
)
return directory / f"{run_name}.jsonl"


def main() -> None:
Expand Down
12 changes: 12 additions & 0 deletions src/yada/traces/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,13 @@

from yada.traces.jsonl import TRACE_LEVELS, TRACE_SCHEMA_VERSION, TraceWriter
from yada.traces.report import (
LocatedTraceEvent,
TraceFormatError,
TraceRun,
TraceStep,
TraceToolExecution,
build_trace_run,
read_located_trace,
read_trace,
reconstruct_model_request,
render_trace_report,
Expand All @@ -11,8 +17,14 @@
__all__ = [
"TRACE_LEVELS",
"TRACE_SCHEMA_VERSION",
"LocatedTraceEvent",
"TraceRun",
"TraceStep",
"TraceFormatError",
"TraceToolExecution",
"TraceWriter",
"build_trace_run",
"read_located_trace",
"read_trace",
"reconstruct_model_request",
"render_trace_report",
Expand Down
10 changes: 8 additions & 2 deletions src/yada/traces/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ def build_parser() -> argparse.ArgumentParser:

parser = argparse.ArgumentParser(
prog="yada-trace",
description="Summarize a Yada JSONL run as a correlated timeline.",
description="Summarize a Yada JSONL run as correlated agent steps.",
)
parser.add_argument("trace", type=Path, help="Path to a Yada JSONL trace.")
parser.add_argument(
Expand All @@ -25,7 +25,12 @@ def build_parser() -> argparse.ArgumentParser:
parser.add_argument(
"--verbose",
action="store_true",
help="Expand event payloads for the complete timeline.",
help="Expand event payloads inside every grouped step.",
)
parser.add_argument(
"--events",
action="store_true",
help="Show the legacy flat event timeline with physical line numbers.",
)
return parser

Expand All @@ -39,6 +44,7 @@ def run_cli(argv: list[str] | None = None) -> int:
args.trace.expanduser().resolve(),
step=args.step,
verbose=args.verbose,
events=args.events,
)
except (OSError, TraceFormatError) as exc:
print(f"yada-trace: {exc}", file=sys.stderr)
Expand Down
Loading