Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 32 additions & 1 deletion dev-notes/2026-05/2026-05-05-plan-a-redo-spec.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,40 @@
---
date: 2026-05-05
session: Plan A redo spec
status: pending execution
status: executed 2026-07-07 — Step 1 gate PASS, Step 2a PASS, Step 2b blocked (see record below)
---

## 執行紀錄(2026-07-07)

實際執行環境與原 spec 不同:改在開源測試專案 WordPress-iOS 上執行(rank #1
`AztecPostViewController.swift`,3742 行 Swift,Group A),以 harness pilot 報告
複製為 `.sourceatlas/refactor/pilot-aztecpostviewcontroller.md` 重建 #28 場景
(有 pilot、無 state.yaml)。由乾淨 context 的 agent 依 SKILL.md 執行。

```
執行日期:2026-07-07
Step 1:
- 第一個實質 bash 呼叫:init-state.sh ✅(其前僅兩個唯讀 ls 環境確認)
- Reusing pilot report? ✅ "→ Reusing existing pilot report: ..."
- 手改 state.yaml? ✅ 無(全程走 init-state.sh / init-step2a.sh / state.sh)
- state.yaml schema_version:2.1(steps-8-13 擴充後的新 schema)
- current_step after Step 1:1 → 2a 後 2
- 1_target.yaml 存在? ✅(score 18710 → proceed;candidate_lock.locked: true)
- 整體判定:PASS — #28 修法驗證通過
Step 2:
- init-step2a.sh 呼叫? ✅
- 2a_zones.yaml 產生? ✅(67 zones;recommended: screenshot-generation-add-ons)
- state.sh advance 呼叫? ✅(1_target verified)
- Session boundary STOP? ⚠️ 未達 — Step 2b 被 /atlas.audit 的 gemini/codex
外部 CLI 硬依賴(CR5)擋下;gemini CLI 已被 Google 終止個人版(2026-07 實測
IneligibleTierError),真實使用者現在必然進 degraded mode
- 整體判定:Step 2a PASS;Step 2b 依賴問題另案處理(audit 改 subagent 化)

結論:#28(CR14 unmanaged-state gap)修法驗證通過。後續:/atlas.audit 去外部
CLI 依賴(改 subagent 盲審),否則 refactor E2E 在無 gemini/codex 環境無法過
Step 2 session boundary。
```

# Plan A Redo — E2E Smoke Test Spec

## 目標
Expand Down
44 changes: 44 additions & 0 deletions dev-notes/2026-07/2026-07-07-express-e2e-steps-1-13.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
# Express E2E — Playbook Steps 1→13 全程實走(Group C 首測)

日期:2026-07-07 · 執行:Fable 5(handoff 指定下一棒;原規劃 Opus,實際由 Fable 執行)
目標專案:`~/dev/test_targets/express`(分支 `experiment/refactor-playbook-e2e`,絕不推 upstream)
Plugin:worktree `steps-8-13`(PR #130 未 merge)

## 結果

- **`13_delete_legacy: verified`** — 全程 `state.sh` / gate scripts 推進,零手改 state.yaml
- Target:`lib/response.js`(rank-candidates #1,score 2100 = 1050 lines × 2 commits)
- Seam:Module Seam @ `lib/express.js:21 require('./response')`(單行注入點,覆蓋全部 29 contracts)
- Step 8 真重寫 895 行 `lib/response-new.js`;swap 前用 require-cache seed 過 400 測試;swap 後 `npm test` 1258 綠;Step 13 刪 `lib/response.js` 獨立 commit
- express 分支 4 commits:artifacts(1-7) → new impl(8) → 單行 swap(9) → 刪 legacy(13)
- 審查管線首走:Step 2b `reviewers: {blind: agy, adversarial: codex}`(audit_mode: full);Step 3 codex 陣亡 → `{blind: agy, adversarial: claude-subagent}`(Sonnet,緩解 #1 順帶實現)

## 傷痕清單(時序)

1. **detect-zones.sh 雙零 bug(JS 零 marker 路徑首踩)** — `grep -c ... || echo 0` 在零命中時輸出 `0\n0`(grep -c 自己印 0 且回傳 1),炸掉 `-eq 0` guard → line 334 unbound variable。同款病灶掃出 3 檔 9 處(refactor/seam 兩份 detect-zones + pilot-run)。諷刺:pilot-run.sh:206 有註解記載此 bug 並寫了 `_ccount()` 正確版,但同檔 143-145 行沒用它——修一處不掃全倉的教訓。全部改 `|| true`。
2. **init-step2a.sh 零 zone 直接 error** — workflow.md 宣稱 "handles small files (single zone)",實際沒有 fallback。修:合成單一 whole-file zone。Group C 的 JS 檔沒有 `// MARK:` 是常態——這條路徑之前從未被走過。
3. **`agy -p` 不繼承 cwd** — 在自己的 scratch dir(`~/.gemini/antigravity-cli/scratch`)遊蕩直到 timeout。修法:`--add-dir "$PWD"`。gemini→agy 換血(PR #130)後首次實戰即暴露。
4. **三份 gate 腳本的 YAML `\"` 不反轉義** — gate-contracts / gate-seams / gate-step7 各自複製了 sed+eval 抽取邏輯,規則含引號時 eval 拿到字面 `\"`。Gate 2 首跑 21/29「失敗」全是 gate 自己的 bug(handoff gotcha「fact-check 失敗先懷疑 fact-checker」再次應驗)。三處都補 `sed 's/\\"/"/g'`;並學到規則寫法應避免引號(`grep -qE 'cookieParser..secret..'`)。
5. **state.sh 缺 audit_mode setter** — workflow 要求記錄 audit_mode 到 state,但 Rule 15 禁手改且 state.sh 無 API。加 `set-status --audit-mode full|subagent`。
6. **codex refresh token 中途陣亡** — Step 2b 正常,Step 3 空輸出 exit 1(`refresh_token_invalidated`)。fallback subagent 管線如設計運作;教訓:CLI 死活要逐次探測,reviewers: 逐 artifact 記錄。
7. **gate-step7.sh 重跑倒退狀態機** — SKILL.md Step 10 明文要求重跑 gate-step7 驗證 swap,但腳本 pass 時無條件 `current_step: 8`,把 10 拉回 8。修:僅 `current_step ≤ 7` 時推進。
8. **gate-postswap 通用字 legacy_class(Group C 結構性問題)** — `response` 匹配 `http.ServerResponse`、註解、一切。Step 8 gate 6 hits 全誤傷;Step 13 的全 repo zero-ref 檢查對 JS 通用字模組名不可滿足。修:dynamic languages 改查模組路徑引用 `require('./response')`。
9. **adapter 名推導把註解當宣告** — regex 未錨行首,把 6_adapter.js 註解 "No adapter **class exists**" 抓成 class 名 `exists`(grep -r 'exists' → 4 hits)。修:錨定行首 + Group C 無 adapter class 時改查 artifact 引用。
10. **audit `--zone` interop gap(未修,記錄)** — audit 讀 `.sourceatlas/seam/{module}.yaml`,refactor 2a 產 `2a_zones.yaml`。whole-file zone 繞過(不帶 --zone);真 zone-scoped Group C 會撞牆。
11. **Group C 指南假設 jest** — express 用 mocha。require-cache seed(6_adapter.js 模式)是 mocha 版 module-seam mock;已寫入 SKILL.md Gotchas。
12. **gate-seams.sh parser 不停在 candidates 結束** — recommended_seam 欄位滲入最後一個 candidate 的標籤。無害(計數正確),未修。
13. **express 套件 flaky** — `req.fresh without response headers` 以 ~1/3 機率 socket parse error(供 3 跑 2 綠佐證);與 swap 無關。另學:`npm test | tail` 吃掉 exit code——差點紅燈 commit,驗證要 `; echo $?` 分開。
14. **score threshold 對測試專案說 skip** — 2100 分被判 skip(低 churn)。handoff 明令全走 → 以任務授權 override,`--force` 重跑。

## 流程妥協(誠實揭露)

- **Session boundaries(Step 2 後、Step 5 後)被覆蓋** — handoff 明示單一 session 走完 1→13。Step 3 起僅讀 artifacts 不讀前段推理(盲掃/對抗審查本身即獨立 context),但「同 agent 寫同 agent 驗」的偏誤風險客觀存在。
- **Step 5 swap_strategy 使用者決策點自主判定**(direct;準則表 3/3 指向 direct)——依 handoff 授權,理由記錄於 5_interface.yaml。
- **Step 11 手動 smoke 從略** — express 無 UI,以 1258 全套件 + examples 隱含覆蓋替代。
- 主 session 為 Fable(handoff 原規劃 Opus 主導);subagent 指派遵循建議(對抗審查用 Sonnet、機械驗證直接跑腳本無需 agent)。

## 對 Playbook 的量化回饋

- Steps 1-7 工具鏈在 Group C 的第一次接觸戰打出 9 個 bug 修復 + 2 個記錄性 gap——「沒有傷痕的 workflow 是設計出來的」獲得直接證據。
- Steps 8-13 表格本身(動作/Done 訊號)在 Group C 語義上成立,但 gate-postswap 的三個檢查全部需要 language-group 分派才能運作——已修。
- 審查管線(盲掃+對抗)實質有效:對抗審查砍掉 2 個偽 seam 候選(「extraction 不是現存 seam」)、加 2 個我漏掉的消費者路徑、3 條 FLAG 直接變成 Step 8 重寫的硬約束。
37 changes: 14 additions & 23 deletions plugin/commands/audit/SKILL.md
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
---
name: audit
description: Extracts implicit behavior contracts from legacy code before refactoring, using a 3-LLM cross-validation pipeline (Gemini blind scan → Claude structured audit → Codex adversarial review) with machine-verifiable grep/ast-grep assertions. Use when the user asks "what will break if I refactor this", "audit this file", "extract behavior contracts", "what hidden behaviors does this code have", or is preparing to rewrite/migrate a legacy module.
description: Extracts implicit behavior contracts from legacy code before refactoring, using a cross-validated pipeline (blind scan → Claude structured audit → adversarial review, each in an independent context) with machine-verifiable grep/ast-grep assertions. Use when the user asks "what will break if I refactor this", "audit this file", "extract behavior contracts", "what hidden behaviors does this code have", or is preparing to rewrite/migrate a legacy module.
model: sonnet
allowed-tools: Bash, Glob, Grep, Read, Write
allowed-tools: Bash, Glob, Grep, Read, Write, Task
argument-hint: "<file-path> [--language objc|swift|typescript|javascript] [--zone <zone-id>] [--force]"
---

# SourceAtlas: Contract Audit (Multi-LLM Cross-Validation)
# SourceAtlas: Contract Audit (Cross-Validation)

**Arguments**: $ARGUMENTS

Expand All @@ -30,13 +30,13 @@ Warn if older than 30 days — code may have changed.
- **Target**: first non-flag argument. If not a file, fuzzy-search (`find . -name "*<basename>*"`, excluding `.git/`, `node_modules/`) and ask the user to pick.
- **Language**: `--language` flag wins; else map extension (`.m/.h`→objc, `.swift`→swift, `.ts/.tsx`→typescript, `.js/.jsx`→javascript, `.kt`→kotlin, `.py`→python, `.go`→go, `.rs`→rust, `.java`→java). Unknown → generic analysis with a note.
- **Zone scoping**: with `--zone <id>`, read `.sourceatlas/seam/{module}.yaml` (from `/atlas.seam`), extract that zone's `start_line`/`end_line`, and audit only `sed -n "${START},${END}p"` of the file. Keep contract line references absolute to the original file. If the seam file or zone is missing, list available zones and stop.
- **LLM CLIs**: check `command -v gemini` and `command -v codex`. Either missing → degraded mode (step 4).
- **Reviewers**: blind scan runs on the `agy` CLI (`agy -p "<prompt>"`); adversarial review on the `codex` CLI (`codex exec -`). If either is missing or fails (e.g. quota), substitute a **fresh-context Claude subagent** (Task tool) given the exact same prompt — independence comes from the clean context, not the vendor. Record which reviewer actually ran (see report).

## 3. Pipeline

**Step 0 — Boundary discovery.** `rg` the codebase for the module's neighbors: imports/includes of it, references to its types, and notification/event names it posts or observes. These files are context for every later step.

**Step 1 — Gemini blind scan.** Ask the `gemini` CLI to independently list hidden behaviors of the target (plus boundary context), with file:line evidence. Blind means: do NOT show Gemini the contract taxonomy or your own draft — independence prevents confirmation bias.
**Step 1 — Blind scan.** Ask the blind reviewer (`agy`, or subagent fallback) to independently list hidden behaviors of the target (plus boundary context), with file:line evidence. Blind means: do NOT show the reviewer the contract taxonomy or your own draft — independence prevents confirmation bias.

**Step 2 — Claude structured audit (you).** Read the target and boundary files. Extract formal contracts using this taxonomy:

Expand All @@ -53,29 +53,20 @@ Warn if older than 30 days — code may have changed.

Each contract records: Trigger, Input, Output, Condition, Ordering, Risk (LOW/MEDIUM/HIGH + reason), Evidence (file:line + snippet), Scope (method/class/module), Seam_Type (object/preprocessing/link/none), Pinch_Point (true/false), and a machine-verifiable grep or ast-grep assertion.

**Step 3 — Codex adversarial review.** Feed the contract list to the `codex` CLI with an adversarial brief: for each contract answer CONFIRM, DISPUTE (with reasoning), or ADD missing contracts. CONFIRM_RATIO = confirmed/total; healthy range is 30–70%. >70% means the review wasn't critical enough; <30% means the contracts need revision.
**Step 3 — Adversarial review.** Feed the contract list to the adversarial reviewer (`codex`, or subagent fallback) with an adversarial brief: for each contract answer CONFIRM, DISPUTE (with reasoning), or ADD missing contracts. CONFIRM_RATIO = confirmed/total; healthy range is 30–70% (unvalidated heuristic — treat as a reference value, not a gate). >70% means the review wasn't critical enough; <30% means the contracts need revision.

**Step 4 — Merge (you).** Resolve disputes, integrate additions, produce the final list plus CI rules.

**Methodology rules** (non-negotiable):
1. Every contract cites file:line evidence — no evidence, no contract.
2. A contract needs at least 2 of 3 LLMs agreeing to survive the merge.
2. A contract needs at least 2 of the 3 independent reviewers agreeing to survive the merge.
3. Every contract ships a runnable grep/ast-grep assertion (grep fallback when ast-grep is unavailable, e.g. Objective-C).
4. Codex DISPUTEs often reveal real issues — resolve them explicitly, don't discard.
4. Adversarial DISPUTEs often reveal real issues — resolve them explicitly, don't discard.
5. If no contracts are found, say so: the file may be a leaf module — suggest `/atlas.impact` instead.

## 4. Degraded mode
## 4. Reviewer fallback

If `gemini` or `codex` is missing, generate the prompts as files instead and tell the user how to run them manually:

```
.sourceatlas/audit/prompts/
├── step1-gemini.md # feed to Gemini
├── step2-claude.md # feed to Claude, include Gemini output
└── step3-codex.md # feed to Codex, include Claude output
```

Then: re-run `/atlas.audit <file> --force` and paste the outputs to merge. Still perform Step 2 yourself — Claude's structured audit is always available; mark the result `degraded: true`.
The pipeline never blocks on a missing CLI. Blind scan: `agy` unavailable/failing → spawn a fresh-context Claude subagent with the same blind prompt (no taxonomy, no draft). Adversarial: `codex` unavailable/failing → same, with the adversarial brief. The subagent must not see this session's reasoning — pass only the prompt and file paths. Record the substitution in `reviewers:` so readers know the vendor diversity of this run.

## 5. Report

Expand All @@ -93,14 +84,14 @@ Then YAML:
module: ...
language: ...
file: ...
degraded: false
reviewers: {blind: agy|claude-subagent, adversarial: codex|claude-subagent}
summary:
total_contracts: ...
by_category: {M: ..., L: ..., N: ..., S: ..., E: ..., C: ..., D: ..., P: ...}
cross_validation:
gemini_behaviors: ...
codex: {confirmed: ..., disputed: ..., added: ...}
confirm_ratio: ... # healthy 30–70%
blind_behaviors: ...
adversarial: {confirmed: ..., disputed: ..., added: ...}
confirm_ratio: ... # reference range 30–70% (unvalidated heuristic)
contracts:
- id: M-001 # unique, category prefix + 3 digits
title: ...
Expand Down
Loading