From f6bef86e8e993f51469d9b0bb5ce1b44e2ef473d Mon Sep 17 00:00:00 2001 From: Eliott Jacopin Date: Sun, 13 Sep 2026 13:01:33 +0200 Subject: [PATCH 1/9] docs(reports): record the index housekeeping architecture and the clear probe R01 (index_housekeeping) fixes the contracts before any code: the store root is derived from one `Index:` line, classification is pure and ordered, `index_prune` deletes store directories under a `project.json` guard, and the cycle runs on step commits. R02 records the probe that forced the deletion design: `colgrep clear` on a gone path exits 1 and leaves the index directory in place, so the retired hook's comment was right and a prune cannot delegate to the CLI. Co-Authored-By: Claude Fable 5.1 --- AGENTS.md | 2 + .../index_housekeeping/00-architecture_v0.md | 309 ++++++++++++++++++ .../00-findings_clear_probe_v0.md | 70 ++++ __reports__/index_housekeeping/README.md | 10 + 4 files changed, 391 insertions(+) create mode 100644 __reports__/index_housekeeping/00-architecture_v0.md create mode 100644 __reports__/index_housekeeping/00-findings_clear_probe_v0.md create mode 100644 __reports__/index_housekeeping/README.md diff --git a/AGENTS.md b/AGENTS.md index 85b0375..fce6ec6 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -85,6 +85,8 @@ disambiguates which one. | `R01` (dev_plugin) | `__reports__/dev_plugin/00-architecture_v0.md` | | `R01` (pypi_publication) | `__reports__/pypi_publication/00-architecture_v0.md` | | `R01` (harness_wiring) | `__reports__/harness_wiring/00-architecture_v0.md` | +| `R01` (index_housekeeping) | `__reports__/index_housekeeping/00-architecture_v0.md` | +| `R02` (index_housekeeping) | `__reports__/index_housekeeping/00-findings_clear_probe_v0.md` | ## Where the rest went diff --git a/__reports__/index_housekeeping/00-architecture_v0.md b/__reports__/index_housekeeping/00-architecture_v0.md new file mode 100644 index 0000000..5a3cf0c --- /dev/null +++ b/__reports__/index_housekeeping/00-architecture_v0.md @@ -0,0 +1,309 @@ +--- +type: architecture +topic: index_housekeeping +date: 2026-09-13 +version: v0 +prior-version: none +decision-required: confirm +--- + +# Index housekeeping for colgrep-mcp — Architecture Analysis (v0) + +Date: 2026-09-13. Report id `R01` (index_housekeeping) in the `AGENTS.md` legend. +Companion: `00-findings_clear_probe_v0.md` (`R02` (index_housekeeping)), the +probe this design rests on. + +## Executive Summary + +- **Problem.** colgrep's index store only grows. Measured on the maintainer's + machine on 2026-09-13 (macOS, colgrep 1.6.2): 165 indexed projects, 3.3 GiB + under the store; 65 indexes (0.92 GiB) name a `project_path` that no + longer exists (48 are agent worktrees); about 15 index machine state + (`/private/tmp` itself with 2 434 units, `~/.cache/uv/...`, + `~/.claude/plugins/cache/...`, per-session scratchpads); dozens are + *shadowed* — a subdirectory of another indexed project, double-indexing + its units because colgrep only folds into an ancestor that was indexed + first; 12 (0.06 GiB) are live but cold. `list_indexes` shows project, + model, units and search count only: no size, no age, no path-exists, no + shadowing. Cleaning up today costs an agent one `index_status` per guess + and one confirmed `index_clear` per project — dozens of calls. +- **Proposed change.** Read the store directly (`project.json` + `state.json` + per index directory, directory size and `state.json` mtime), classify + every index (`orphaned`, `machine_state`, `shadowed`, `cold`, live), and + expose that once: an enriched `list_indexes` / `colgrep://indexes` with a + `stale_only` filter, a new `index_prune` tool (dry run by default, + grouped candidates, `confirm=true` or elicitation to delete), a + `housekeeping` prompt, a `doctor` hint when the store carries orphans or + a machine-state root, and a skill section. Plus one `build(repo)` leaf: + exclude `(PR #N)` merge subjects from the changelog. +- **The probe that shaped the adapter.** `colgrep clear ` exits 1 + with `Error: No such file or directory (os error 2)` and leaves the index + directory in place (R02). Prune therefore deletes the index directory + under the store itself, guarded by the directory's own `project.json` + naming the path being pruned (C5). +- **Non-goals.** No per-session hook (hooks stay at bare-interpreter cost, + harness_wiring R01 §C3). No change to `search`, `expand`, `index_build`, + `index_clear`. No hard-coded platform store path (Windows differs). No + attempt to rebuild or migrate indexes. +- **Biggest risks.** Deleting a directory a live project depends on — closed + by the exact-`project.json` guard and by never issuing `colgrep clear` + for a prune (the folding rule cannot bite a direct directory delete). A + Windows path convention the classifier mis-reads — closed by the + `machine_state_roots` drift test against the hook and by Windows CI. +- **Validation.** `fake_colgrep.py` gains a `FAKE_COLGREP_STORE` knob so a + synthetic store under `tmp_path` drives `--stats` and `status`; pure + classification tests with injected home/roots; tool tests through the + in-memory client (dry run, confirm, elicitation accept/decline/failure, + guard); the surface dump-and-diff (`list_tools`/`list_resources`/ + `list_prompts` JSON before and after) cited in the PR body. + +## Current State + +```mermaid +graph TD + A((agent)) -->|list_indexes| T[tools_index.list_indexes] + T -->|colgrep --stats| CG[colgrep CLI] + CG --> ST[(store
indices/*/project.json
indices/*/state.json
indices/*/index/)] + T -->|project, model, units, searches| A + A -->|index_status per guess| T2[index_status] + A -->|index_clear per project, confirm| T3[index_clear] + T3 -->|colgrep clear path| CG + CG -. exit 1 when path is gone .-> T3 +``` + +Every fact an agent needs to decide "is this index worth keeping" — does the +path still exist, how big is it, when was it last touched, is it inside +another indexed project — is on disk in the store, and none of it reaches +the agent. + +## Proposed State + +```mermaid +graph TD + subgraph server["colgrep-mcp server"] + STORE[store.py
read_store: I/O
classify: pure] + LI[list_indexes
stale_only] + PR[index_prune
dry_run, classes, days, max_searches, confirm] + DR[doctor
hints: INDEX_STORE_STALE] + RES[colgrep://indexes] + HK[prompt housekeeping] + AD[adapter.stats / status
adapter.remove_index_dir] + end + CG[colgrep CLI] --> AD + AD -->|Index: line once| STORE + ST[(store dirs)] --> STORE + STORE --> LI & PR & DR & RES + LI & PR & DR & RES & HK --> A((agent)) + PR -->|rmtree index_dir
after project.json guard| ST +``` + +## Key Flows + +```mermaid +sequenceDiagram + participant A as agent + participant P as index_prune + participant S as store.py + participant CG as colgrep + participant FS as store dirs + A->>P: index_prune() (dry_run=true) + P->>CG: --stats + P->>CG: status + CG-->>P: Index: / → store root = parent + P->>S: read_store(root) → entries; classify(entries, stats, ...) + S-->>P: candidates grouped by class, sizes + P-->>A: text table + structured candidates, "call again with dry_run=false, confirm=true" + A->>P: index_prune(dry_run=false, confirm=true) + P->>P: recompute candidates (never trust a stale list) + loop each candidate + P->>FS: re-read /project.json == candidate.project? + P->>FS: rmtree() under the project lock + end + P-->>A: pruned[], failed[], bytes freed; notifies colgrep://indexes +``` + +## Contracts & Invariants + +### C1 — the store root is derived, never hard-coded + +`store_root(adapter)`: take `adapter.stats()`, call `adapter.status()` on the +first project whose path exists, and use the parent of its `Index:` line. +Cached on the adapter for the process lifetime (the store does not move +under a running server). Returns `None` when no indexed project exists on +disk — then `list_indexes` degrades to today's four fields and +`index_prune` reports `[INDEX_STORE_UNKNOWN]` rather than guess a platform +path. The `Index:` line is already parsed (`textparse.parse_status`); no +new text format enters the adapter. + +### C2 — one store entry per index directory + +`store.read_store(root) -> list[StoreEntry]` reads every direct child +directory holding a `project.json`: + +| Field | Source | +|:--|:--| +| `index_dir` | the child directory (absolute) | +| `project` | `project.json["project_path"]` | +| `model` | `project.json["model"]` | +| `files` | `len(state.json["files"])` | +| `search_count` | `state.json["search_count"]` | +| `size_bytes` | one `os.scandir` walk of the directory (15 ms for 165 indexes, measured) | +| `last_modified` | `state.json` mtime (colgrep rewrites it on every search), else the directory mtime | + +A child without `project.json` is skipped, never an error: the store is +colgrep's, not ours. `state.json` is read whole (25 ms for 165 on the same +machine) — the file-hash map is the bulk of it and is not kept. + +### C3 — classification is pure and ordered + +`store.classify(entries, *, now, home, machine_roots, days, max_searches)` +assigns exactly one class per entry, first match wins: + +| Class | Rule | +|:--|:--| +| `orphaned` | `project` does not exist on disk | +| `machine_state` | `project` is under a machine-state root (the system temp directory, `~/Library`, `~/AppData`) or has a dot-prefixed component under the home directory, **and** no ancestor up to the filesystem root carries a `.git` entry (Claude Code's own `.claude/worktrees/` are source corpora) | +| `shadowed` | `project` is a strict descendant of another entry's `project` that exists on disk; `shadowed_by` names it | +| `cold` | `search_count <= max_searches` and `last_modified` is at least `days` old | +| live | otherwise | + +The machine-state roots are the hook's `machine_state_roots(home)` +(`hooks/colgrep_policy.py`) restated in the server — the hook is stdlib-only +and ships outside the PyPI package, so it cannot be imported — and a drift +test asserts both functions return the same list for the same home. The +`.git` walk costs one `lstat` per path component and runs only for the +handful of machine-state candidates, never for the 150 live entries. + +### C4 — `list_indexes` grows, its block format is extended, its budget stays + +`IndexInfo` gains optional fields, all `None` when the store root is +unknown or the project is absent from the store: + +```text +IndexInfo: project, model, units_indexed, search_count, + + path_exists: bool | None, size_bytes: int | None, + last_modified: str | None (ISO 8601), shadowed_by: str | None, + stale: "orphaned" | "machine_state" | "shadowed" | None +``` + +`stale` carries the parameter-free classes only; `cold` needs `days` and +`max_searches` and lives in `index_prune`. `list_indexes(stale_only=false)` +filters on `stale is not None`. The text block appends +`size= modified=` and a trailing `[orphaned]` / +`[machine_state]` / `[shadowed by ]` tag; the header appends the +store total and the per-class counts. The renderer stays +`tools_search._render_budgeted` — the text budget landed in v0.2.0 +(`fix(index): cap list_indexes text at the text budget like every other +renderer`), so the "missing budget" the campaign brief carried over is +already closed; this cycle changes only what the blocks say. Client-visible +shape change, pinned by tests (`maintainer-policy` §Client-visible). + +`colgrep://indexes` serves the same enriched `IndexList`. + +### C5 — `index_prune` + +```text +index_prune(classes=["orphaned","machine_state","shadowed"], + days=30, max_searches=1, dry_run=true, confirm=false) + -> PruneResult(dry_run, store_root, candidates: [PruneCandidate], total_bytes, + pruned: [project], failed: [project], freed_bytes) +PruneCandidate: project, index_dir, class, size_bytes, last_modified, + search_count, shadowed_by +``` + +- `dry_run=true` (default) lists candidates grouped by class with sizes and + ends with the exact next call. `cold` is opt-in through `classes`. +- `dry_run=false` recomputes the candidates (a list from an earlier call is + advice, not a contract), then deletes only with `confirm=true` or an + accepted elicitation — the `index_clear` flow verbatim: no elicitation + capability → `[CONFIRMATION_REQUIRED]`; the elicitation call itself + failing → `[CONFIRMATION_REQUIRED]`; declined → `pruned=[]`, text + "Not pruned (declined)". +- Deletion is `adapter.remove_index_dir(index_dir, project)`: the directory + must be a direct child of the store root, and its `project.json` must + name `project` at the moment of deletion; otherwise the candidate lands + in `failed` and nothing is removed. Never `colgrep clear`: for a shadowed + or a gone path that command either fails (R02) or clears the ancestor + project that folded the path (colgrep_mcp R05 D3) — the exact failure + `index_clear`'s `PROJECT_ROOT_MISMATCH` exists to prevent. Each removal + runs under `project_lock(project)`. +- After any deletion, `colgrep://indexes` is notified once. + +### C6 — `doctor` hints + +`Doctor` gains `hints: list[str]` (the `problems` list keeps meaning "not +ok"). When the store root is known and the classification finds orphaned or +machine-state entries, one hint is appended: +`[INDEX_STORE_STALE] orphaned, machine-state indexes () …`. +`Code.INDEX_STORE_STALE` and `Code.INDEX_STORE_UNKNOWN` join `errors.HINTS` +and so `colgrep://errors`. `doctor` already spawns `--version` and +`settings`; this adds `--stats`, one `status` and the store read (~40 ms). + +### C7 — `housekeeping` prompt and skill section + +`housekeeping(days="30")` beside `explore`/`locate`/`impact`: (1) +`list_indexes(stale_only=true)`; (2) `index_prune()` dry run, review the +grouped table, decide whether `cold` belongs in `classes`; (3) +`index_prune(dry_run=false, confirm=true, classes=[...])`; (4) re-run +`list_indexes` to confirm. The prompt has no `path` argument, so +`complete_path` is untouched. `skills/colgrep-search/SKILL.md` gains a +decision-table row and a "housekeeping" workflow; `guide.md` gains the tool +and the classes; both READMEs' Tools tables gain the row (`test_readme`). + +### C8 — changelog excludes merge subjects + +`changelog_pattern` becomes +`^(?!.*\(PR #\d+\)$)(feat|fix|perf|BREAKING CHANGE)(\(.+\))?(!)?`: +commitizen applies it with `re.match` to the commit subject +(`changelog.generate_tree_from_commits`), so a negative lookahead anchored at +the start excludes a subject ending in `(PR #N)`. Oracle: `cz changelog +--dry-run` for a released version before and after; the diff must remove +exactly the duplicated `(PR #N)` lines (`maintainer-policy` §Drift tests). +`test_changelog.py` stays green (headings untouched). + +### Error model + +| Situation | Surface | +|:--|:--| +| no indexed project exists on disk | `list_indexes`: four legacy fields, others `None`; `index_prune`, `doctor`: `[INDEX_STORE_UNKNOWN]` (prune raises it, doctor lists it as a hint) | +| a candidate's `project.json` changed or vanished before deletion | `failed` entry, nothing removed, tool succeeds | +| `rmtree` fails (permissions, a file held open) | `failed` entry with the OS error text, other candidates proceed | +| `dry_run=false` without confirmation | `[CONFIRMATION_REQUIRED]`, as `index_clear` | + +## Alternatives Considered + +| Decision | Options | Chosen | Why | +|:--|:--|:--|:--| +| D1 How prune removes an index | (a) `colgrep clear `; (b) delete the index directory under the store | (b) | R02: `clear` exits 1 on a gone path and leaves the directory; for a shadowed path it would clear the folding ancestor. (b) removes exactly the directory whose `project.json` names the candidate. | +| D2 Store root | (a) platform default path per OS; (b) an env var; (c) parent of the `Index:` line of one `status` call | (c) | No platform table to keep right for Windows; no new configuration surface; the parser already exists. Degrades explicitly when no indexed project exists on disk. | +| D3 Where classification lives | (a) inside `tools_index.py`; (b) a new `store.py` with I/O (`read_store`) and pure logic (`classify`) apart | (b) | Mirrors the `adapter.py` / `textparse.py` split: a classification bug is never mistaken for an I/O bug, and the pure half is tested with injected home/roots so Windows CI's temp-under-home layout cannot flip a result. | +| D4 Machine-state convention | (a) import the hook; (b) restate the roots in the server and pin with a drift test; (c) move the hook into the package | (b) | (a) impossible across distributions; (c) rejected already for hook latency (harness_wiring D2). | +| D5 Sizes | (a) eager `scandir` per index on every call; (b) lazy / opt-in | (a) | Measured 15 ms for 165 indexes; size is the field the change exists for. | +| D6 Age source | (a) index directory mtime; (b) `state.json` mtime; (c) `project.json` mtime | (b) | colgrep rewrites `state.json` on every search (`search_count`), so it is the last-use time; the directory mtime coincides in practice and is the fallback. | +| D7 Class exposure in `list_indexes` | (a) four booleans; (b) one `stale` field with the parameter-free classes | (b) | One field to filter on (`stale_only`); `cold` needs parameters and belongs to `index_prune`. | +| D8 Prune defaults | (a) all four classes; (b) `cold` opt-in | (b) | Orphaned, machine-state and shadowed indexes are dead weight by construction; a cold live project is a judgement the agent should make with the numbers in front of it. | +| D9 Hook or doctor | (a) a `SessionStart` hook that warns about the store; (b) a `doctor` hint | (b) | Hooks must stay at bare-interpreter cost and carry the rule only (harness_wiring R01 §C3, §C4); `doctor` is the self-check an agent runs when something looks off. | +| D10 Roadmap tree | (a) `dirtree-rdm`; (b) step commits | (b) | Single implementer, every leaf lead-sized, precedent `pypi_publication` and `harness_wiring` D9. | +| D11 Text budget for `list_indexes` | (a) add it; (b) nothing | (b) | Already landed in v0.2.0; the brief's premise was stale. Recorded here so the next cycle does not carry it again. | + +## Risks & Mitigations + +| # | Risk | Likelihood | Mitigation | +|:--|:--|:--|:--| +| 1 | Prune deletes a directory a live project uses | low | C5 guard: direct child of the store root, `project.json` equals the candidate at deletion time; `shadowed` deletes the descendant, never the ancestor; `colgrep clear` never used by prune. | +| 2 | Windows: the runner's temp directory sits under the home directory, so a `tmp_path` project reads as machine state | certain in CI | Pure tests inject `home`/`machine_roots`; tool tests use `orphaned` (path-independent) or monkeypatch `store.machine_state_roots`; Windows CI verdict read in full (`campaign-lead`). | +| 3 | No indexed project exists on disk, so the store root cannot be derived | low | Explicit `[INDEX_STORE_UNKNOWN]`; `list_indexes` still answers with the legacy fields. | +| 4 | The enriched block breaks a client that parsed the old line | low | Fields are appended, the first four tokens are unchanged; pinned by the updated format test. | +| 5 | `state.json` grows large on big projects (file-hash map), so the read pass is not free | low | 25 ms for 165 on the maintainer's machine; only `files` length and `search_count` are kept. | +| 6 | `changelog_pattern` drops a genuine entry | low | Dry-run diff before/after shows only `(PR #N)` lines removed. | + +## Roadmap Recommendation + +No roadmap tree (D10). Step commits, one concern each, on this branch: + +1. `docs(reports): record the index housekeeping architecture and the clear probe` +2. `feat(index): read the colgrep store and enrich list_indexes with size, age, path and shadowing` — `store.py`, models, adapter store-root cache, `fake_colgrep.py` knob, tests, resource. +3. `feat(index): add index_prune to remove orphaned, machine-state, shadowed and cold indexes` — tool, adapter `remove_index_dir`, error codes, tests. +4. `feat(prompts): add the housekeeping prompt and a doctor hint for a stale index store` +5. `docs(skill): teach the housekeeping workflow in the search skill, guide and READMEs` +6. `build(repo): exclude merge subjects ending in (PR #N) from the changelog` diff --git a/__reports__/index_housekeeping/00-findings_clear_probe_v0.md b/__reports__/index_housekeeping/00-findings_clear_probe_v0.md new file mode 100644 index 0000000..80c0368 --- /dev/null +++ b/__reports__/index_housekeeping/00-findings_clear_probe_v0.md @@ -0,0 +1,70 @@ +--- +type: findings +topic: index_housekeeping +date: 2026-09-13 +version: v0 +prior-version: none +--- + +# Does `colgrep clear` remove the index of a path that no longer exists? (v0) + +Report id `R02` (index_housekeeping). Probe run on 2026-09-13, macOS, colgrep +1.6.2, on one orphaned index whose `project_path` had no indexed ancestor +(so a folding clear could not have reached another project either way). + +## Result + +**No.** `colgrep clear ` exits 1 without touching the store. Prune +must remove the index directory itself (R01 D1, C5). The retired machine +hook's comment ("clear merely errors when the directory no longer exists") +is confirmed. + +## Method + +1. Survey the store read-only (`project.json` per index directory; a path is + orphaned when `os.path.exists(project_path)` is false): 165 indexes, 65 + orphaned. +2. Pick the smallest orphan with **no indexed ancestor** — the orphan first + chosen (`__canons__` under a removed worktree) was rejected because its + parent worktree was itself indexed, which would have made the probe + ambiguous. +3. Run `status` then `clear` on the gone path; check the directory after. + +## Transcript + +Paths are the maintainer's, shown with the `/Users/me` placeholder. + +```text +$ colgrep status "/Users/me/src/explore/colgrep_mcp/server/colgrep_mcp" --color never +Error: No such file or directory (os error 2) +exit=1 + +$ colgrep clear "/Users/me/src/explore/colgrep_mcp/server/colgrep_mcp" --color never +Error: No such file or directory (os error 2) +exit=1 + +$ ls "~/Library/Application Support/colgrep/indices/colgrep_mcp-a1887fc5" +index project.json state.json +$ ls "~/Library/Application Support/colgrep/indices" | wc -l +165 +``` + +## Store facts confirmed on the way + +| Fact | Value | +|:--|:--| +| Index directory layout | `/-<8 hex>/{project.json, state.json, index/, .lock}` | +| `project.json` | `{"project_path", "project_name", "model"}` — no timestamps | +| `state.json` | `{"cli_version", "index_format_version", "files": {rel: {content_hash, mtime, size}}, "ignored_files", "search_count", "dirty"}` | +| `state.json` mtime vs directory mtime | identical to the second on every sampled index, including the three searched during this session — `state.json` is rewritten on each search | +| Size pass (`os.scandir`, all 165) | 15 ms median of 5 | +| `project.json` + `state.json` read pass | 25 ms | +| Indexed pairs where one project is a strict descendant of another | 130 | +| Orphaned indexes | 65 of 165 | + +## Steering + +- Prune deletes `/` directly, guarded by that directory's own + `project.json` (R01 C5). Confirmed necessary, not just preferable. +- `status` also fails on a gone path, so the store root must be derived from + a `status` call on a project that *exists* (R01 C1). diff --git a/__reports__/index_housekeeping/README.md b/__reports__/index_housekeeping/README.md new file mode 100644 index 0000000..94e8822 --- /dev/null +++ b/__reports__/index_housekeeping/README.md @@ -0,0 +1,10 @@ +# index_housekeeping — reports + +Seventh campaign of this repository (2026-09-13, after `harness_wiring` 0.4.0): give agents the facts and the one tool needed to keep colgrep's index store lean — size, age, path-exists and shadowing per index, an `index_prune` tool with a dry run and grouped candidates, a `housekeeping` prompt, a `doctor` hint — plus the changelog fix carried from the previous cycle. Single-agent, no roadmap tree: step commits on `claude/dreamy-austin-2682cd`. + +## Round 00 +- `00-architecture_v0.md` — R01 (index_housekeeping): store discovery from the `Index:` line (C1), the store entry model (C2), the ordered classification (C3), the enriched `list_indexes` (C4), `index_prune` and its deletion guard (C5), the `doctor` hint (C6), the prompt and skill (C7), the changelog pattern (C8); decisions D1–D11; risk register; step-commit plan. +- `00-findings_clear_probe_v0.md` — R02 (index_housekeeping): `colgrep clear` on a gone path exits 1 and leaves the index directory; store layout, mtime and cost facts measured on the maintainer's machine. + +## Status +Open. Time-boxed to four hours from 12:54 CEST on 2026-09-13. From a94a0b9347bde28db9c4e729d0f3b77b2f77713f Mon Sep 17 00:00:00 2001 From: Eliott Jacopin Date: Sun, 13 Sep 2026 13:07:40 +0200 Subject: [PATCH 2/9] feat(index): read the colgrep store and enrich list_indexes with size, age, path and shadowing MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit An agent deciding which of 165 indexes to keep needed one index_status per guess: the facts — does the path exist, how big is the index, when was it last used, does another indexed project already cover it — sit in each index directory's project.json and state.json and never reached the client (index_housekeeping R01 §C2–C4). The new store module reads the store once (40 ms for 165 indexes) and classifies every entry; the store root is the parent of one `Index:` line rather than a platform path (R01 §C1, R02: `status` fails on a gone path, so an existing project is asked). The machine-state roots restate the hook's and a drift test pins them equal (R01 D4). `list_indexes(stale_only=...)`, `IndexInfo` and `colgrep://indexes` carry the new fields; the four legacy tokens stay at the front of each text block. Co-Authored-By: Claude Fable 5.1 --- server/colgrep_mcp/adapter.py | 30 ++++ server/colgrep_mcp/models.py | 16 ++ server/colgrep_mcp/resources.py | 10 +- server/colgrep_mcp/store.py | 265 ++++++++++++++++++++++++++++++ server/colgrep_mcp/tools_index.py | 67 +++++++- server/tests/conftest.py | 48 ++++++ server/tests/fake_colgrep.py | 45 +++++ server/tests/test_resources.py | 20 +++ server/tests/test_store.py | 171 +++++++++++++++++++ server/tests/test_tools_index.py | 81 +++++++++ 10 files changed, 740 insertions(+), 13 deletions(-) create mode 100644 server/colgrep_mcp/store.py create mode 100644 server/tests/test_store.py diff --git a/server/colgrep_mcp/adapter.py b/server/colgrep_mcp/adapter.py index f54724f..f74ec44 100644 --- a/server/colgrep_mcp/adapter.py +++ b/server/colgrep_mcp/adapter.py @@ -117,6 +117,9 @@ def __init__( # Set by `_run` after every spawn; for tests only (proving no zombie # process survives a `ColgrepTimeout`), never read by production code. self._last_proc: asyncio.subprocess.Process | None = None + # `store_root()`'s cache: the answer may legitimately be `None`, hence the flag. + self._store_root: Path | None = None + self._store_root_cached = False def with_stderr(self, on_stderr: StderrCallback | None) -> ColgrepAdapter: """A shallow copy sharing `binary`/`timeout_s` but a different `on_stderr`. @@ -280,6 +283,33 @@ async def stats(self) -> list[IndexInfo]: stdout, _stderr, _rc = await self._run(["--stats"]) return parse_stats(stdout) + async def store_root(self, stats: list[IndexInfo] | None = None) -> Path | None: + """The index store directory: the parent of the `Index:` line `status` prints for + the first indexed project that still exists on disk (index_housekeeping R01 §C1). + + Derived, never hard-coded (the platform path differs on Windows), and + cached for the adapter's lifetime: the store does not move under a + running server. `stats` may be passed by a caller that already + fetched it, saving the spawn. `None` when no indexed project exists + on disk — `status` fails on a gone path (R02), so nothing can be + asked for its `Index:` line. + """ + if self._store_root_cached: + return self._store_root + infos = await self.stats() if stats is None else stats + for info in infos: + if not os.path.isdir(info.project): + continue + try: + st = await self.status(Path(info.project)) + except ColgrepError: + continue + if st.index_path: + self._store_root = Path(st.index_path).parent + break + self._store_root_cached = True + return self._store_root + async def settings(self) -> dict[str, str]: stdout, _stderr, _rc = await self._run(["settings"]) return parse_settings(stdout) diff --git a/server/colgrep_mcp/models.py b/server/colgrep_mcp/models.py index b60a4dc..c7a6758 100644 --- a/server/colgrep_mcp/models.py +++ b/server/colgrep_mcp/models.py @@ -88,10 +88,26 @@ class IndexInfo(BaseModel): model: str units_indexed: int search_count: int + # Store-derived fields (index_housekeeping R01 §C4); all `None` when the + # store root could not be derived or the project is absent from the store. + path_exists: bool | None = Field(default=None, description="Whether `project` still exists on disk.") + size_bytes: int | None = Field(default=None, description="Bytes under the index directory.") + last_modified: str | None = Field(default=None, description="ISO 8601 UTC time of the last search or update.") + shadowed_by: str | None = Field( + default=None, description="An indexed, existing ancestor project that double-indexes this one's units." + ) + stale: str | None = Field( + default=None, + description="`orphaned` (path gone), `machine_state` (temp, cache or hidden tree) or `shadowed`; " + "`None` for a live project. `cold` needs parameters and is reported by `index_prune` only.", + ) class IndexList(BaseModel): indexes: list[IndexInfo] + store_root: str | None = Field(default=None, description="The index store directory, when it could be derived.") + total_bytes: int | None = Field(default=None, description="Bytes under the whole store.") + total: int | None = Field(default=None, description="Indexed projects before any `stale_only` filter.") class IndexBuildResult(BaseModel): diff --git a/server/colgrep_mcp/resources.py b/server/colgrep_mcp/resources.py index 563ecbf..265b5cd 100644 --- a/server/colgrep_mcp/resources.py +++ b/server/colgrep_mcp/resources.py @@ -38,8 +38,8 @@ from .adapter import ColgrepError from .errors import HINTS, Code, from_adapter_error -from .models import IndexList from .server import get_adapter +from .store import index_list def _map_adapter_error(exc: ColgrepError) -> ResourceError: @@ -96,13 +96,15 @@ async def settings_resource() -> dict[str, str]: async def indexes_resource() -> dict[str, Any]: - """`colgrep://indexes` — every indexed project on this machine (`IndexList`).""" + """`colgrep://indexes` — every indexed project on this machine (`IndexList`), with the + store-derived size, age, path-exists and shadowing fields `list_indexes` carries + (index_housekeeping R01 §C4).""" adapter = get_adapter() try: - infos = await adapter.stats() + result = await index_list(adapter) except ColgrepError as exc: raise _map_adapter_error(exc) from exc - return IndexList(indexes=infos).model_dump() + return result.model_dump() async def status_resource(path: str, ctx: Context) -> dict[str, Any]: diff --git a/server/colgrep_mcp/store.py b/server/colgrep_mcp/store.py new file mode 100644 index 0000000..472402f --- /dev/null +++ b/server/colgrep_mcp/store.py @@ -0,0 +1,265 @@ +"""colgrep's index store on disk, read and classified (index_housekeeping R01 §C1–C3). + +The only module that knows the store's layout: one child directory per +index holding `project.json` and `state.json` (R02). `read_store` is the +I/O half (one `scandir` walk and two JSON reads per index, 40 ms for 165 +indexes on the maintainer's machine); `classify` is the pure half, given +`now`, `home` and the machine-state roots explicitly so a test can pin a +verdict on any OS — the CI runners' temp directory sits under the home +directory on Windows and under a hidden tree on macOS, so an implicit +`Path.home()` would flip results between jobs (R01 risk 2). `index_list` +joins both with `colgrep --stats` for the tools and the resource. +""" + +from __future__ import annotations + +import asyncio +import json +import os +import tempfile +import time +from dataclasses import dataclass +from datetime import UTC, datetime +from pathlib import Path + +from .adapter import ColgrepAdapter +from .models import IndexInfo, IndexList + +ORPHANED = "orphaned" +MACHINE_STATE = "machine_state" +SHADOWED = "shadowed" +COLD = "cold" + +#: Classes that need no parameter; `list_indexes` reports them as `stale`. +STALE_CLASSES: tuple[str, ...] = (ORPHANED, MACHINE_STATE, SHADOWED) +#: Every class `index_prune` accepts; `cold` needs `days` and `max_searches`. +PRUNE_CLASSES: tuple[str, ...] = (*STALE_CLASSES, COLD) + + +@dataclass(frozen=True) +class StoreEntry: + """One index directory as colgrep left it on disk (R01 §C2).""" + + index_dir: Path + project: str + model: str + files: int | None + search_count: int | None + size_bytes: int + last_modified: float + + +@dataclass(frozen=True) +class Classified: + """A `StoreEntry` with its verdict (R01 §C3): `kind` is a class name or `None` for live.""" + + entry: StoreEntry + kind: str | None + path_exists: bool + shadowed_by: str | None + + +# --- I/O ------------------------------------------------------------------------ + + +def _dir_size(path: Path) -> int: + """Bytes under `path`: one `scandir` walk, no `Path.stat()` per file (`maintainer-policy` §Hardware-first).""" + total = 0 + stack = [str(path)] + while stack: + try: + with os.scandir(stack.pop()) as it: + for entry in it: + if entry.is_dir(follow_symlinks=False): + stack.append(entry.path) + elif entry.is_file(follow_symlinks=False): + total += entry.stat(follow_symlinks=False).st_size + except OSError: + continue + return total + + +def _read_entry(index_dir: Path) -> StoreEntry | None: + try: + project = json.loads((index_dir / "project.json").read_text(encoding="utf-8")) + except (OSError, ValueError): + return None + project_path = project.get("project_path") + if not isinstance(project_path, str) or not project_path: + return None + files: int | None = None + search_count: int | None = None + last_modified = index_dir.stat().st_mtime + state_file = index_dir / "state.json" + try: + # `state.json` is rewritten on every search (R02), so its mtime is the + # last-use time; the directory mtime coincides and is the fallback. + last_modified = state_file.stat().st_mtime + state = json.loads(state_file.read_text(encoding="utf-8")) + state_files = state.get("files") + files = len(state_files) if isinstance(state_files, dict) else None + sc = state.get("search_count") + search_count = sc if isinstance(sc, int) else None + except (OSError, ValueError): + pass + return StoreEntry( + index_dir=index_dir, + project=project_path, + model=str(project.get("model", "")), + files=files, + search_count=search_count, + size_bytes=_dir_size(index_dir), + last_modified=last_modified, + ) + + +def read_store(root: Path) -> list[StoreEntry]: + """Every index directory under `root` that carries a `project.json`; others are skipped, never an error.""" + entries: list[StoreEntry] = [] + try: + children = sorted(p for p in root.iterdir() if p.is_dir()) + except OSError: + return entries + for child in children: + entry = _read_entry(child) + if entry is not None: + entries.append(entry) + return entries + + +# --- classification (pure) ------------------------------------------------------- + + +def machine_state_roots(home: str) -> list[str]: + """The hook's `machine_state_roots` (`hooks/colgrep_policy.py`), restated: the hook + is stdlib-only and ships outside this package, so it cannot be imported; a drift + test pins the two lists equal (R01 D4).""" + return [ + os.path.realpath(tempfile.gettempdir()), + os.path.join(home, "Library"), + os.path.join(home, "AppData"), + ] + + +def _under(path: str, root: str) -> bool: + return path == root or path.startswith(root.rstrip(os.sep) + os.sep) + + +def _in_work_tree(path: str) -> bool: + """Whether any ancestor of `path` (itself included) carries a `.git` entry. + + One `lstat` per path component, no `git` subprocess: it runs only for the + few machine-state candidates, and it keeps Claude Code's own + `.claude/worktrees/` (a real work tree under a hidden directory) + out of the machine-state class. + """ + p = Path(path) + for candidate in (p, *p.parents): + try: + if (candidate / ".git").exists(): + return True + except OSError: + return False + return False + + +def is_machine_state(path: str, *, home: str, roots: list[str]) -> bool: + """Application state, not authored material: the hook's `is_source_corpus` inverted for + an indexed path (R01 §C3), minus the `git` subprocess.""" + if any(_under(path, root) for root in roots): + hidden_or_state = True + elif _under(path, home) and path != home: + inner = path[len(home) :] + hidden_or_state = any(part.startswith(".") for part in inner.split(os.sep) if part) + else: + return False + return hidden_or_state and not _in_work_tree(path) + + +def shadowing_ancestor(project: str, live_projects: set[str]) -> str | None: + """The nearest indexed, existing ancestor project of `project`, or `None`.""" + for parent in Path(project).parents: + if str(parent) in live_projects: + return str(parent) + return None + + +def classify( + entries: list[StoreEntry], + *, + now: float, + home: str, + machine_roots: list[str], + days: int = 30, + max_searches: int = 1, +) -> list[Classified]: + """One class per entry, first rule wins: orphaned, machine_state, shadowed, cold, live (R01 §C3).""" + exists = {e.project: os.path.exists(e.project) for e in entries} + live_projects = {p for p, ok in exists.items() if ok} + cutoff = now - days * 86400 + out: list[Classified] = [] + for e in entries: + path_exists = exists[e.project] + shadowed_by = shadowing_ancestor(e.project, live_projects) if path_exists else None + if not path_exists: + kind: str | None = ORPHANED + elif is_machine_state(e.project, home=home, roots=machine_roots): + kind = MACHINE_STATE + elif shadowed_by is not None: + kind = SHADOWED + elif (e.search_count or 0) <= max_searches and e.last_modified <= cutoff: + kind = COLD + else: + kind = None + out.append(Classified(entry=e, kind=kind, path_exists=path_exists, shadowed_by=shadowed_by)) + return out + + +def classify_now(entries: list[StoreEntry], *, days: int = 30, max_searches: int = 1) -> list[Classified]: + """`classify` against this machine: the real clock, home directory and machine-state roots.""" + home = os.path.realpath(os.path.expanduser("~")) + roots = machine_state_roots(home) + return classify(entries, now=time.time(), home=home, machine_roots=roots, days=days, max_searches=max_searches) + + +def iso_utc(ts: float) -> str: + return datetime.fromtimestamp(ts, tz=UTC).isoformat(timespec="seconds") + + +# --- the joined view -------------------------------------------------------------- + + +def _enrich(info: IndexInfo, verdict: Classified | None) -> IndexInfo: + if verdict is None: + return info + return info.model_copy( + update={ + "path_exists": verdict.path_exists, + "size_bytes": verdict.entry.size_bytes, + "last_modified": iso_utc(verdict.entry.last_modified), + "shadowed_by": verdict.shadowed_by, + "stale": verdict.kind if verdict.kind in STALE_CLASSES else None, + } + ) + + +async def index_list(adapter: ColgrepAdapter, *, stale_only: bool = False) -> IndexList: + """`colgrep --stats` joined with the store by project path (R01 §C4). + + The store read runs in a worker thread: 40 ms of blocking I/O on a + 165-index store must not stall other tool calls on the event loop. When + the store root cannot be derived (no indexed project exists on disk, R01 + §C1) the four legacy fields are all a caller gets. + """ + infos = await adapter.stats() + root = await adapter.store_root(infos) + total_bytes: int | None = None + if root is not None: + entries = await asyncio.to_thread(read_store, root) + by_project = {c.entry.project: c for c in classify_now(entries)} + infos = [_enrich(info, by_project.get(info.project)) for info in infos] + total_bytes = sum(e.size_bytes for e in entries) + total = len(infos) + if stale_only: + infos = [info for info in infos if info.stale is not None] + return IndexList(indexes=infos, store_root=str(root) if root else None, total_bytes=total_bytes, total=total) diff --git a/server/colgrep_mcp/tools_index.py b/server/colgrep_mcp/tools_index.py index 1854c7a..696b065 100644 --- a/server/colgrep_mcp/tools_index.py +++ b/server/colgrep_mcp/tools_index.py @@ -16,7 +16,7 @@ from mcp.types import CallToolResult, ClientCapabilities, ElicitationCapability, TextContent, ToolAnnotations from pydantic import BaseModel, Field -from . import tools_search +from . import store, tools_search from .adapter import ColgrepError, ColgrepNotFound from .errors import Code, tool_error, translate_adapter_errors from .locks import project_lock @@ -76,11 +76,47 @@ def _render_status(status: IndexStatus) -> str: return "\n".join(lines) +def _mib(size_bytes: int) -> str: + return f"{size_bytes / 2**20:.1f}MiB" + + +def _human_size(size_bytes: int) -> str: + return f"{size_bytes / 2**30:.2f}GiB" if size_bytes >= 2**30 else _mib(size_bytes) + + def _index_block(info: IndexInfo) -> str: - return f"{info.project} model={info.model} units={info.units_indexed} searches={info.search_count}" + """The pre-0.5 four tokens first, then the store-derived ones (index_housekeeping + R01 §C4): a client that read the old line still finds it at the front.""" + text = f"{info.project} model={info.model} units={info.units_indexed} searches={info.search_count}" + if info.size_bytes is not None: + text += f" size={_mib(info.size_bytes)}" + if info.last_modified: + text += f" modified={info.last_modified[:10]}" + if info.stale == store.SHADOWED: + text += f" [shadowed by {info.shadowed_by}]" + elif info.stale: + text += f" [{info.stale}]" + return text + + +def _index_header(result: IndexList, stale_only: bool) -> str: + shown = len(result.indexes) + total = result.total if result.total is not None else shown + header = ( + f"{shown} stale of {total} indexed projects on this machine" + if stale_only + else f"{shown} indexed projects on this machine" + ) + if result.total_bytes is None: + return header + counts = {kind: sum(1 for i in result.indexes if i.stale == kind) for kind in store.STALE_CLASSES} + return ( + f"{header} ({_human_size(result.total_bytes)} in store; {counts[store.ORPHANED]} orphaned, " + f"{counts[store.MACHINE_STATE]} machine-state, {counts[store.SHADOWED]} shadowed)" + ) -def _render_index_list(result: IndexList, budget: int) -> tuple[str, bool]: +def _render_index_list(result: IndexList, budget: int, *, stale_only: bool = False) -> tuple[str, bool]: """Render `result` through the one budgeted renderer (R01 §C7), capped at `budget` chars like every other tool's text listing (R01 consistency §Token-budget invariant); `list_indexes`' text is machine-global and was @@ -91,8 +127,10 @@ def _render_index_list(result: IndexList, budget: int) -> tuple[str, bool]: to list) — kept byte-identical to the pre-budget rendering. """ if not result.indexes: + if stale_only and result.total: + return f"No stale indexes among the {result.total} indexed projects on this machine.", False return "No indexed projects on this machine.", False - header = f"{len(result.indexes)} indexed projects on this machine" + header = _index_header(result, stale_only) blocks = [_index_block(info) for info in result.indexes] return tools_search._render_budgeted( header, blocks, lambda remaining: f"[{remaining} more indexes in structured_content]", [], budget @@ -158,15 +196,26 @@ async def index_status( ) -async def list_indexes(*, ctx: Context) -> CallToolResult: - """List every project colgrep has indexed on this machine, with model and unit counts.""" +async def list_indexes( + stale_only: Annotated[ + bool, + Field( + description="Only indexes whose project path is gone (`orphaned`), sits in a temp, cache or hidden " + "tree (`machine_state`), or lies inside another indexed project (`shadowed`)." + ), + ] = False, + *, + ctx: Context, +) -> CallToolResult: + """List every project colgrep has indexed on this machine: model, units, searches, index size, + last use, whether the path still exists and whether another indexed project shadows it. + Use `index_prune` to remove the stale ones.""" adapter = get_adapter(ctx) settings = get_settings(ctx) async with translate_adapter_errors(): - infos: list[IndexInfo] = await adapter.stats() + result = await store.index_list(adapter, stale_only=stale_only) - result = IndexList(indexes=infos) - text, capped = _render_index_list(result, settings.text_budget) + text, capped = _render_index_list(result, settings.text_budget, stale_only=stale_only) if capped: await safe_log( ctx, diff --git a/server/tests/conftest.py b/server/tests/conftest.py index e26435d..f3968e6 100644 --- a/server/tests/conftest.py +++ b/server/tests/conftest.py @@ -59,3 +59,51 @@ def settings_env(monkeypatch, fake_colgrep_bin, tmp_path) -> dict[str, str]: for k, v in env.items(): monkeypatch.setenv(k, v) return env + + +@pytest.fixture +def fake_store(monkeypatch, tmp_path): + """A synthetic colgrep index store under `tmp_path`, wired to the fake binary. + + Returns `add(name, project, *, search_count=0, files=3, age_days=0, model=...)`, + which writes `//{project.json,state.json,index/}` the way colgrep + lays them out (index_housekeeping R02) and back-dates `state.json` by + `age_days`. `project` is any path string: pass one that exists for a live + project, one that does not for an orphan. + """ + import json + import os + import time + + store = tmp_path / "indices" + store.mkdir() + monkeypatch.setenv("FAKE_COLGREP_STORE", str(store)) + + def add(name, project, *, search_count=0, files=3, age_days=0, model="lightonai/LateOn-Code-edge"): + d = store / name + (d / "index").mkdir(parents=True) + (d / "index" / "blob").write_bytes(b"x" * 1024) + (d / "project.json").write_text( + json.dumps({"project_path": str(project), "project_name": name, "model": model}), encoding="utf-8" + ) + state = d / "state.json" + state.write_text( + json.dumps( + { + "cli_version": "1.6.2", + "index_format_version": 2, + "files": {f"f{i}.py": {"content_hash": i, "mtime": 0, "size": 1} for i in range(files)}, + "ignored_files": {}, + "search_count": search_count, + "dirty": False, + } + ), + encoding="utf-8", + ) + if age_days: + then = time.time() - age_days * 86400 + os.utime(state, (then, then)) + return d + + add.root = store + return add diff --git a/server/tests/fake_colgrep.py b/server/tests/fake_colgrep.py index 058c257..8b6fddb 100755 --- a/server/tests/fake_colgrep.py +++ b/server/tests/fake_colgrep.py @@ -21,6 +21,12 @@ `Project:` line instead of echoing the requested path — simulates colgrep folding a path into an already-registered ancestor project (R05 D3) + FAKE_COLGREP_STORE a directory laid out like colgrep's index store + (`/project.json` + `state.json`, R02): `--stats` + lists its projects (units = files in state.json), + `status ` names `Index: /` for a + project it holds and exits 1 like the real binary + when the path does not exist on disk (R02) """ import json @@ -35,6 +41,24 @@ MODEL = "lightonai/LateOn-Code-edge" +def _store_entries(store): + """`(name, project.json, state.json)` per index directory, in name order.""" + out = [] + for name in sorted(os.listdir(store)): + pj = os.path.join(store, name, "project.json") + if not os.path.isfile(pj): + continue + with open(pj, encoding="utf-8") as f: + project = json.load(f) + state = {} + sj = os.path.join(store, name, "state.json") + if os.path.isfile(sj): + with open(sj, encoding="utf-8") as f: + state = json.load(f) + out.append((name, project, state)) + return out + + def main(argv): # Force UTF-8 with bare `\n` line endings regardless of platform. Python's # default text-mode stdout/stderr on Windows (a) encodes with the @@ -64,7 +88,16 @@ def main(argv): if "--version" in args: print("colgrep 1.6.2") return 0 + store = os.environ.get("FAKE_COLGREP_STORE") if "--stats" in args: + if store: + for _name, project, state in _store_entries(store): + files = state.get("files") or {} + print( + f"Project: {project['project_path']}\n Model: {project.get('model', MODEL)}\n" + f" Functions indexed: {len(files)}\n Search count: {state.get('search_count', 0)}\n" + ) + return 0 print(f"Project: /tmp/fake-corpus\n Model: {MODEL}\n Functions indexed: 3\n Search count: 7\n") print(f"Project: /tmp/other\n Model: {MODEL}\n Functions indexed: 649\n Search count: 1\n") return 0 @@ -78,6 +111,18 @@ def main(argv): return 0 if sub == "status": path = next((a for a in args[1:] if not a.startswith("-")), ".") + if store: + if not os.path.exists(path): + sys.stderr.write("Error: No such file or directory (os error 2)\n") + return 1 + for name, project, _state in _store_entries(store): + if project["project_path"] == path: + print( + f"Project: {path}\nModel: {project.get('model', MODEL)}\n" + f"Index: {os.path.join(store, name)}\n\n" + "Run any search to update the index, or `colgrep clear` to rebuild from scratch." + ) + return 0 if os.environ.get("FAKE_COLGREP_INDEXED", "1") == "0": print(f"No index found for {path} [{MODEL}]\nRun `colgrep ` to create one.") else: diff --git a/server/tests/test_resources.py b/server/tests/test_resources.py index b8fdf4a..f9c062c 100644 --- a/server/tests/test_resources.py +++ b/server/tests/test_resources.py @@ -152,3 +152,23 @@ async def test_read_errors_resource_lists_every_code(settings_env): for code in Code: assert f"`{code}`" in text assert HINTS[code] in text + + +async def test_read_indexes_carries_store_fields(settings_env, fake_store, tmp_path, monkeypatch): + """`colgrep://indexes` serves the same enriched `IndexList` as `list_indexes` (R01 §C4).""" + import colgrep_mcp.store as store_module + + live = tmp_path / "live" + live.mkdir() + monkeypatch.setattr(store_module, "machine_state_roots", lambda home: []) + fake_store("live-0001", live, search_count=2) + fake_store("gone-0002", tmp_path / "gone") + + async with Client(build(), raise_exceptions=True) as client: + result = await client.read_resource("colgrep://indexes") + + payload = json.loads(result.contents[0].text) + assert payload["store_root"] == str(fake_store.root) + by_project = {i["project"]: i for i in payload["indexes"]} + assert by_project[str(live)]["stale"] is None and by_project[str(live)]["size_bytes"] > 0 + assert by_project[str(tmp_path / "gone")]["stale"] == "orphaned" diff --git a/server/tests/test_store.py b/server/tests/test_store.py new file mode 100644 index 0000000..f51b222 --- /dev/null +++ b/server/tests/test_store.py @@ -0,0 +1,171 @@ +"""Tests for `colgrep_mcp.store` (index_housekeeping R01 §C1–C3): the store read, the +ordered classification with injected clock/home/roots, and the drift guard that keeps +the server's machine-state roots equal to the hook's. +""" + +from __future__ import annotations + +import importlib.util +import os +import time +from pathlib import Path + +import pytest + +from colgrep_mcp import store +from colgrep_mcp.adapter import ColgrepAdapter +from colgrep_mcp.config import Settings + +REPO_ROOT = Path(__file__).resolve().parents[2] +HOOK_SCRIPT = REPO_ROOT / "hooks" / "colgrep_policy.py" + +pytestmark = pytest.mark.anyio + + +NOW = 1_800_000_000.0 + + +def _entry(project: str, *, search_count: int = 5, age_days: float = 0, now: float = NOW) -> store.StoreEntry: + return store.StoreEntry( + index_dir=Path("/store") / (Path(project).name or "root"), + project=project, + model="m", + files=3, + search_count=search_count, + size_bytes=1024, + last_modified=now - age_days * 86400, + ) + + +def _classify(entries, *, home: str, roots: list[str] | None = None, **kw): + verdicts = store.classify(entries, now=NOW, home=home, machine_roots=roots or [], **kw) + return {c.entry.project: c for c in verdicts} + + +# --- read_store ----------------------------------------------------------------------- + + +def test_read_store_reads_project_state_size_and_state_mtime(fake_store, tmp_path): + live = tmp_path / "live" + live.mkdir() + fake_store("live-0001", live, search_count=4, files=7, age_days=40) + (fake_store.root / "not-an-index").mkdir() # no project.json: skipped, not an error + (fake_store.root / "stray-file").write_text("x") + + entries = store.read_store(fake_store.root) + + assert [e.project for e in entries] == [str(live)] + e = entries[0] + assert e.files == 7 + assert e.search_count == 4 + assert e.size_bytes > 1024 # the blob plus the two JSON files + assert time.time() - e.last_modified > 39 * 86400 # `state.json` mtime, back-dated by the fixture + + +def test_read_store_missing_root_is_empty(tmp_path): + assert store.read_store(tmp_path / "nowhere") == [] + + +# --- classify --------------------------------------------------------------------------- + + +def test_classify_orphaned_wins_over_everything(tmp_path): + gone = str(tmp_path / "gone") + verdict = _classify([_entry(gone, search_count=0, age_days=400)], home=str(tmp_path), roots=[str(tmp_path)]) + assert verdict[gone].kind == store.ORPHANED + assert verdict[gone].path_exists is False + + +def test_classify_machine_state_under_a_root_and_under_a_hidden_home_dir(tmp_path): + home = tmp_path / "home" + temp = tmp_path / "temp" + under_temp = temp / "scratch" + hidden = home / ".cache" / "uv" / "archive" + for d in (under_temp, hidden): + d.mkdir(parents=True) + verdict = _classify([_entry(str(under_temp)), _entry(str(hidden))], home=str(home), roots=[str(temp)]) + assert verdict[str(under_temp)].kind == store.MACHINE_STATE + assert verdict[str(hidden)].kind == store.MACHINE_STATE + + +def test_classify_hidden_work_tree_is_not_machine_state(tmp_path): + """Claude Code's own `.claude/worktrees/` is a real work tree under a hidden + directory: a `.git` entry anywhere up the path keeps it out of `machine_state`.""" + home = tmp_path / "home" + wt = home / ".claude" / "worktrees" / "feature" + wt.mkdir(parents=True) + (wt / ".git").write_text("gitdir: elsewhere\n") + verdict = _classify([_entry(str(wt))], home=str(home)) + assert verdict[str(wt)].kind is None + + +def test_classify_shadowed_names_the_nearest_live_ancestor(tmp_path): + root = tmp_path / "repo" + mid = root / "pkg" + leaf = mid / "sub" + leaf.mkdir(parents=True) + verdict = _classify([_entry(str(root)), _entry(str(mid)), _entry(str(leaf))], home=str(tmp_path / "home")) + assert verdict[str(root)].kind is None + assert verdict[str(mid)].kind == store.SHADOWED and verdict[str(mid)].shadowed_by == str(root) + assert verdict[str(leaf)].kind == store.SHADOWED and verdict[str(leaf)].shadowed_by == str(mid) + + +def test_classify_child_of_an_orphaned_ancestor_is_not_shadowed(tmp_path): + child = tmp_path / "gone" / "child" + child.mkdir(parents=True) + gone = str(tmp_path / "gone-other") + verdict = _classify([_entry(gone), _entry(str(child))], home=str(tmp_path / "home")) + assert verdict[str(child)].kind is None + assert verdict[str(child)].shadowed_by is None + + +def test_classify_cold_needs_both_few_searches_and_age(tmp_path): + live = tmp_path / "live" + live.mkdir() + p = str(live) + home = str(tmp_path / "home") + assert _classify([_entry(p, search_count=1, age_days=31)], home=home)[p].kind == store.COLD + assert _classify([_entry(p, search_count=2, age_days=31)], home=home)[p].kind is None + assert _classify([_entry(p, search_count=0, age_days=29)], home=home)[p].kind is None + assert _classify([_entry(p, search_count=3, age_days=31)], home=home, max_searches=3)[p].kind == store.COLD + assert _classify([_entry(p, search_count=0, age_days=10)], home=home, days=7)[p].kind == store.COLD + + +# --- the hook convention is the server convention ---------------------------------------- + + +def test_machine_state_roots_match_the_hook(): + """R01 D4: the hook is stdlib-only and ships outside the PyPI package, so the + roots are restated in `store.py`; this pins the two lists equal.""" + spec = importlib.util.spec_from_file_location("colgrep_policy", HOOK_SCRIPT) + hook = importlib.util.module_from_spec(spec) + assert spec.loader is not None + spec.loader.exec_module(hook) + home = os.path.realpath(os.path.expanduser("~")) + assert store.machine_state_roots(home) == hook.machine_state_roots(home) + + +# --- store_root ---------------------------------------------------------------------------- + + +async def test_store_root_is_the_index_line_parent_and_is_cached(fake_store, settings_env, tmp_path, monkeypatch): + live = tmp_path / "live" + live.mkdir() + fake_store("gone-0001", tmp_path / "gone") # `status` on it exits 1 (R02): skipped + fake_store("live-0001", live) + argv_file = tmp_path / "argv.json" + monkeypatch.setenv("FAKE_COLGREP_ARGV_FILE", str(argv_file)) + + settings = Settings.from_env() + adapter = ColgrepAdapter(binary=settings.binary, timeout_s=30) + assert await adapter.store_root() == fake_store.root + argv_file.unlink() + assert await adapter.store_root() == fake_store.root + assert not argv_file.exists() # cached: no second spawn + + +async def test_store_root_is_none_when_no_indexed_project_exists(fake_store, settings_env, tmp_path): + fake_store("gone-0001", tmp_path / "gone") + settings = Settings.from_env() + adapter = ColgrepAdapter(binary=settings.binary, timeout_s=30) + assert await adapter.store_root() is None diff --git a/server/tests/test_tools_index.py b/server/tests/test_tools_index.py index fb4c694..a78234c 100644 --- a/server/tests/test_tools_index.py +++ b/server/tests/test_tools_index.py @@ -164,6 +164,87 @@ async def fake_stats(self): assert len(result.structured_content["indexes"]) == 400 +@pytest.fixture +def housekeeping_store(fake_store, tmp_path, monkeypatch): + """A store with one live project, one shadowed child, one orphan and one machine-state + entry (index_housekeeping R01 §C3). The machine-state roots are pinned to one directory + under `tmp_path` because the CI runners' own temp directory sits under the home + directory on Windows (R01 risk 2) — `tmp_path` itself must never read as machine state.""" + live = tmp_path / "live" + (live / "sub").mkdir(parents=True) + scratch = tmp_path / "temp" / "scratch" + scratch.mkdir(parents=True) + monkeypatch.setattr(tools_index.store, "machine_state_roots", lambda home: [str(tmp_path / "temp")]) + fake_store("live-0001", live, search_count=9, files=5) + fake_store("sub-0002", live / "sub", search_count=1, files=2) + fake_store("gone-0003", tmp_path / "gone", search_count=1, files=4, age_days=45) + fake_store("scratch-0004", scratch, search_count=0, files=1) + return {"live": live, "sub": live / "sub", "gone": tmp_path / "gone", "scratch": scratch} + + +async def test_list_indexes_carries_store_fields(settings_env, housekeeping_store): + """R01 §C4: size, age, path-exists, shadowing and the `stale` class ride along.""" + async with Client(build(), raise_exceptions=True) as client: + result = await client.call_tool("list_indexes", {}) + + assert not result.is_error + by_project = {i["project"]: i for i in result.structured_content["indexes"]} + assert set(by_project) == {str(p) for p in housekeeping_store.values()} + live = by_project[str(housekeeping_store["live"])] + assert live["path_exists"] is True and live["stale"] is None and live["size_bytes"] > 1024 + assert live["last_modified"][:2] == "20" + sub = by_project[str(housekeeping_store["sub"])] + assert sub["stale"] == "shadowed" and sub["shadowed_by"] == str(housekeeping_store["live"]) + gone = by_project[str(housekeeping_store["gone"])] + assert gone["stale"] == "orphaned" and gone["path_exists"] is False + assert by_project[str(housekeeping_store["scratch"])]["stale"] == "machine_state" + assert result.structured_content["total"] == 4 + assert result.structured_content["total_bytes"] > 4 * 1024 + + text = result.content[0].text + assert text.splitlines()[0].startswith("4 indexed projects on this machine (") + assert "1 orphaned, 1 machine-state, 1 shadowed)" in text.splitlines()[0] + assert f"{housekeeping_store['gone']} model=lightonai/LateOn-Code-edge units=4 searches=1 size=" in text + assert " [orphaned]" in text + assert f" [shadowed by {housekeeping_store['live']}]" in text + assert " [machine_state]" in text + + +async def test_list_indexes_stale_only_filters_and_keeps_the_total(settings_env, housekeeping_store): + async with Client(build(), raise_exceptions=True) as client: + result = await client.call_tool("list_indexes", {"stale_only": True}) + + assert not result.is_error + kinds = sorted(i["stale"] for i in result.structured_content["indexes"]) + assert kinds == ["machine_state", "orphaned", "shadowed"] + assert result.structured_content["total"] == 4 + assert result.content[0].text.splitlines()[0].startswith("3 stale of 4 indexed projects on this machine (") + + +async def test_list_indexes_stale_only_with_nothing_stale(settings_env, fake_store, tmp_path, monkeypatch): + live = tmp_path / "live" + live.mkdir() + monkeypatch.setattr(tools_index.store, "machine_state_roots", lambda home: []) + fake_store("live-0001", live, search_count=3) + + async with Client(build(), raise_exceptions=True) as client: + result = await client.call_tool("list_indexes", {"stale_only": True}) + + assert result.structured_content["indexes"] == [] + assert result.content[0].text == "No stale indexes among the 1 indexed projects on this machine." + + +async def test_list_indexes_without_a_derivable_store_keeps_the_legacy_fields(settings_env): + """R01 §C1: the fake's default `--stats` projects do not exist on disk, so no + `status` call can yield an `Index:` line; every store field stays `None`.""" + async with Client(build(), raise_exceptions=True) as client: + result = await client.call_tool("list_indexes", {}) + + assert result.structured_content["store_root"] is None + for info in result.structured_content["indexes"]: + assert info["size_bytes"] is None and info["stale"] is None and info["path_exists"] is None + + # --- doctor --------------------------------------------------------------------- From 8f733b7becdc17588ecb3bc6db21275281ce8d30 Mon Sep 17 00:00:00 2001 From: Eliott Jacopin Date: Sun, 13 Sep 2026 13:12:06 +0200 Subject: [PATCH 3/9] feat(index): add index_prune to remove orphaned, machine-state, shadowed and cold indexes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cleaning the store used to cost one confirmed index_clear per project, dozens of calls on a machine with 65 orphaned indexes (index_housekeeping R01 §Executive Summary). index_prune classifies the whole store once, lists the candidates grouped by class as a dry run by default, and with dry_run=false removes them after confirm=true or an accepted elicitation — the index_clear flow, now one shared helper. It never runs `colgrep clear`: that command exits 1 on a gone path (R02) and clears the folding ancestor on a shadowed one, so the removal is a directory delete guarded by the directory being a direct child of the store and its project.json naming the candidate at deletion time (R01 §C5). `cold` is opt-in because a live but idle project is a judgement call, not dead weight (R01 D8). Both READMEs' tool tables gain the row so test_readme stays green. Co-Authored-By: Claude Fable 5.1 --- README.md | 3 +- server/README.md | 3 +- server/colgrep_mcp/errors.py | 2 + server/colgrep_mcp/models.py | 20 +++ server/colgrep_mcp/store.py | 29 ++++ server/colgrep_mcp/tools_index.py | 229 ++++++++++++++++++++++++------ server/tests/test_errors.py | 1 + server/tests/test_stdio.py | 2 +- server/tests/test_store.py | 33 +++++ server/tests/test_tools_index.py | 177 ++++++++++++++++++++++- 10 files changed, 450 insertions(+), 49 deletions(-) diff --git a/README.md b/README.md index 7720fb2..f402efd 100644 --- a/README.md +++ b/README.md @@ -106,7 +106,8 @@ claude mcp add colgrep -- uv run --quiet --directory /path/to/colgrep-mcp/server | `index_status` | Whether a path is indexed, with which model, where, how big. | | `index_build` | Build or refresh an index now, with progress notifications. | | `index_clear` | Delete a project's index. Asks for confirmation (elicitation) or requires `confirm=true`. | -| `list_indexes` | Every indexed project on this machine. | +| `index_prune` | Remove orphaned, machine-state, shadowed and (opt-in) cold indexes. Dry run by default; `confirm=true` or elicitation to delete. | +| `list_indexes` | Every indexed project on this machine, with size, last use, whether the path still exists and who shadows it; `stale_only` filters. | | `doctor` | Environment self-check: binary, version, settings, default root. | `search` defaults to hybrid mode. Pass `pattern` (a regex) to pre-filter units by text before semantic ranking, `include`/`exclude`/`exclude_dir` to scope, `limit` to size the result. Text output is capped by a character budget; the full result is always in `structured_content`. diff --git a/server/README.md b/server/README.md index 0f052df..f4c45ad 100644 --- a/server/README.md +++ b/server/README.md @@ -52,7 +52,8 @@ Either command is a stdio MCP server; register it in your client as such: | `index_status` | Whether a path is indexed, with which model, where, how big. | | `index_build` | Build or refresh an index now, with progress notifications. | | `index_clear` | Delete a project's index, after confirmation. | -| `list_indexes` | Every indexed project on this machine. | +| `index_prune` | Remove orphaned, machine-state, shadowed and (opt-in) cold indexes; dry run by default. | +| `list_indexes` | Every indexed project on this machine, with size, last use, path-exists and shadowing. | | `doctor` | Environment self-check: binary, version, settings, default root. | Resources (`colgrep://guide`, `colgrep://settings`, `colgrep://indexes`, `colgrep://status/{+path}`, `colgrep://errors`) and prompts (`explore`, `locate`, `impact`) come with it; the guide resource teaches the agent how to compose queries. diff --git a/server/colgrep_mcp/errors.py b/server/colgrep_mcp/errors.py index 2b89a5c..62bc23b 100644 --- a/server/colgrep_mcp/errors.py +++ b/server/colgrep_mcp/errors.py @@ -38,6 +38,7 @@ class Code(StrEnum): COLGREP_FAILED = "COLGREP_FAILED" COLGREP_TIMEOUT = "COLGREP_TIMEOUT" BAD_HIT_ID = "BAD_HIT_ID" + INDEX_STORE_UNKNOWN = "INDEX_STORE_UNKNOWN" #: One imperative sentence per code, naming the tool/argument to reach for next. @@ -63,6 +64,7 @@ class Code(StrEnum): Code.COLGREP_FAILED: "Read the stderr tail; run `doctor`; retry with simpler arguments.", Code.COLGREP_TIMEOUT: "Call `index_build` on the path first, then retry.", Code.BAD_HIT_ID: "Use `hit_id` values exactly as returned by `search`: `:-`.", + Code.INDEX_STORE_UNKNOWN: "Run `index_build` on a project that exists on disk so the store can be located; retry.", } diff --git a/server/colgrep_mcp/models.py b/server/colgrep_mcp/models.py index c7a6758..a043000 100644 --- a/server/colgrep_mcp/models.py +++ b/server/colgrep_mcp/models.py @@ -127,6 +127,26 @@ class IndexClearResult(BaseModel): cleared: bool +class PruneCandidate(BaseModel): + project: str + index_dir: str + kind: str = Field(description="`orphaned`, `machine_state`, `shadowed` or `cold` (index_housekeeping R01 §C3).") + size_bytes: int + last_modified: str + search_count: int | None = None + shadowed_by: str | None = None + + +class PruneResult(BaseModel): + dry_run: bool + store_root: str + candidates: list[PruneCandidate] + total_bytes: int = Field(description="Bytes under every candidate's index directory.") + pruned: list[str] = Field(default_factory=list, description="Projects whose index directory was removed.") + failed: list[str] = Field(default_factory=list, description="`: ` per candidate left in place.") + freed_bytes: int = 0 + + class Doctor(BaseModel): colgrep_path: str | None version: str | None diff --git a/server/colgrep_mcp/store.py b/server/colgrep_mcp/store.py index 472402f..d6c9917 100644 --- a/server/colgrep_mcp/store.py +++ b/server/colgrep_mcp/store.py @@ -16,6 +16,7 @@ import asyncio import json import os +import shutil import tempfile import time from dataclasses import dataclass @@ -226,6 +227,34 @@ def iso_utc(ts: float) -> str: return datetime.fromtimestamp(ts, tz=UTC).isoformat(timespec="seconds") +# --- removal -------------------------------------------------------------------------- + + +class StoreError(Exception): + """A removal guard refused (R01 §C5); the directory was left in place.""" + + +def remove_index_dir(root: Path, index_dir: Path, project: str) -> None: + """Delete `index_dir` only if it is a direct child of `root` whose `project.json` + names `project` right now (index_housekeeping R01 §C5). + + Never `colgrep clear`: on a gone path it exits 1 (R02), and on a folded + path it clears the ancestor project instead — the failure `index_clear`'s + `PROJECT_ROOT_MISMATCH` exists to prevent. Re-reading `project.json` at + deletion time means a candidate list from an earlier call, or a store + that changed under us, can never point the delete at another project. + """ + if index_dir.parent != root: + raise StoreError(f"{index_dir} is not directly under the store root {root}") + try: + named = json.loads((index_dir / "project.json").read_text(encoding="utf-8")).get("project_path") + except (OSError, ValueError) as exc: + raise StoreError(f"{index_dir}/project.json unreadable: {exc}") from exc + if named != project: + raise StoreError(f"{index_dir}/project.json names {named!r}, not {project!r}") + shutil.rmtree(index_dir) + + # --- the joined view -------------------------------------------------------------- diff --git a/server/colgrep_mcp/tools_index.py b/server/colgrep_mcp/tools_index.py index 696b065..7cdead7 100644 --- a/server/colgrep_mcp/tools_index.py +++ b/server/colgrep_mcp/tools_index.py @@ -1,5 +1,6 @@ """Index management tools: `index_status`, `list_indexes`, `doctor`, `index_build`, -`index_clear` (R01 §Tools; R05 D2 heartbeat, D3 project-root refusal, M2 safe_log). +`index_clear` (R01 §Tools; R05 D2 heartbeat, D3 project-root refusal, M2 safe_log) +and `index_prune` (index_housekeeping R01 §C5). """ from __future__ import annotations @@ -9,7 +10,7 @@ import shutil import time from pathlib import Path -from typing import Annotated +from typing import Annotated, Literal from mcp.server import MCPServer from mcp.server.mcpserver import Context @@ -21,7 +22,16 @@ from .errors import Code, tool_error, translate_adapter_errors from .locks import project_lock from .logging_utils import safe_log, safe_notify_resource_updated, safe_progress -from .models import Doctor, IndexBuildResult, IndexClearResult, IndexInfo, IndexList, IndexStatus +from .models import ( + Doctor, + IndexBuildResult, + IndexClearResult, + IndexInfo, + IndexList, + IndexStatus, + PruneCandidate, + PruneResult, +) from .paths import client_roots, default_root, resolve_target_paths from .server import READ_ONLY_TOOL, get_adapter, get_settings, register_tool @@ -32,7 +42,7 @@ class Confirm(BaseModel): - """Elicitation schema for `index_clear`'s human-in-the-loop confirmation.""" + """Elicitation schema for the human-in-the-loop confirmation of `index_clear` and `index_prune`.""" confirm: bool @@ -42,6 +52,32 @@ async def _resolve_one(path: str | None, ctx: Context) -> Path: return (await resolve_target_paths(ctx, [path] if path else None))[0] +async def _confirmed(ctx: Context, confirm: bool, *, question: str, refusal: str) -> bool: + """The one confirmation flow for a destructive tool (R05 M3; index_housekeeping R01 §C5). + + `confirm=true` short-circuits. Otherwise the client must advertise the + elicitation capability, and the elicitation call itself must succeed — + either failing is a technical refusal (`CONFIRMATION_REQUIRED`), never + a silent decline (F8). Returns `False` only for a real decline. + """ + if confirm: + return True + try: + has_elicitation = ctx.session.check_client_capability(ClientCapabilities(elicitation=ElicitationCapability())) + except Exception: # noqa: BLE001 - treat any capability-check failure as "unavailable" + has_elicitation = False + if not has_elicitation: + raise tool_error(Code.CONFIRMATION_REQUIRED, refusal) + try: + res = await ctx.elicit(question, schema=Confirm) + except Exception as exc: # noqa: BLE001 - the elicitation call itself failing + # (e.g. `NoBackChannelError`) is a technical failure, not a user + # decision — it must not be collapsed into the same silent + # "declined" response a real decline gets. + raise tool_error(Code.CONFIRMATION_REQUIRED, f"{refusal[:-1]} (elicitation failed: {exc}).") from exc + return res.action == "accept" and bool(res.data.confirm) + + def _match_stats(status: IndexStatus, stats: list[IndexInfo]) -> IndexInfo | None: """Find `status.project`'s entry in `stats` without a `Path.resolve()` syscall per candidate. @@ -151,6 +187,44 @@ def _render_doctor(doc: Doctor) -> str: return "\n".join(lines) +def _candidate_block(c: PruneCandidate) -> str: + text = f" {c.project} size={_mib(c.size_bytes)} modified={c.last_modified[:10]} searches={c.search_count}" + if c.shadowed_by: + text += f" shadowed by {c.shadowed_by}" + return text + + +def _render_prune(result: PruneResult, classes: list[str], budget: int) -> tuple[str, bool]: + """Candidates grouped by class through the one budgeted renderer (R01 §C7), the exact + next call as the trailing note (index_housekeeping R01 §C5).""" + if not result.candidates: + return f"Nothing to prune in {result.store_root} for classes {classes}.", False + n = len(result.candidates) + if result.dry_run: + header = f"{n} prune candidates ({_human_size(result.total_bytes)}) in {result.store_root}" + classes_arg = "[" + ", ".join(f'"{c}"' for c in classes) + "]" + notes = [ + "Dry run: nothing removed. Call " + f"index_prune(dry_run=false, confirm=true, classes={classes_arg}) to remove them." + ] + else: + header = ( + f"Pruned {len(result.pruned)} of {n} indexes ({_human_size(result.freed_bytes)} freed) " + f"from {result.store_root}" + ) + notes = [f"failed: {line}" for line in result.failed] + blocks: list[str] = [] + for kind in store.PRUNE_CLASSES: + group = [c for c in result.candidates if c.kind == kind] + if not group: + continue + blocks.append(f"{kind} ({len(group)}, {_human_size(sum(c.size_bytes for c in group))}):") + blocks += [_candidate_block(c) for c in group] + return tools_search._render_budgeted( + header, blocks, lambda remaining: f" [{remaining} more lines in structured_content]", notes, budget + ) + + def _build_summary_line(result: IndexBuildResult) -> str: if result.up_to_date: return f"Index is up to date for {result.project}" @@ -353,42 +427,18 @@ async def index_clear( f"colgrep would clear the index for {st.project}, which also covers other directories than {resolved}.", ) - if not confirm: - has_elicitation = False - try: - has_elicitation = ctx.session.check_client_capability( - ClientCapabilities(elicitation=ElicitationCapability()) - ) - except Exception: # noqa: BLE001 - treat any capability-check failure as "unavailable" - has_elicitation = False - - if not has_elicitation: - raise tool_error( - Code.CONFIRMATION_REQUIRED, - f"Refusing to delete the index for {resolved} without confirmation.", - ) - - try: - res = await ctx.elicit( - f"Delete the colgrep index for {resolved}? This cannot be undone.", - schema=Confirm, - ) - except Exception as exc: # noqa: BLE001 - the elicitation call itself failing - # (e.g. `NoBackChannelError`) is a technical failure, not a user - # decision — it must not be collapsed into the same silent - # `cleared=False` "declined" response a real decline gets. - # Surface the same coded refusal as "no elicitation capability". - raise tool_error( - Code.CONFIRMATION_REQUIRED, - f"Refusing to delete the index for {resolved} without confirmation (elicitation failed: {exc}).", - ) from exc - - if res.action != "accept" or not res.data.confirm: - result = IndexClearResult(project=str(resolved), cleared=False) - return CallToolResult( - content=[TextContent(type="text", text="Not cleared (declined)")], - structured_content=result.model_dump(), - ) + ok = await _confirmed( + ctx, + confirm, + question=f"Delete the colgrep index for {resolved}? This cannot be undone.", + refusal=f"Refusing to delete the index for {resolved} without confirmation.", + ) + if not ok: + result = IndexClearResult(project=str(resolved), cleared=False) + return CallToolResult( + content=[TextContent(type="text", text="Not cleared (declined)")], + structured_content=result.model_dump(), + ) async with project_lock(resolved): async with translate_adapter_errors(path=resolved): @@ -403,6 +453,99 @@ async def index_clear( ) +PruneClass = Literal["orphaned", "machine_state", "shadowed", "cold"] + + +async def index_prune( + classes: Annotated[ + list[PruneClass], + Field( + description="Which indexes count as candidates: `orphaned` (project path gone), `machine_state` " + "(temp, cache or hidden tree), `shadowed` (inside another indexed project), `cold` (at most " + "`max_searches` searches and untouched for `days`; opt-in)." + ), + ] = ["orphaned", "machine_state", "shadowed"], # noqa: B006 - pydantic copies the default per call + days: Annotated[int, Field(description="Age in days for `cold`.", ge=0)] = 30, + max_searches: Annotated[int, Field(description="Search count at or below which an index is `cold`.", ge=0)] = 1, + dry_run: Annotated[bool, Field(description="List the candidates without removing anything (default).")] = True, + confirm: Annotated[bool, Field(description="With dry_run=false: remove without an elicitation prompt.")] = False, + *, + ctx: Context, +) -> CallToolResult: + """Remove stale colgrep indexes in one call instead of one `index_clear` per project: orphaned + (path gone), machine-state (temp, cache, hidden tree), shadowed (inside another indexed project) + and, opt-in, cold ones. Dry run by default; `dry_run=false` with `confirm=true` or an accepted + elicitation deletes the index directories. Never touches a live project's index.""" + adapter = get_adapter(ctx) + settings = get_settings(ctx) + + async with translate_adapter_errors(): + infos = await adapter.stats() + root = await adapter.store_root(infos) + if root is None: + raise tool_error(Code.INDEX_STORE_UNKNOWN, "No indexed project exists on disk, so the store cannot be located.") + + entries = await asyncio.to_thread(store.read_store, root) + verdicts = store.classify_now(entries, days=days, max_searches=max_searches) + candidates = [ + PruneCandidate( + project=v.entry.project, + index_dir=str(v.entry.index_dir), + kind=v.kind, + size_bytes=v.entry.size_bytes, + last_modified=store.iso_utc(v.entry.last_modified), + search_count=v.entry.search_count, + shadowed_by=v.shadowed_by, + ) + for v in verdicts + if v.kind is not None and v.kind in classes + ] + result = PruneResult( + dry_run=dry_run, + store_root=str(root), + candidates=candidates, + total_bytes=sum(c.size_bytes for c in candidates), + ) + + if not dry_run and candidates: + ok = await _confirmed( + ctx, + confirm, + question=( + f"Remove {len(candidates)} colgrep indexes ({_human_size(result.total_bytes)}) from {root}? " + "This cannot be undone." + ), + refusal=f"Refusing to remove {len(candidates)} indexes from {root} without confirmation.", + ) + if not ok: + return CallToolResult( + content=[TextContent(type="text", text="Not pruned (declined)")], + structured_content=result.model_copy(update={"dry_run": True}).model_dump(), + ) + for c in candidates: + async with project_lock(Path(c.project)): + try: + # Not through `translate_adapter_errors`: one refused or failed + # removal must not abort the batch, it lands in `failed`. + await asyncio.to_thread(store.remove_index_dir, root, Path(c.index_dir), c.project) + except (store.StoreError, OSError) as exc: + result.failed.append(f"{c.project}: {exc}") + continue + result.pruned.append(c.project) + result.freed_bytes += c.size_bytes + if result.pruned: + await safe_notify_resource_updated(ctx, "colgrep://indexes") + + text, capped = _render_prune(result, list(classes), settings.text_budget) + if capped: + budget = settings.text_budget + await safe_log(ctx, "warning", f"index_prune text truncated to {budget} chars; see structured_content") + return CallToolResult( + content=[TextContent(type="text", text=text)], + structured_content=result.model_dump(), + ) + + # --- registration -------------------------------------------------------------- @@ -425,3 +568,9 @@ def register(mcp: MCPServer) -> None: title="Clear index", annotations=ToolAnnotations(destructive_hint=True, open_world_hint=False), ) + register_tool( + mcp, + index_prune, + title="Prune stale indexes", + annotations=ToolAnnotations(destructive_hint=True, idempotent_hint=True, open_world_hint=False), + ) diff --git a/server/tests/test_errors.py b/server/tests/test_errors.py index 4404c16..953e70c 100644 --- a/server/tests/test_errors.py +++ b/server/tests/test_errors.py @@ -35,6 +35,7 @@ def test_code_enum_has_exactly_the_specified_members(): "COLGREP_FAILED", "COLGREP_TIMEOUT", "BAD_HIT_ID", + "INDEX_STORE_UNKNOWN", } diff --git a/server/tests/test_stdio.py b/server/tests/test_stdio.py index 94585a9..b4bfb17 100644 --- a/server/tests/test_stdio.py +++ b/server/tests/test_stdio.py @@ -59,7 +59,7 @@ async def test_stdio_round_trip(fake_colgrep_bin, tmp_path): assert client.server_info.name == "colgrep" tools = await client.list_tools() - assert len(tools.tools) == 8 + assert len(tools.tools) == 9 resources = await client.list_resources() # 3 today (guide, settings, indexes); a fourth (colgrep://errors) is diff --git a/server/tests/test_store.py b/server/tests/test_store.py index f51b222..5a5eb21 100644 --- a/server/tests/test_store.py +++ b/server/tests/test_store.py @@ -169,3 +169,36 @@ async def test_store_root_is_none_when_no_indexed_project_exists(fake_store, set settings = Settings.from_env() adapter = ColgrepAdapter(binary=settings.binary, timeout_s=30) assert await adapter.store_root() is None + + +# --- remove_index_dir -------------------------------------------------------------------- + + +def test_remove_index_dir_removes_a_matching_child(fake_store, tmp_path): + d = fake_store("gone-0001", tmp_path / "gone") + store.remove_index_dir(fake_store.root, d, str(tmp_path / "gone")) + assert not d.exists() + + +def test_remove_index_dir_refuses_a_project_mismatch(fake_store, tmp_path): + d = fake_store("live-0001", tmp_path / "live") + with pytest.raises(store.StoreError, match="names"): + store.remove_index_dir(fake_store.root, d, str(tmp_path / "gone")) + assert d.is_dir() + + +def test_remove_index_dir_refuses_a_directory_outside_the_root(fake_store, tmp_path): + d = fake_store("live-0001", tmp_path / "live") + with pytest.raises(store.StoreError, match="not directly under"): + store.remove_index_dir(tmp_path / "elsewhere", d, str(tmp_path / "live")) + with pytest.raises(store.StoreError, match="not directly under"): + store.remove_index_dir(fake_store.root, d / "index", str(tmp_path / "live")) + assert d.is_dir() + + +def test_remove_index_dir_refuses_a_missing_project_json(fake_store, tmp_path): + d = fake_store("live-0001", tmp_path / "live") + (d / "project.json").unlink() + with pytest.raises(store.StoreError, match="unreadable"): + store.remove_index_dir(fake_store.root, d, str(tmp_path / "live")) + assert d.is_dir() diff --git a/server/tests/test_tools_index.py b/server/tests/test_tools_index.py index a78234c..76184b6 100644 --- a/server/tests/test_tools_index.py +++ b/server/tests/test_tools_index.py @@ -7,6 +7,7 @@ from __future__ import annotations import asyncio +import dataclasses import json import pytest @@ -179,7 +180,10 @@ def housekeeping_store(fake_store, tmp_path, monkeypatch): fake_store("sub-0002", live / "sub", search_count=1, files=2) fake_store("gone-0003", tmp_path / "gone", search_count=1, files=4, age_days=45) fake_store("scratch-0004", scratch, search_count=0, files=1) - return {"live": live, "sub": live / "sub", "gone": tmp_path / "gone", "scratch": scratch} + old = tmp_path / "old" + old.mkdir() + fake_store("old-0005", old, search_count=1, files=2, age_days=45) + return {"live": live, "sub": live / "sub", "gone": tmp_path / "gone", "scratch": scratch, "old": old} async def test_list_indexes_carries_store_fields(settings_env, housekeeping_store): @@ -198,11 +202,12 @@ async def test_list_indexes_carries_store_fields(settings_env, housekeeping_stor gone = by_project[str(housekeeping_store["gone"])] assert gone["stale"] == "orphaned" and gone["path_exists"] is False assert by_project[str(housekeeping_store["scratch"])]["stale"] == "machine_state" - assert result.structured_content["total"] == 4 - assert result.structured_content["total_bytes"] > 4 * 1024 + assert by_project[str(housekeeping_store["old"])]["stale"] is None # `cold` is index_prune's, not `stale` + assert result.structured_content["total"] == 5 + assert result.structured_content["total_bytes"] > 5 * 1024 text = result.content[0].text - assert text.splitlines()[0].startswith("4 indexed projects on this machine (") + assert text.splitlines()[0].startswith("5 indexed projects on this machine (") assert "1 orphaned, 1 machine-state, 1 shadowed)" in text.splitlines()[0] assert f"{housekeeping_store['gone']} model=lightonai/LateOn-Code-edge units=4 searches=1 size=" in text assert " [orphaned]" in text @@ -217,8 +222,8 @@ async def test_list_indexes_stale_only_filters_and_keeps_the_total(settings_env, assert not result.is_error kinds = sorted(i["stale"] for i in result.structured_content["indexes"]) assert kinds == ["machine_state", "orphaned", "shadowed"] - assert result.structured_content["total"] == 4 - assert result.content[0].text.splitlines()[0].startswith("3 stale of 4 indexed projects on this machine (") + assert result.structured_content["total"] == 5 + assert result.content[0].text.splitlines()[0].startswith("3 stale of 5 indexed projects on this machine (") async def test_list_indexes_stale_only_with_nothing_stale(settings_env, fake_store, tmp_path, monkeypatch): @@ -484,3 +489,163 @@ async def decline(context: object, params: ElicitRequestParams) -> ElicitResult: argv = json.loads(argv_file.read_text()) # only `status` ran; `clear` must not have. assert "clear" not in argv + + +# --- index_prune ----------------------------------------------------------------------- + + +def _store_names(root): + return sorted(p.name for p in root.iterdir() if p.is_dir()) + + +async def test_index_prune_dry_run_groups_candidates_and_removes_nothing(settings_env, housekeeping_store, fake_store): + """R01 §C5: the default call is a dry run over the three parameter-free classes.""" + async with Client(build(), raise_exceptions=True) as client: + result = await client.call_tool("index_prune", {}) + + assert not result.is_error + sc = result.structured_content + assert sc["dry_run"] is True and sc["pruned"] == [] and sc["failed"] == [] + assert sc["store_root"] == str(fake_store.root) + kinds = {c["project"]: c["kind"] for c in sc["candidates"]} + assert kinds == { + str(housekeeping_store["gone"]): "orphaned", + str(housekeeping_store["scratch"]): "machine_state", + str(housekeeping_store["sub"]): "shadowed", + } + assert sc["total_bytes"] == sum(c["size_bytes"] for c in sc["candidates"]) > 3 * 1024 + assert _store_names(fake_store.root) == ["gone-0003", "live-0001", "old-0005", "scratch-0004", "sub-0002"] + + lines = result.content[0].text.splitlines() + assert lines[0].startswith("3 prune candidates (") + assert "orphaned (1, " in result.content[0].text + assert "machine_state (1, " in result.content[0].text + assert "shadowed (1, " in result.content[0].text + assert f" shadowed by {housekeeping_store['live']}" in result.content[0].text + assert lines[-1] == ( + "Dry run: nothing removed. Call index_prune(dry_run=false, confirm=true, " + 'classes=["orphaned", "machine_state", "shadowed"]) to remove them.' + ) + + +async def test_index_prune_cold_is_opt_in(settings_env, housekeeping_store): + async with Client(build(), raise_exceptions=True) as client: + default = await client.call_tool("index_prune", {}) + cold = await client.call_tool("index_prune", {"classes": ["cold"]}) + cold_recent = await client.call_tool("index_prune", {"classes": ["cold"], "days": 60}) + cold_busy = await client.call_tool("index_prune", {"classes": ["cold"], "max_searches": 0}) + + assert all("cold" != c["kind"] for c in default.structured_content["candidates"]) + assert [c["project"] for c in cold.structured_content["candidates"]] == [str(housekeeping_store["old"])] + assert cold_recent.structured_content["candidates"] == [] + assert cold_busy.structured_content["candidates"] == [] + assert cold_recent.content[0].text.startswith("Nothing to prune in ") + + +async def test_index_prune_rejects_an_unknown_class(settings_env, housekeeping_store): + async with Client(build(), raise_exceptions=True) as client: + result = await client.call_tool("index_prune", {"classes": ["live"]}) + assert result.is_error + + +async def test_index_prune_confirm_removes_only_the_candidates(settings_env, housekeeping_store, fake_store): + async with Client(build(), raise_exceptions=True) as client: + result = await client.call_tool("index_prune", {"dry_run": False, "confirm": True}) + + assert not result.is_error + sc = result.structured_content + assert sc["dry_run"] is False + assert sorted(sc["pruned"]) == sorted(str(housekeeping_store[k]) for k in ("gone", "scratch", "sub")) + assert sc["failed"] == [] + assert sc["freed_bytes"] == sc["total_bytes"] > 0 + assert _store_names(fake_store.root) == ["live-0001", "old-0005"] + assert result.content[0].text.startswith("Pruned 3 of 3 indexes (") + + +async def test_index_prune_without_confirm_and_no_elicitation_refused(settings_env, housekeeping_store, fake_store): + async with Client(build(), raise_exceptions=True) as client: + result = await client.call_tool("index_prune", {"dry_run": False}) + + assert result.is_error + text = result.content[0].text + assert f"[{Code.CONFIRMATION_REQUIRED}] " in text + assert text.endswith(f"Next: {HINTS[Code.CONFIRMATION_REQUIRED]}") + assert len(_store_names(fake_store.root)) == 5 + + +async def test_index_prune_elicitation_accept_removes(settings_env, housekeeping_store, fake_store): + async def accept(context: object, params: ElicitRequestParams) -> ElicitResult: + assert "Remove 3 colgrep indexes" in params.message + return ElicitResult(action="accept", content={"confirm": True}) + + async with Client(build(), raise_exceptions=True, elicitation_callback=accept, mode="legacy") as client: + result = await client.call_tool("index_prune", {"dry_run": False}) + + assert not result.is_error + assert len(result.structured_content["pruned"]) == 3 + assert _store_names(fake_store.root) == ["live-0001", "old-0005"] + + +async def test_index_prune_elicitation_decline_removes_nothing(settings_env, housekeeping_store, fake_store): + async def decline(context: object, params: ElicitRequestParams) -> ElicitResult: + return ElicitResult(action="decline") + + async with Client(build(), raise_exceptions=True, elicitation_callback=decline, mode="legacy") as client: + result = await client.call_tool("index_prune", {"dry_run": False}) + + assert not result.is_error + assert result.content[0].text == "Not pruned (declined)" + assert result.structured_content["pruned"] == [] + assert len(_store_names(fake_store.root)) == 5 + + +async def test_index_prune_nothing_to_prune_needs_no_confirmation(settings_env, fake_store, tmp_path, monkeypatch): + live = tmp_path / "live" + live.mkdir() + monkeypatch.setattr(tools_index.store, "machine_state_roots", lambda home: []) + fake_store("live-0001", live, search_count=3) + + async with Client(build(), raise_exceptions=True) as client: + result = await client.call_tool("index_prune", {"dry_run": False}) + + assert not result.is_error + assert result.content[0].text.startswith("Nothing to prune in ") + + +async def test_index_prune_guard_refuses_a_directory_naming_another_project( + settings_env, housekeeping_store, fake_store, monkeypatch +): + """R01 §C5: the delete re-reads `project.json`; a candidate whose directory names a + different project (a store that changed under us) lands in `failed` and stays.""" + real_read_store = tools_index.store.read_store + live_dir = fake_store.root / "live-0001" + + def tampered(root): + entries = real_read_store(root) + # Point the orphan's candidate at the live project's directory. + return [ + e if e.project != str(housekeeping_store["gone"]) else dataclasses.replace(e, index_dir=live_dir) + for e in entries + ] + + monkeypatch.setattr(tools_index.store, "read_store", tampered) + + async with Client(build(), raise_exceptions=True) as client: + result = await client.call_tool("index_prune", {"dry_run": False, "confirm": True, "classes": ["orphaned"]}) + + assert not result.is_error + sc = result.structured_content + assert sc["pruned"] == [] + assert len(sc["failed"]) == 1 and "project.json names" in sc["failed"][0] + assert live_dir.is_dir() + assert "failed: " in result.content[0].text + + +async def test_index_prune_without_a_derivable_store_is_a_coded_error(settings_env): + async with Client(build(), raise_exceptions=True) as client: + result = await client.call_tool("index_prune", {}) + + assert result.is_error + text = result.content[0].text + assert f"[{Code.INDEX_STORE_UNKNOWN}] " in text + assert text.endswith(f"Next: {HINTS[Code.INDEX_STORE_UNKNOWN]}") From 01b90d03c49aefb850dbfae85e795c07f265e477 Mon Sep 17 00:00:00 2001 From: Eliott Jacopin Date: Sun, 13 Sep 2026 13:14:45 +0200 Subject: [PATCH 4/9] feat(index): hint at orphaned and machine-state indexes from doctor MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit doctor is the self-check an agent runs when something looks off, so it is where a stale store gets noticed without a per-session hook — hooks stay at bare-interpreter cost and carry the search rule only (harness_wiring R01 §C3; index_housekeeping R01 §C6, D9). The verdict is a `hints` entry, never a `problems` one: a bloated store does not make the environment not-ok. Shadowed indexes are listed by list_indexes but not nagged about; a machine with no index at all gets no hint; a store whose every project is gone gets INDEX_STORE_UNKNOWN because nothing can be asked for its `Index:` line (R01 §C1). Co-Authored-By: Claude Fable 5.1 --- server/colgrep_mcp/errors.py | 2 ++ server/colgrep_mcp/models.py | 5 ++++ server/colgrep_mcp/tools_index.py | 42 +++++++++++++++++++++++++++-- server/tests/test_errors.py | 1 + server/tests/test_tools_index.py | 45 +++++++++++++++++++++++++++++++ 5 files changed, 93 insertions(+), 2 deletions(-) diff --git a/server/colgrep_mcp/errors.py b/server/colgrep_mcp/errors.py index 62bc23b..02c37e1 100644 --- a/server/colgrep_mcp/errors.py +++ b/server/colgrep_mcp/errors.py @@ -39,6 +39,7 @@ class Code(StrEnum): COLGREP_TIMEOUT = "COLGREP_TIMEOUT" BAD_HIT_ID = "BAD_HIT_ID" INDEX_STORE_UNKNOWN = "INDEX_STORE_UNKNOWN" + INDEX_STORE_STALE = "INDEX_STORE_STALE" #: One imperative sentence per code, naming the tool/argument to reach for next. @@ -65,6 +66,7 @@ class Code(StrEnum): Code.COLGREP_TIMEOUT: "Call `index_build` on the path first, then retry.", Code.BAD_HIT_ID: "Use `hit_id` values exactly as returned by `search`: `:-`.", Code.INDEX_STORE_UNKNOWN: "Run `index_build` on a project that exists on disk so the store can be located; retry.", + Code.INDEX_STORE_STALE: "Review them with `index_prune` (a dry run); rerun with `dry_run=false, confirm=true`.", } diff --git a/server/colgrep_mcp/models.py b/server/colgrep_mcp/models.py index a043000..b6ba83f 100644 --- a/server/colgrep_mcp/models.py +++ b/server/colgrep_mcp/models.py @@ -155,3 +155,8 @@ class Doctor(BaseModel): root_source: str ok: bool problems: list[str] = Field(default_factory=list) + hints: list[str] = Field( + default_factory=list, + description="`[CODE] ...` advice that does not make the environment not-ok, e.g. a stale index store " + "(index_housekeeping R01 §C6).", + ) diff --git a/server/colgrep_mcp/tools_index.py b/server/colgrep_mcp/tools_index.py index 7cdead7..98381d5 100644 --- a/server/colgrep_mcp/tools_index.py +++ b/server/colgrep_mcp/tools_index.py @@ -18,8 +18,8 @@ from pydantic import BaseModel, Field from . import store, tools_search -from .adapter import ColgrepError, ColgrepNotFound -from .errors import Code, tool_error, translate_adapter_errors +from .adapter import ColgrepAdapter, ColgrepError, ColgrepNotFound +from .errors import HINTS, Code, note, tool_error, translate_adapter_errors from .locks import project_lock from .logging_utils import safe_log, safe_notify_resource_updated, safe_progress from .models import ( @@ -184,6 +184,8 @@ def _render_doctor(doc: Doctor) -> str: lines.append("settings: " + ", ".join(f"{k}={v}" for k, v in doc.settings.items())) for problem in doc.problems: lines.append(f"problem: {problem}") + for hint in doc.hints: + lines.append(f"hint: {hint}") return "\n".join(lines) @@ -327,6 +329,10 @@ async def doctor(*, ctx: Context) -> CallToolResult: except ColgrepError as exc: problems.append(f"colgrep settings failed: {exc}") + hints: list[str] = [] + if version is not None: + hints = await _store_hints(adapter) + doc = Doctor( colgrep_path=colgrep_path, version=version, @@ -335,6 +341,7 @@ async def doctor(*, ctx: Context) -> CallToolResult: root_source=root_source, ok=not problems, problems=problems, + hints=hints, ) return CallToolResult( content=[TextContent(type="text", text=_render_doctor(doc))], @@ -342,6 +349,37 @@ async def doctor(*, ctx: Context) -> CallToolResult: ) +async def _store_hints(adapter: ColgrepAdapter) -> list[str]: + """`doctor`'s look at the index store (index_housekeeping R01 §C6, D9): a hint, never a + problem, when the store carries orphaned or machine-state indexes — or when every + indexed project is gone, so the store cannot even be located. A fresh machine with + no index at all gets no hint; a `--stats` failure is left to the tools that need it. + """ + try: + infos = await adapter.stats() + root = await adapter.store_root(infos) + except ColgrepError: + return [] + if root is None: + if infos: + return [note(Code.INDEX_STORE_UNKNOWN, f"{len(infos)} indexed projects, none of which exists on disk.")] + return [] + entries = await asyncio.to_thread(store.read_store, root) + verdicts = store.classify_now(entries) + orphaned = [v for v in verdicts if v.kind == store.ORPHANED] + machine = [v for v in verdicts if v.kind == store.MACHINE_STATE] + if not orphaned and not machine: + return [] + size = sum(v.entry.size_bytes for v in (*orphaned, *machine)) + return [ + note( + Code.INDEX_STORE_STALE, + f"{len(orphaned)} orphaned and {len(machine)} machine-state indexes ({_human_size(size)}) in {root}. " + f"{HINTS[Code.INDEX_STORE_STALE]}", + ) + ] + + # --- mutating tools ------------------------------------------------------------ diff --git a/server/tests/test_errors.py b/server/tests/test_errors.py index 953e70c..8961d32 100644 --- a/server/tests/test_errors.py +++ b/server/tests/test_errors.py @@ -36,6 +36,7 @@ def test_code_enum_has_exactly_the_specified_members(): "COLGREP_TIMEOUT", "BAD_HIT_ID", "INDEX_STORE_UNKNOWN", + "INDEX_STORE_STALE", } diff --git a/server/tests/test_tools_index.py b/server/tests/test_tools_index.py index 76184b6..d7b5fc0 100644 --- a/server/tests/test_tools_index.py +++ b/server/tests/test_tools_index.py @@ -305,6 +305,51 @@ async def list_roots(context: object) -> ListRootsResult: assert doc["default_root"] == str(tmp_path.resolve()) +async def test_doctor_hints_at_a_stale_store(settings_env, housekeeping_store, fake_store): + """R01 §C6: orphaned and machine-state indexes are a hint, not a problem — `ok` stays true.""" + async with Client(build(), raise_exceptions=True) as client: + result = await client.call_tool("doctor", {}) + + doc = result.structured_content + assert doc["ok"] is True and doc["problems"] == [] + assert len(doc["hints"]) == 1 + hint = doc["hints"][0] + assert hint.startswith(f"[{Code.INDEX_STORE_STALE}] 1 orphaned and 1 machine-state indexes (") + assert str(fake_store.root) in hint + assert hint.endswith(HINTS[Code.INDEX_STORE_STALE]) + assert f"hint: [{Code.INDEX_STORE_STALE}]" in result.content[0].text + + +async def test_doctor_no_hint_for_a_clean_store(settings_env, fake_store, tmp_path, monkeypatch): + live = tmp_path / "live" + (live / "sub").mkdir(parents=True) + monkeypatch.setattr(tools_index.store, "machine_state_roots", lambda home: []) + fake_store("live-0001", live, search_count=3) + fake_store("sub-0002", live / "sub", search_count=3) # shadowed: reported by list_indexes, not nagged about + + async with Client(build(), raise_exceptions=True) as client: + result = await client.call_tool("doctor", {}) + + assert result.structured_content["hints"] == [] + assert "hint:" not in result.content[0].text + + +async def test_doctor_hints_when_every_indexed_project_is_gone(settings_env): + """The fake's default `--stats` names two projects that do not exist on disk.""" + async with Client(build(), raise_exceptions=True) as client: + result = await client.call_tool("doctor", {}) + + hints = result.structured_content["hints"] + assert len(hints) == 1 and hints[0].startswith(f"[{Code.INDEX_STORE_UNKNOWN}] 2 indexed projects") + + +async def test_doctor_no_hint_on_a_machine_with_no_index(settings_env, fake_store): + async with Client(build(), raise_exceptions=True) as client: + result = await client.call_tool("doctor", {}) + + assert result.structured_content["hints"] == [] + + # --- index_build ------------------------------------------------------------------ From eabe0f5de719230c270922b7dbc0b4a217bff5a7 Mon Sep 17 00:00:00 2001 From: Eliott Jacopin Date: Sun, 13 Sep 2026 13:16:16 +0200 Subject: [PATCH 5/9] feat(prompts): add the housekeeping prompt that walks list, dry run, prune and check MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The tools alone still leave an agent to invent the order; the prompt fixes it as list_indexes(stale_only) → index_prune dry run → the confirmed prune with exactly the reviewed classes → list again (index_housekeeping R01 §C7), with `cold` explicitly a judgement to opt into. It takes no `path`, so the shared `complete_path` completion is untouched. Co-Authored-By: Claude Fable 5.1 --- server/colgrep_mcp/prompts.py | 24 +++++++++++++++++++++++- server/tests/test_prompts.py | 20 ++++++++++++++++++++ server/tests/test_stdio.py | 2 +- 3 files changed, 44 insertions(+), 2 deletions(-) diff --git a/server/colgrep_mcp/prompts.py b/server/colgrep_mcp/prompts.py index 012e079..52cee56 100644 --- a/server/colgrep_mcp/prompts.py +++ b/server/colgrep_mcp/prompts.py @@ -1,4 +1,4 @@ -"""Prompts (`explore`, `locate`, `impact`) and the shared `path` completion. +"""Prompts (`explore`, `locate`, `impact`, `housekeeping`) and the shared `path` completion. R01 §Prompts / R05 D5, D11: the prompt text must say `limit=None` is exhaustive only together with `pattern` (colgrep's own runtime default, @@ -137,6 +137,27 @@ def impact(change: str, path: str | None = None) -> str: ) +def housekeeping(days: str = "30") -> str: + """housekeeping — review and prune colgrep's index store on this machine.""" + return ( + "Clean up colgrep's index store on this machine. Use only the colgrep-mcp tools below; " + "never delete anything under the store by hand.\n\n" + "1. list_indexes(stale_only=true) — every index whose project path is gone (orphaned), " + "sits in a temp, cache or hidden tree (machine_state) or lies inside another indexed " + "project (shadowed). The header carries the store size and the per-class counts.\n" + "2. index_prune() — a dry run over those three classes, grouped by class with sizes. " + 'Review it. Add "cold" only if live projects with at most 1 search and untouched for ' + f'{days}+ days should go too: index_prune(classes=["orphaned", "machine_state", "shadowed", ' + f'"cold"], days={days}).\n' + "3. index_prune(dry_run=false, confirm=true, classes=[...]) with exactly the classes you " + "reviewed. The candidates are recomputed at that moment, so an index whose project came " + "back is skipped; anything the guard refused is listed under failed.\n" + "4. list_indexes(stale_only=true) again to confirm.\n\n" + "Output contract: report the candidate count and size per class before and after, the " + "bytes freed from the prune result, and every failed entry verbatim." + ) + + async def complete_path( ref: PromptReference | ResourceTemplateReference, argument: CompletionArgument, @@ -170,4 +191,5 @@ def register(mcp: MCPServer) -> None: mcp.prompt(title="Explore: answer a how/why/where question")(explore) mcp.prompt(title="Locate: find where a symbol/behaviour lives")(locate) mcp.prompt(title="Impact: find what breaks before changing something")(impact) + mcp.prompt(title="Housekeeping: review and prune the index store")(housekeeping) mcp.completion()(complete_path) diff --git a/server/tests/test_prompts.py b/server/tests/test_prompts.py index fdb171b..8c75149 100644 --- a/server/tests/test_prompts.py +++ b/server/tests/test_prompts.py @@ -30,6 +30,7 @@ async def test_list_prompts_arguments(settings_env): "explore": {"question": True, "path": False}, "locate": {"target": True, "path": False}, "impact": {"change": True, "path": False}, + "housekeeping": {"days": False}, } @@ -150,3 +151,22 @@ async def slow_stats(self): assert call_count["n"] == 1 assert set(first.completion.values) == {"/tmp/fake-corpus", "/tmp/other"} assert set(second.completion.values) == {"/tmp/fake-corpus", "/tmp/other"} + + +async def test_housekeeping_prompt_text_contract(settings_env): + """index_housekeeping R01 §C7: list → dry run → confirmed prune → list again.""" + async with Client(build(), raise_exceptions=True) as client: + result = await client.get_prompt("housekeeping", {"days": "45"}) + text = result.messages[0].content.text + assert result.messages[0].role == "user" + assert text.index("list_indexes(stale_only=true)") < text.index("index_prune()") + assert text.index("index_prune()") < text.index("index_prune(dry_run=false, confirm=true") + assert "days=45" in text and "45+ days" in text + assert '"cold"' in text and "failed" in text + assert "by hand" in text + + +async def test_housekeeping_prompt_defaults_to_thirty_days(settings_env): + async with Client(build(), raise_exceptions=True) as client: + result = await client.get_prompt("housekeeping", {}) + assert "days=30" in result.messages[0].content.text diff --git a/server/tests/test_stdio.py b/server/tests/test_stdio.py index b4bfb17..ae08b1d 100644 --- a/server/tests/test_stdio.py +++ b/server/tests/test_stdio.py @@ -70,7 +70,7 @@ async def test_stdio_round_trip(fake_colgrep_bin, tmp_path): assert len(templates.resource_templates) == 1 prompts = await client.list_prompts() - assert len(prompts.prompts) == 3 + assert len(prompts.prompts) == 4 result = await client.call_tool("search", {"query": "config parsing"}) assert result.is_error is False From cd1b0a8a803305f8a875d31c5e5100c4897f2b95 Mon Sep 17 00:00:00 2001 From: Eliott Jacopin Date: Sun, 13 Sep 2026 13:17:41 +0200 Subject: [PATCH 6/9] docs(skill): teach the housekeeping workflow in the search skill, guide and readmes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The tools and the prompt exist; the skill is what makes an agent reach for them, and the guide is where it reads the classes and the confirmation flow before a non-trivial call (index_housekeeping R01 §C7). The guide's new section also states why index_prune never runs `colgrep clear` (R02), so an agent does not "help" by clearing a shadowed path and take the ancestor project's index with it. The architecture report's C5 wording now names `store.remove_index_dir`, where the removal actually landed. Co-Authored-By: Claude Fable 5.1 --- README.md | 6 +-- .../index_housekeeping/00-architecture_v0.md | 15 +++++--- server/README.md | 2 +- server/colgrep_mcp/guide.md | 37 ++++++++++++++++++- skills/colgrep-search/SKILL.md | 29 ++++++++++++++- 5 files changed, 76 insertions(+), 13 deletions(-) diff --git a/README.md b/README.md index f402efd..315f5eb 100644 --- a/README.md +++ b/README.md @@ -108,7 +108,7 @@ claude mcp add colgrep -- uv run --quiet --directory /path/to/colgrep-mcp/server | `index_clear` | Delete a project's index. Asks for confirmation (elicitation) or requires `confirm=true`. | | `index_prune` | Remove orphaned, machine-state, shadowed and (opt-in) cold indexes. Dry run by default; `confirm=true` or elicitation to delete. | | `list_indexes` | Every indexed project on this machine, with size, last use, whether the path still exists and who shadows it; `stale_only` filters. | -| `doctor` | Environment self-check: binary, version, settings, default root. | +| `doctor` | Environment self-check: binary, version, settings, default root; hints at a stale index store. | `search` defaults to hybrid mode. Pass `pattern` (a regex) to pre-filter units by text before semantic ranking, `include`/`exclude`/`exclude_dir` to scope, `limit` to size the result. Text output is capped by a character budget; the full result is always in `structured_content`. @@ -118,13 +118,13 @@ claude mcp add colgrep -- uv run --quiet --directory /path/to/colgrep-mcp/server |:--|:--| | `colgrep://guide` | The agent guide: how to compose queries, when to use which tool. | | `colgrep://settings` | colgrep's current configuration. | -| `colgrep://indexes` | Indexed projects. | +| `colgrep://indexes` | Indexed projects, with size, last use, path-exists and shadowing per index. | | `colgrep://status/{+path}` | Index status for a path. | | `colgrep://errors` | Error and hint codes with the next step for each. | ### Prompts -`explore` (knowledge-acquisition loop for a question), `locate` (where a symbol or behaviour lives), `impact` (what a change touches). Each takes an optional `path`. +`explore` (knowledge-acquisition loop for a question), `locate` (where a symbol or behaviour lives), `impact` (what a change touches) — each takes an optional `path` — and `housekeeping` (review and prune the index store; optional `days`). ### Skill diff --git a/__reports__/index_housekeeping/00-architecture_v0.md b/__reports__/index_housekeeping/00-architecture_v0.md index 5a3cf0c..4351d3e 100644 --- a/__reports__/index_housekeeping/00-architecture_v0.md +++ b/__reports__/index_housekeeping/00-architecture_v0.md @@ -87,7 +87,7 @@ graph TD DR[doctor
hints: INDEX_STORE_STALE] RES[colgrep://indexes] HK[prompt housekeeping] - AD[adapter.stats / status
adapter.remove_index_dir] + AD[adapter.stats / status
adapter.store_root] end CG[colgrep CLI] --> AD AD -->|Index: line once| STORE @@ -219,10 +219,11 @@ PruneCandidate: project, index_dir, class, size_bytes, last_modified, capability → `[CONFIRMATION_REQUIRED]`; the elicitation call itself failing → `[CONFIRMATION_REQUIRED]`; declined → `pruned=[]`, text "Not pruned (declined)". -- Deletion is `adapter.remove_index_dir(index_dir, project)`: the directory - must be a direct child of the store root, and its `project.json` must - name `project` at the moment of deletion; otherwise the candidate lands - in `failed` and nothing is removed. Never `colgrep clear`: for a shadowed +- Deletion is `store.remove_index_dir(root, index_dir, project)` — in the + store module, not the adapter, because it touches the store's layout and + never the binary: the directory must be a direct child of the store root, + and its `project.json` must name `project` at the moment of deletion; + otherwise the candidate lands in `failed` and nothing is removed. Never `colgrep clear`: for a shadowed or a gone path that command either fails (R02) or clears the ancestor project that folded the path (colgrep_mcp R05 D3) — the exact failure `index_clear`'s `PROJECT_ROOT_MISMATCH` exists to prevent. Each removal @@ -236,7 +237,9 @@ ok"). When the store root is known and the classification finds orphaned or machine-state entries, one hint is appended: `[INDEX_STORE_STALE] orphaned, machine-state indexes () …`. `Code.INDEX_STORE_STALE` and `Code.INDEX_STORE_UNKNOWN` join `errors.HINTS` -and so `colgrep://errors`. `doctor` already spawns `--version` and +and so `colgrep://errors`. `INDEX_STORE_UNKNOWN` is hinted only when +`--stats` lists projects and none exists on disk — a machine with no index +at all gets no hint. `doctor` already spawns `--version` and `settings`; this adds `--stats`, one `status` and the store read (~40 ms). ### C7 — `housekeeping` prompt and skill section diff --git a/server/README.md b/server/README.md index f4c45ad..3e1a1e0 100644 --- a/server/README.md +++ b/server/README.md @@ -56,7 +56,7 @@ Either command is a stdio MCP server; register it in your client as such: | `list_indexes` | Every indexed project on this machine, with size, last use, path-exists and shadowing. | | `doctor` | Environment self-check: binary, version, settings, default root. | -Resources (`colgrep://guide`, `colgrep://settings`, `colgrep://indexes`, `colgrep://status/{+path}`, `colgrep://errors`) and prompts (`explore`, `locate`, `impact`) come with it; the guide resource teaches the agent how to compose queries. +Resources (`colgrep://guide`, `colgrep://settings`, `colgrep://indexes`, `colgrep://status/{+path}`, `colgrep://errors`) and prompts (`explore`, `locate`, `impact`, `housekeeping`) come with it; the guide resource teaches the agent how to compose queries. ## License diff --git a/server/colgrep_mcp/guide.md b/server/colgrep_mcp/guide.md index ce0ceff..b2ba9d6 100644 --- a/server/colgrep_mcp/guide.md +++ b/server/colgrep_mcp/guide.md @@ -38,9 +38,15 @@ doesn't share vocabulary with the query. whole project: the tool refuses when the project root differs from the path you gave and tells you the root to pass explicitly. `index_status` shows both `requested_path` and `project`. -- `list_indexes` — every indexed project on this machine, with sizes. +- `list_indexes` — every indexed project on this machine: model, units, + searches, index size, last use, whether the project path still exists, + and which indexed ancestor shadows it. `stale_only=true` keeps only the + `orphaned`, `machine_state` and `shadowed` ones (see Housekeeping). +- `index_prune` — remove stale indexes in one call. Dry run by default; + see Housekeeping for the classes and the confirmation flow. - `doctor` — environment self-check (binary found, version, default root). - Use when a tool call fails for an unclear reason. + Use when a tool call fails for an unclear reason. A `hint:` line names a + stale index store when it carries orphaned or machine-state indexes. ## Writing queries @@ -149,6 +155,33 @@ widen `paths`. - Do not skip `index_status`/`index_build` on a repository you know is large and has never been searched, then be surprised a `search` call times out. +## Housekeeping + +colgrep's index store only grows: every path you ever searched keeps its +index, including scratch directories, removed worktrees and subdirectories +of a project that was indexed later. `list_indexes` classifies each index: + +- `orphaned` — the project path no longer exists on disk. +- `machine_state` — the path is in the system temp directory, a platform + state tree (`~/Library`, `~/AppData`) or a hidden directory under home + (`~/.cache`, `~/.claude/...`) and is not a git work tree. +- `shadowed` — the path lies inside another indexed, existing project + (`shadowed_by`): its units are indexed twice, because colgrep only folds + a path into an ancestor that was indexed *first*. +- `cold` — the path exists, has at most `max_searches` searches and was + last touched `days` ago or more. Reported by `index_prune` only, opt-in. + +`index_prune()` is a dry run over the first three classes, grouped by class +with sizes and the exact next call. `index_prune(dry_run=false, +confirm=true, classes=[...])` deletes those index directories; without +`confirm` it asks through elicitation, exactly like `index_clear`. The +candidates are recomputed at deletion time and each directory is removed +only if its own `project.json` still names the candidate, so a stale +listing can never delete a live project's index. `index_prune` never runs +`colgrep clear`: on a gone path that command fails, and on a shadowed path +it would clear the ancestor project instead. The `housekeeping` prompt +walks the whole sequence. + ## Codes Every failure and every degraded success carries a stable `[CODE]` prefix, diff --git a/skills/colgrep-search/SKILL.md b/skills/colgrep-search/SKILL.md index 7576641..e04d3e6 100644 --- a/skills/colgrep-search/SKILL.md +++ b/skills/colgrep-search/SKILL.md @@ -1,6 +1,6 @@ --- name: colgrep-search -description: 'Use before shell grep/rg whenever a question is about meaning, not literal text — locating where or how something is implemented, mapping an unfamiliar codebase, finding every call site before a refactor or rename, or checking whether a repo already does X before building it again. colgrep-mcp''s tools (search, find_files, expand, index_status/index_build/index_clear, list_indexes, doctor) run hybrid semantic + keyword search over code units (functions, classes, methods, docs sections), not raw lines, so natural-language queries surface relevant code even when it shares no vocabulary with the query. Anti-pattern: reaching for shell grep/rg, or the Grep tool, to answer a "where/how does X work" question — literal-text search misses renamed, refactored, or differently-worded implementations. Plain grep is still fine for a single already-known literal string inside one file you already have open.' +description: 'Use before shell grep/rg whenever a question is about meaning, not literal text — locating where or how something is implemented, mapping an unfamiliar codebase, finding every call site before a refactor or rename, or checking whether a repo already does X before building it again. colgrep-mcp''s tools (search, find_files, expand, index_status/index_build/index_clear/index_prune, list_indexes, doctor) run hybrid semantic + keyword search over code units (functions, classes, methods, docs sections), not raw lines, so natural-language queries surface relevant code even when it shares no vocabulary with the query. Anti-pattern: reaching for shell grep/rg, or the Grep tool, to answer a "where/how does X work" question — literal-text search misses renamed, refactored, or differently-worded implementations. Plain grep is still fine for a single already-known literal string inside one file you already have open.' --- # colgrep-search @@ -28,6 +28,7 @@ serve (extensionless or lock files, an inverted match). | "Is this repo indexed? Will search be slow?" | `index_status` | `path` | | "This repo is large and cold" | `index_build` then `search` | `path` | | "What's already indexed here?" | `list_indexes` / `doctor` | — | +| "The index store is huge / which indexes are dead?" | `list_indexes` then `index_prune` | `stale_only=true`; prune is a dry run until `dry_run=false, confirm=true` | Never pass `pattern` alone with an empty `query` — pair a semantic query with `pattern`, don't replace it. See `../../server/colgrep_mcp/guide.md` @@ -102,6 +103,32 @@ Worked example — "what calls `parse_config`, before I change its signature?": {"tool": "find_files", "arguments": {"query": "tests for parse_config", "pattern": "parse_config", "include": ["*test*"]}} ``` +### housekeeping — "clean up colgrep's index store" + +Every path ever searched keeps an index, so the store fills with removed +worktrees, scratch directories and subdirectories of projects indexed later. +Run `list_indexes(stale_only=true)` to see the `orphaned` (path gone), +`machine_state` (temp, cache, hidden tree) and `shadowed` (inside another +indexed project) indexes with their sizes; then `index_prune()` for a dry +run grouped by class; then the same call with `dry_run=false, confirm=true` +and exactly the classes you reviewed. Add `"cold"` to `classes` only after +deciding that live projects with at most one search and no use for `days` +should go too. Never delete under the store by hand — the tool re-checks +each directory's `project.json` before removing it. The `housekeeping` MCP +prompt carries this sequence. + +```json +{"tool": "list_indexes", "arguments": {"stale_only": true}} +``` + +```json +{"tool": "index_prune", "arguments": {}} +``` + +```json +{"tool": "index_prune", "arguments": {"dry_run": false, "confirm": true, "classes": ["orphaned", "machine_state", "shadowed"]}} +``` + ## Details For the full argument surface — query composition, `fixed_string`/ From 2ae95d2104c8980bbefe46141fea163bf41bd738 Mon Sep 17 00:00:00 2001 From: Eliott Jacopin Date: Sun, 13 Sep 2026 13:20:55 +0200 Subject: [PATCH 7/9] build(repo): exclude merge subjects ending in (PR #N) from the changelog MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `gh pr merge --subject " (PR #N)"` repeats the step's subject on the merge commit, so v0.3.0 and v0.4.0 each listed one entry twice; in v0.3.0 GitHub had also copied the PR title into the merge body, which commitizen parses as a third line. commitizen applies `changelog_pattern` with `re.match` to the whole message, so the negative lookahead stops at the first newline rather than `$` — a `$`-anchored first attempt passed the subject-only form and changed nothing. Oracle (`maintainer-policy` §Drift tests): `cz changelog --dry-run 0.3.0..0.4.0` with incremental mode off, before and after. Before reproduces the committed CHANGELOG.md byte for byte; the diff after removes exactly the `(PR #7)` line of v0.4.0 and the `(PR #6)` merge's two lines of v0.3.0, nothing else. `probe_cz_check.sh` still meets every expectation; `test_changelog.py` gains a guard that fails against the v0.4.0 pattern and against the `$`-anchored attempt. Co-Authored-By: Claude Fable 5.1 --- server/pyproject.toml | 8 +++++++- server/tests/test_changelog.py | 32 ++++++++++++++++++++++++++++++++ 2 files changed, 39 insertions(+), 1 deletion(-) diff --git a/server/pyproject.toml b/server/pyproject.toml index d48cfa8..3527153 100644 --- a/server/pyproject.toml +++ b/server/pyproject.toml @@ -87,7 +87,13 @@ bump_pattern = "^((BREAKING[\\-\\ ]CHANGE|feat|fix|refactor|perf|test|docs|build bump_map = { "^.+!$" = "MAJOR", "^BREAKING[\\-\\ ]CHANGE" = "MAJOR", "^feat" = "MINOR", "^fix" = "PATCH", "^perf" = "PATCH" } change_type_order = ["BREAKING CHANGE", "feat", "fix", "perf"] commit_parser = "^(?Pfeat|fix|refactor|perf|test|docs|build|chore|release)\\((?P[a-z][a-z0-9-]*)\\)(?P!)?:\\s(?P.*)?" -changelog_pattern = "^(feat|fix|perf|BREAKING CHANGE)(\\(.+\\))?(!)?" +# A merge commit's subject repeats the PR's `feat`/`fix` subject with a +# `(PR #N)` suffix (landing-and-release §Landing a PR), so without the lookahead +# every release listed that entry twice (v0.3.0, v0.4.0). commitizen applies +# this with `re.match` to the whole message, subject and body +# (`changelog.generate_tree_from_commits`), so the lookahead stops at the first +# newline rather than at `$`. +changelog_pattern = "^(?!.*\\(PR #\\d+\\)(\\n|$))(feat|fix|perf|BREAKING CHANGE)(\\(.+\\))?(!)?" change_type_map = { feat = "Added", fix = "Fixed", perf = "Changed" } message_template = "{{change_type}}({{scope}}): {{message}}" diff --git a/server/tests/test_changelog.py b/server/tests/test_changelog.py index a46b5fd..02bc4ae 100644 --- a/server/tests/test_changelog.py +++ b/server/tests/test_changelog.py @@ -11,6 +11,7 @@ from __future__ import annotations import re +import tomllib from pathlib import Path REPO_ROOT = Path(__file__).resolve().parents[2] @@ -24,3 +25,34 @@ def test_version_headings_use_the_commitizen_shape(): assert headings, "CHANGELOG.md has no `## ` version headings" bad = [h for h in headings if not _HEADING_RE.match(h)] assert not bad, f"headings commitizen's incremental mode cannot parse: {bad}" + + +def test_changelog_pattern_skips_merge_subjects_and_keeps_step_subjects(): + """`gh pr merge --subject " (PR #N)"` makes the merge commit repeat + the step's subject; both matched `changelog_pattern` and every release since + v0.3.0 listed the entry twice. commitizen applies the pattern with `re.match` + to the *whole* message, body included (`changelog.generate_tree_from_commits`, + probed on commitizen 4.18), so the messages here carry a body: a pattern that + anchors the suffix with `$` passes the subject-only form and still lists the + merge twice. Regression test: the `(PR #N)` cases fail against the pattern + that shipped in v0.4.0 and against that `$`-anchored first attempt.""" + config = tomllib.loads((REPO_ROOT / "server" / "pyproject.toml").read_text()) + pattern = re.compile(config["tool"]["commitizen"]["customize"]["changelog_pattern"]) + body = "\n\nWhy the change exists.\n\nCo-Authored-By: someone " + + kept = [ + "feat(plugin): ship the search policy, grep redirect and worktree reap as plugin hooks", + "fix(plugin): treat the system temp directory and appdata as machine state in the hook gate", + "perf(search): skip the per-lock resolve", + "feat(search)!: rename limit", + ] + skipped = [ + "feat(plugin): ship the search policy, grep redirect and worktree reap as plugin hooks (PR #7)", + "fix(repo): replace local machine paths (PR #12)", + "docs(reports): close the harness_wiring reports index after the v0.4.0 release", + "refactor(index): split the renderer", + ] + for message in (*kept, *(s + body for s in kept)): + assert pattern.match(message), message + for message in (*skipped, *(s + body for s in skipped)): + assert not pattern.match(message), message From 73b47bf3c17ef8a4508cfa9b2b3699e2359bc737 Mon Sep 17 00:00:00 2001 From: Eliott Jacopin Date: Sun, 13 Sep 2026 13:24:54 +0200 Subject: [PATCH 8/9] fix(index): treat the posix /tmp as machine state in the classifier and the hook gate MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The read-only dogfood of the classifier on the maintainer's store showed /private/tmp — indexed with 2 434 units, the project every session scratchpad folds into — as a live project: `tempfile.gettempdir()` is `/var/folders/.../T` on macOS, so nothing named /tmp as machine state. `/tmp` is a system temp directory on every POSIX system whatever the per-user one is; it joins the roots in both the hook and the server, so the drift test that pins the two lists equal stays green and the hook's grep gate fails open under /tmp the way it already does under the per-user temp directory. The new test failed against the old roots on macOS (gettempdir and /tmp differ there); on Linux the two coincide and it passed before the fix. Co-Authored-By: Claude Fable 5.1 --- hooks/colgrep_policy.py | 14 ++++++++++++-- server/colgrep_mcp/store.py | 11 ++++++++++- server/tests/test_store.py | 11 +++++++++++ 3 files changed, 33 insertions(+), 3 deletions(-) diff --git a/hooks/colgrep_policy.py b/hooks/colgrep_policy.py index 4f60817..9d1de0f 100644 --- a/hooks/colgrep_policy.py +++ b/hooks/colgrep_policy.py @@ -155,13 +155,23 @@ def machine_state_roots(home: str) -> list: the system temp directory is machine state wherever it lives — on Windows it is `%LOCALAPPDATA%\\Temp`, under the home directory with no dot-prefixed component, which is how the first Windows CI run of these hooks read a - pytest `tmp_path` as a source corpus (harness_wiring PR #7). + pytest `tmp_path` as a source corpus (harness_wiring PR #7). The server's + `store.machine_state_roots` restates this list; a test pins them equal. """ - return [ + roots = [ os.path.realpath(tempfile.gettempdir()), os.path.join(home, "Library"), os.path.join(home, "AppData"), ] + # The per-user temp directory is not the only one: macOS puts it under + # `/var/folders` while `/private/tmp` stays a system temp directory, and + # the indexes under it (every session scratchpad) read as live projects + # until it was listed here (index_housekeeping README §Status). + if os.name == "posix": + posix_tmp = os.path.realpath("/tmp") + if posix_tmp not in roots: + roots.append(posix_tmp) + return roots def is_source_corpus(path: str) -> bool: diff --git a/server/colgrep_mcp/store.py b/server/colgrep_mcp/store.py index d6c9917..c528c5e 100644 --- a/server/colgrep_mcp/store.py +++ b/server/colgrep_mcp/store.py @@ -135,11 +135,20 @@ def machine_state_roots(home: str) -> list[str]: """The hook's `machine_state_roots` (`hooks/colgrep_policy.py`), restated: the hook is stdlib-only and ships outside this package, so it cannot be imported; a drift test pins the two lists equal (R01 D4).""" - return [ + roots = [ os.path.realpath(tempfile.gettempdir()), os.path.join(home, "Library"), os.path.join(home, "AppData"), ] + # The per-user temp directory is not the only one: macOS puts it under + # `/var/folders` while `/private/tmp` stays a system temp directory, and + # the indexes under it (every session scratchpad) read as live projects + # until it was listed here (index_housekeeping README §Status). + if os.name == "posix": + posix_tmp = os.path.realpath("/tmp") + if posix_tmp not in roots: + roots.append(posix_tmp) + return roots def _under(path: str, root: str) -> bool: diff --git a/server/tests/test_store.py b/server/tests/test_store.py index 5a5eb21..dbbfa6c 100644 --- a/server/tests/test_store.py +++ b/server/tests/test_store.py @@ -7,6 +7,7 @@ import importlib.util import os +import sys import time from pathlib import Path @@ -145,6 +146,16 @@ def test_machine_state_roots_match_the_hook(): assert store.machine_state_roots(home) == hook.machine_state_roots(home) +@pytest.mark.skipif(sys.platform == "win32", reason="no POSIX /tmp on Windows") +def test_machine_state_roots_include_the_posix_tmp(): + """`tempfile.gettempdir()` is `/var/folders/.../T` on macOS, so `/private/tmp` — indexed + with 2 434 units on the maintainer's machine, every scratchpad folding into it — read as a + live project (index_housekeeping README §Status). `/tmp` is a system temp directory on + every POSIX system whatever the per-user one is.""" + home = os.path.realpath(os.path.expanduser("~")) + assert os.path.realpath("/tmp") in store.machine_state_roots(home) + + # --- store_root ---------------------------------------------------------------------------- From 2f07482ca19f145353ed3c8417c372fea6c5d758 Mon Sep 17 00:00:00 2001 From: Eliott Jacopin Date: Sun, 13 Sep 2026 13:25:29 +0200 Subject: [PATCH 9/9] docs(reports): record the index_housekeeping cycle's outcome and PR #9 in its reports index The topic README is where the next cycle reads what landed, what was measured and what was left undone; this fills its Status section before the merge so the branch carries its own record. Co-Authored-By: Claude Fable 5.1 --- __reports__/index_housekeeping/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/__reports__/index_housekeeping/README.md b/__reports__/index_housekeeping/README.md index 94e8822..cf2385f 100644 --- a/__reports__/index_housekeeping/README.md +++ b/__reports__/index_housekeeping/README.md @@ -7,4 +7,4 @@ Seventh campaign of this repository (2026-09-13, after `harness_wiring` 0.4.0): - `00-findings_clear_probe_v0.md` — R02 (index_housekeeping): `colgrep clear` on a gone path exits 1 and leaves the index directory; store layout, mtime and cost facts measured on the maintainer's machine. ## Status -Open. Time-boxed to four hours from 12:54 CEST on 2026-09-13. +Implemented on the branch and open as PR #9 (2026-09-13, within the four-hour box from 12:54 CEST). Every R01 leaf landed as a step commit: the store module and enriched `list_indexes`, `index_prune`, the `doctor` hint, the `housekeeping` prompt, the skill/guide/README text, the `changelog_pattern` fix. Pinned by `server/tests/test_store.py` (classification with injected clock/home/roots, the guarded delete, the roots drift test against the hook) and the `index_prune`/`list_indexes`/`doctor` tests in `test_tools_index.py`. Read-only dogfood on the maintainer's store: 165 indexes, 3.3 GiB, classified 65 orphaned / 8 machine-state / 46 shadowed / 4 cold / 42 live in 321 ms (after the fix below; before it `/private/tmp`, 2 434 units, read as live on macOS because the per-user temp directory sits under `/var/folders` — fixed before merge by adding the POSIX `/tmp` to the shared machine-state roots, hook and server alike). Not run: the real `index_prune` against that store (the PI's call), Codex and Cursor (no CLI here). Release is the next step, from the main checkout.