feat(index): list, classify and prune stale colgrep indexes with a housekeeping prompt

AGENTS.md

@@ -85,6 +85,8 @@ disambiguates which one.
 | `R01` (dev_plugin) | `__reports__/dev_plugin/00-architecture_v0.md` |
 | `R01` (pypi_publication) | `__reports__/pypi_publication/00-architecture_v0.md` |
 | `R01` (harness_wiring) | `__reports__/harness_wiring/00-architecture_v0.md` |
+| `R01` (index_housekeeping) | `__reports__/index_housekeeping/00-architecture_v0.md` |
+| `R02` (index_housekeeping) | `__reports__/index_housekeeping/00-findings_clear_probe_v0.md` |
 
 ## Where the rest went
 

README.md

@@ -106,8 +106,9 @@ claude mcp add colgrep -- uv run --quiet --directory /path/to/colgrep-mcp/server
 | `index_status` | Whether a path is indexed, with which model, where, how big. |
 | `index_build` | Build or refresh an index now, with progress notifications. |
 | `index_clear` | Delete a project's index. Asks for confirmation (elicitation) or requires `confirm=true`. |
-| `list_indexes` | Every indexed project on this machine. |
-| `doctor` | Environment self-check: binary, version, settings, default root. |
+| `index_prune` | Remove orphaned, machine-state, shadowed and (opt-in) cold indexes. Dry run by default; `confirm=true` or elicitation to delete. |
+| `list_indexes` | Every indexed project on this machine, with size, last use, whether the path still exists and who shadows it; `stale_only` filters. |
+| `doctor` | Environment self-check: binary, version, settings, default root; hints at a stale index store. |
 
 `search` defaults to hybrid mode. Pass `pattern` (a regex) to pre-filter units by text before semantic ranking, `include`/`exclude`/`exclude_dir` to scope, `limit` to size the result. Text output is capped by a character budget; the full result is always in `structured_content`.
 
@@ -117,13 +118,13 @@ claude mcp add colgrep -- uv run --quiet --directory /path/to/colgrep-mcp/server
 |:--|:--|
 | `colgrep://guide` | The agent guide: how to compose queries, when to use which tool. |
 | `colgrep://settings` | colgrep's current configuration. |
-| `colgrep://indexes` | Indexed projects. |
+| `colgrep://indexes` | Indexed projects, with size, last use, path-exists and shadowing per index. |
 | `colgrep://status/{+path}` | Index status for a path. |
 | `colgrep://errors` | Error and hint codes with the next step for each. |
 
 ### Prompts
 
-`explore` (knowledge-acquisition loop for a question), `locate` (where a symbol or behaviour lives), `impact` (what a change touches). Each takes an optional `path`.
+`explore` (knowledge-acquisition loop for a question), `locate` (where a symbol or behaviour lives), `impact` (what a change touches) — each takes an optional `path` — and `housekeeping` (review and prune the index store; optional `days`).
 
 ### Skill
 

__reports__/index_housekeeping/00-findings_clear_probe_v0.md

@@ -0,0 +1,70 @@
+---
+type: findings
+topic: index_housekeeping
+date: 2026-09-13
+version: v0
+prior-version: none
+---
+
+# Does `colgrep clear` remove the index of a path that no longer exists? (v0)
+
+Report id `R02` (index_housekeeping). Probe run on 2026-09-13, macOS, colgrep
+1.6.2, on one orphaned index whose `project_path` had no indexed ancestor
+(so a folding clear could not have reached another project either way).
+
+## Result
+
+**No.** `colgrep clear <gone path>` exits 1 without touching the store. Prune
+must remove the index directory itself (R01 D1, C5). The retired machine
+hook's comment ("clear merely errors when the directory no longer exists")
+is confirmed.
+
+## Method
+
+1. Survey the store read-only (`project.json` per index directory; a path is
+   orphaned when `os.path.exists(project_path)` is false): 165 indexes, 65
+   orphaned.
+2. Pick the smallest orphan with **no indexed ancestor** — the orphan first
+   chosen (`__canons__` under a removed worktree) was rejected because its
+   parent worktree was itself indexed, which would have made the probe
+   ambiguous.
+3. Run `status` then `clear` on the gone path; check the directory after.
+
+## Transcript
+
+Paths are the maintainer's, shown with the `/Users/me` placeholder.
+
+```text
+$ colgrep status "/Users/me/src/explore/colgrep_mcp/server/colgrep_mcp" --color never
+Error: No such file or directory (os error 2)
+exit=1
+
+$ colgrep clear "/Users/me/src/explore/colgrep_mcp/server/colgrep_mcp" --color never
+Error: No such file or directory (os error 2)
+exit=1
+
+$ ls "~/Library/Application Support/colgrep/indices/colgrep_mcp-a1887fc5"
+index project.json state.json
+$ ls "~/Library/Application Support/colgrep/indices" | wc -l
+165
+```
+
+## Store facts confirmed on the way
+
+| Fact | Value |
+|:--|:--|
+| Index directory layout | `<store>/<project_name>-<8 hex>/{project.json, state.json, index/, .lock}` |
+| `project.json` | `{"project_path", "project_name", "model"}` — no timestamps |
+| `state.json` | `{"cli_version", "index_format_version", "files": {rel: {content_hash, mtime, size}}, "ignored_files", "search_count", "dirty"}` |
+| `state.json` mtime vs directory mtime | identical to the second on every sampled index, including the three searched during this session — `state.json` is rewritten on each search |
+| Size pass (`os.scandir`, all 165) | 15 ms median of 5 |
+| `project.json` + `state.json` read pass | 25 ms |
+| Indexed pairs where one project is a strict descendant of another | 130 |
+| Orphaned indexes | 65 of 165 |
+
+## Steering
+
+- Prune deletes `<store>/<dir>` directly, guarded by that directory's own
+  `project.json` (R01 C5). Confirmed necessary, not just preferable.
+- `status` also fails on a gone path, so the store root must be derived from
+  a `status` call on a project that *exists* (R01 C1).

__reports__/index_housekeeping/README.md

@@ -0,0 +1,10 @@
+# index_housekeeping — reports
+
+Seventh campaign of this repository (2026-09-13, after `harness_wiring` 0.4.0): give agents the facts and the one tool needed to keep colgrep's index store lean — size, age, path-exists and shadowing per index, an `index_prune` tool with a dry run and grouped candidates, a `housekeeping` prompt, a `doctor` hint — plus the changelog fix carried from the previous cycle. Single-agent, no roadmap tree: step commits on `claude/dreamy-austin-2682cd`.
+
+## Round 00
+- `00-architecture_v0.md` — R01 (index_housekeeping): store discovery from the `Index:` line (C1), the store entry model (C2), the ordered classification (C3), the enriched `list_indexes` (C4), `index_prune` and its deletion guard (C5), the `doctor` hint (C6), the prompt and skill (C7), the changelog pattern (C8); decisions D1–D11; risk register; step-commit plan.
+- `00-findings_clear_probe_v0.md` — R02 (index_housekeeping): `colgrep clear` on a gone path exits 1 and leaves the index directory; store layout, mtime and cost facts measured on the maintainer's machine.
+
+## Status
+Implemented on the branch and open as PR #9 (2026-09-13, within the four-hour box from 12:54 CEST). Every R01 leaf landed as a step commit: the store module and enriched `list_indexes`, `index_prune`, the `doctor` hint, the `housekeeping` prompt, the skill/guide/README text, the `changelog_pattern` fix. Pinned by `server/tests/test_store.py` (classification with injected clock/home/roots, the guarded delete, the roots drift test against the hook) and the `index_prune`/`list_indexes`/`doctor` tests in `test_tools_index.py`. Read-only dogfood on the maintainer's store: 165 indexes, 3.3 GiB, classified 65 orphaned / 8 machine-state / 46 shadowed / 4 cold / 42 live in 321 ms (after the fix below; before it `/private/tmp`, 2 434 units, read as live on macOS because the per-user temp directory sits under `/var/folders` — fixed before merge by adding the POSIX `/tmp` to the shared machine-state roots, hook and server alike). Not run: the real `index_prune` against that store (the PI's call), Codex and Cursor (no CLI here). Release is the next step, from the main checkout.

hooks/colgrep_policy.py

@@ -155,13 +155,23 @@ def machine_state_roots(home: str) -> list:
     the system temp directory is machine state wherever it lives — on Windows
     it is `%LOCALAPPDATA%\\Temp`, under the home directory with no dot-prefixed
     component, which is how the first Windows CI run of these hooks read a
-    pytest `tmp_path` as a source corpus (harness_wiring PR #7).
+    pytest `tmp_path` as a source corpus (harness_wiring PR #7). The server's
+    `store.machine_state_roots` restates this list; a test pins them equal.
     """
-    return [
+    roots = [
         os.path.realpath(tempfile.gettempdir()),
         os.path.join(home, "Library"),
         os.path.join(home, "AppData"),
     ]
+    # The per-user temp directory is not the only one: macOS puts it under
+    # `/var/folders` while `/private/tmp` stays a system temp directory, and
+    # the indexes under it (every session scratchpad) read as live projects
+    # until it was listed here (index_housekeeping README §Status).
+    if os.name == "posix":
+        posix_tmp = os.path.realpath("/tmp")
+        if posix_tmp not in roots:
+            roots.append(posix_tmp)
+    return roots
 
 
 def is_source_corpus(path: str) -> bool:

server/README.md

@@ -52,10 +52,11 @@ Either command is a stdio MCP server; register it in your client as such:
 | `index_status` | Whether a path is indexed, with which model, where, how big. |
 | `index_build` | Build or refresh an index now, with progress notifications. |
 | `index_clear` | Delete a project's index, after confirmation. |
-| `list_indexes` | Every indexed project on this machine. |
+| `index_prune` | Remove orphaned, machine-state, shadowed and (opt-in) cold indexes; dry run by default. |
+| `list_indexes` | Every indexed project on this machine, with size, last use, path-exists and shadowing. |
 | `doctor` | Environment self-check: binary, version, settings, default root. |
 
-Resources (`colgrep://guide`, `colgrep://settings`, `colgrep://indexes`, `colgrep://status/{+path}`, `colgrep://errors`) and prompts (`explore`, `locate`, `impact`) come with it; the guide resource teaches the agent how to compose queries.
+Resources (`colgrep://guide`, `colgrep://settings`, `colgrep://indexes`, `colgrep://status/{+path}`, `colgrep://errors`) and prompts (`explore`, `locate`, `impact`, `housekeeping`) come with it; the guide resource teaches the agent how to compose queries.
 
 ## License
 

server/colgrep_mcp/adapter.py

@@ -117,6 +117,9 @@ def __init__(
         # Set by `_run` after every spawn; for tests only (proving no zombie
         # process survives a `ColgrepTimeout`), never read by production code.
         self._last_proc: asyncio.subprocess.Process | None = None
+        # `store_root()`'s cache: the answer may legitimately be `None`, hence the flag.
+        self._store_root: Path | None = None
+        self._store_root_cached = False
 
     def with_stderr(self, on_stderr: StderrCallback | None) -> ColgrepAdapter:
         """A shallow copy sharing `binary`/`timeout_s` but a different `on_stderr`.
@@ -280,6 +283,33 @@ async def stats(self) -> list[IndexInfo]:
         stdout, _stderr, _rc = await self._run(["--stats"])
         return parse_stats(stdout)
 
+    async def store_root(self, stats: list[IndexInfo] | None = None) -> Path | None:
+        """The index store directory: the parent of the `Index:` line `status` prints for
+        the first indexed project that still exists on disk (index_housekeeping R01 §C1).
+
+        Derived, never hard-coded (the platform path differs on Windows), and
+        cached for the adapter's lifetime: the store does not move under a
+        running server. `stats` may be passed by a caller that already
+        fetched it, saving the spawn. `None` when no indexed project exists
+        on disk — `status` fails on a gone path (R02), so nothing can be
+        asked for its `Index:` line.
+        """
+        if self._store_root_cached:
+            return self._store_root
+        infos = await self.stats() if stats is None else stats
+        for info in infos:
+            if not os.path.isdir(info.project):
+                continue
+            try:
+                st = await self.status(Path(info.project))
+            except ColgrepError:
+                continue
+            if st.index_path:
+                self._store_root = Path(st.index_path).parent
+                break
+        self._store_root_cached = True
+        return self._store_root
+
     async def settings(self) -> dict[str, str]:
         stdout, _stderr, _rc = await self._run(["settings"])
         return parse_settings(stdout)

server/colgrep_mcp/errors.py

@@ -38,6 +38,8 @@ class Code(StrEnum):
     COLGREP_FAILED = "COLGREP_FAILED"
     COLGREP_TIMEOUT = "COLGREP_TIMEOUT"
     BAD_HIT_ID = "BAD_HIT_ID"
+    INDEX_STORE_UNKNOWN = "INDEX_STORE_UNKNOWN"
+    INDEX_STORE_STALE = "INDEX_STORE_STALE"
 
 
 #: One imperative sentence per code, naming the tool/argument to reach for next.
@@ -63,6 +65,8 @@ class Code(StrEnum):
     Code.COLGREP_FAILED: "Read the stderr tail; run `doctor`; retry with simpler arguments.",
     Code.COLGREP_TIMEOUT: "Call `index_build` on the path first, then retry.",
     Code.BAD_HIT_ID: "Use `hit_id` values exactly as returned by `search`: `<abs file>:<line>-<end_line>`.",
+    Code.INDEX_STORE_UNKNOWN: "Run `index_build` on a project that exists on disk so the store can be located; retry.",
+    Code.INDEX_STORE_STALE: "Review them with `index_prune` (a dry run); rerun with `dry_run=false, confirm=true`.",
 }
 
 

server/colgrep_mcp/guide.md

@@ -38,9 +38,15 @@ doesn't share vocabulary with the query.
   whole project: the tool refuses when the project root differs from the
   path you gave and tells you the root to pass explicitly. `index_status`
   shows both `requested_path` and `project`.
-- `list_indexes` — every indexed project on this machine, with sizes.
+- `list_indexes` — every indexed project on this machine: model, units,
+  searches, index size, last use, whether the project path still exists,
+  and which indexed ancestor shadows it. `stale_only=true` keeps only the
+  `orphaned`, `machine_state` and `shadowed` ones (see Housekeeping).
+- `index_prune` — remove stale indexes in one call. Dry run by default;
+  see Housekeeping for the classes and the confirmation flow.
 - `doctor` — environment self-check (binary found, version, default root).
-  Use when a tool call fails for an unclear reason.
+  Use when a tool call fails for an unclear reason. A `hint:` line names a
+  stale index store when it carries orphaned or machine-state indexes.
 
 ## Writing queries
 
@@ -149,6 +155,33 @@ widen `paths`.
 - Do not skip `index_status`/`index_build` on a repository you know is large
   and has never been searched, then be surprised a `search` call times out.
 
+## Housekeeping
+
+colgrep's index store only grows: every path you ever searched keeps its
+index, including scratch directories, removed worktrees and subdirectories
+of a project that was indexed later. `list_indexes` classifies each index:
+
+- `orphaned` — the project path no longer exists on disk.
+- `machine_state` — the path is in the system temp directory, a platform
+  state tree (`~/Library`, `~/AppData`) or a hidden directory under home
+  (`~/.cache`, `~/.claude/...`) and is not a git work tree.
+- `shadowed` — the path lies inside another indexed, existing project
+  (`shadowed_by`): its units are indexed twice, because colgrep only folds
+  a path into an ancestor that was indexed *first*.
+- `cold` — the path exists, has at most `max_searches` searches and was
+  last touched `days` ago or more. Reported by `index_prune` only, opt-in.
+
+`index_prune()` is a dry run over the first three classes, grouped by class
+with sizes and the exact next call. `index_prune(dry_run=false,
+confirm=true, classes=[...])` deletes those index directories; without
+`confirm` it asks through elicitation, exactly like `index_clear`. The
+candidates are recomputed at deletion time and each directory is removed
+only if its own `project.json` still names the candidate, so a stale
+listing can never delete a live project's index. `index_prune` never runs
+`colgrep clear`: on a gone path that command fails, and on a shadowed path
+it would clear the ancestor project instead. The `housekeeping` prompt
+walks the whole sequence.
+
 ## Codes
 
 Every failure and every degraded success carries a stable `[CODE]` prefix,

server/colgrep_mcp/models.py

@@ -88,10 +88,26 @@ class IndexInfo(BaseModel):
     model: str
     units_indexed: int
     search_count: int
+    # Store-derived fields (index_housekeeping R01 §C4); all `None` when the
+    # store root could not be derived or the project is absent from the store.
+    path_exists: bool | None = Field(default=None, description="Whether `project` still exists on disk.")
+    size_bytes: int | None = Field(default=None, description="Bytes under the index directory.")
+    last_modified: str | None = Field(default=None, description="ISO 8601 UTC time of the last search or update.")
+    shadowed_by: str | None = Field(
+        default=None, description="An indexed, existing ancestor project that double-indexes this one's units."
+    )
+    stale: str | None = Field(
+        default=None,
+        description="`orphaned` (path gone), `machine_state` (temp, cache or hidden tree) or `shadowed`; "
+        "`None` for a live project. `cold` needs parameters and is reported by `index_prune` only.",
+    )
 
 
 class IndexList(BaseModel):
     indexes: list[IndexInfo]
+    store_root: str | None = Field(default=None, description="The index store directory, when it could be derived.")
+    total_bytes: int | None = Field(default=None, description="Bytes under the whole store.")
+    total: int | None = Field(default=None, description="Indexed projects before any `stale_only` filter.")
 
 
 class IndexBuildResult(BaseModel):
@@ -111,6 +127,26 @@ class IndexClearResult(BaseModel):
     cleared: bool
 
 
+class PruneCandidate(BaseModel):
+    project: str
+    index_dir: str
+    kind: str = Field(description="`orphaned`, `machine_state`, `shadowed` or `cold` (index_housekeeping R01 §C3).")
+    size_bytes: int
+    last_modified: str
+    search_count: int | None = None
+    shadowed_by: str | None = None
+
+
+class PruneResult(BaseModel):
+    dry_run: bool
+    store_root: str
+    candidates: list[PruneCandidate]
+    total_bytes: int = Field(description="Bytes under every candidate's index directory.")
+    pruned: list[str] = Field(default_factory=list, description="Projects whose index directory was removed.")
+    failed: list[str] = Field(default_factory=list, description="`<project>: <reason>` per candidate left in place.")
+    freed_bytes: int = 0
+
+
 class Doctor(BaseModel):
     colgrep_path: str | None
     version: str | None
@@ -119,3 +155,8 @@ class Doctor(BaseModel):
     root_source: str
     ok: bool
     problems: list[str] = Field(default_factory=list)
+    hints: list[str] = Field(
+        default_factory=list,
+        description="`[CODE] ...` advice that does not make the environment not-ok, e.g. a stale index store "
+        "(index_housekeeping R01 §C6).",
+    )