Skip to content

feat(index): list, classify and prune stale colgrep indexes with a housekeeping prompt - #9

Merged
LittleCoinCoin merged 9 commits into
mainfrom
claude/dreamy-austin-2682cd
Sep 13, 2026
Merged

LittleCoinCoin merged 9 commits into
mainfrom
claude/dreamy-austin-2682cd

Conversation

@LittleCoinCoin

@LittleCoinCoin LittleCoinCoin commented Sep 13, 2026 •

Copy link
Copy Markdown
Member

Seventh campaign, index_housekeeping (R01/R02 in __reports__/index_housekeeping/). Gives agents the facts and the one tool needed to keep colgrep's index store lean, plus the changelog fix carried from the harness_wiring cycle.

What changes for a client

Surface dump-and-diff (Client.list_tools() / list_resources() / list_resource_templates() / list_prompts() JSON on main vs this branch):

  • list_indexes: description and input_schema changed (new stale_only argument). Its IndexInfo rows gain path_exists, size_bytes, last_modified, shadowed_by, stale; IndexList gains store_root, total_bytes, total. The four legacy tokens stay at the front of every text block.
  • new tool index_prune(classes, days, max_searches, dry_run=true, confirm=false) — destructive, idempotent, not open-world.
  • doctor: tool listing identical; structured_content gains hints ([INDEX_STORE_STALE] …, [INDEX_STORE_UNKNOWN] …).
  • colgrep://indexes: description changed; serves the same enriched IndexList.
  • new prompt housekeeping(days="30").
  • search, find_files, expand, index_status, index_build, index_clear, the other resources, the status template and the three existing prompts: byte-identical.
  • new codes INDEX_STORE_UNKNOWN, INDEX_STORE_STALE in colgrep://errors.

Why

Measured on the maintainer's machine on 2026-09-13: 165 indexes, 3.3 GiB; 65 orphaned (path gone), 46 shadowed (inside another indexed project), 7 machine-state, 4 cold. list_indexes showed none of that, so a clean-up cost one index_status per guess and one confirmed index_clear per project.

The probe that shaped the design (R02)

colgrep clear <gone path> exits 1 with Error: No such file or directory (os error 2) and leaves the index directory in place, so index_prune deletes store directories itself, guarded by the directory being a direct child of the store root and its project.json naming the candidate at deletion time. It never runs colgrep clear (on a shadowed path that would clear the folding ancestor). The store root is the parent of one Index: line, never a platform path.

Read-only dogfood against the real store

store.index_list over this machine's 165 indexes: 321 ms including the --stats and one status spawn; classification 65 orphaned / 8 machine-state / 46 shadowed / 4 cold / 42 live. Nothing was deleted; the real index_prune was not run.

The first pass classified /private/tmp (2 434 units, the project every session scratchpad folds into) as live: macOS's per-user temp directory sits under /var/folders, so nothing named /tmp as machine state. fix(index): treat the posix /tmp as machine state in the classifier and the hook gate adds it to the shared roots in both the hook and the server; the regression test failed against the old roots on macOS (on Linux the two directories coincide, so it passed there before the fix).

Changelog fix (build(repo))

cz changelog --dry-run 0.3.0..0.4.0 with incremental mode off reproduces the committed CHANGELOG.md before the change; after it the diff removes exactly the (PR #7) line of v0.4.0 and the (PR #6) merge's two lines of v0.3.0 (GitHub had copied the PR title into that merge's body). commitizen matches changelog_pattern against the whole message, so the lookahead stops at the first newline; a $-anchored first attempt changed nothing and the new test in test_changelog.py fails against it.

Portability

First exercise of: the hook gate's roots on Windows with the os.name == "posix" branch skipped, reading a synthetic store under tmp_path on Windows, Path(index_path).parent on a drive-rooted Index: line, os.scandir size walks, shutil.rmtree of store directories. The pure classifier takes home and the machine-state roots explicitly because the Windows runner's temp directory sits under the home directory (R01 risk 2). Read the Windows job, not just the badge.

Gates (from server/)

  • uv run pytest: 306 passed, 1 skipped
  • uv run ruff check: All checks passed!
  • uv run ruff format --check: 46 files already formatted
  • uv run cz check --rev-range main..HEAD: Commit validation: successful!
  • claude plugin validate . and ./dev: Validation passed
  • dev/skills/landing-and-release/scripts/probe_cz_check.sh: all expectations met

Unfinished

Nothing planned in R01 is left out. Not done, deliberately: the real index_prune was not run against the maintainer's store (the PI decides when); no dirtree-rdm roadmap (R01 D10, step commits).

🤖 Generated with Claude Code

LittleCoinCoin and others added 9 commits September 13, 2026 13:04
…ar probe

R01 (index_housekeeping) fixes the contracts before any code: the store
root is derived from one `Index:` line, classification is pure and
ordered, `index_prune` deletes store directories under a `project.json`
guard, and the cycle runs on step commits. R02 records the probe that
forced the deletion design: `colgrep clear` on a gone path exits 1 and
leaves the index directory in place, so the retired hook's comment was
right and a prune cannot delegate to the CLI.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…, age, path and shadowing

An agent deciding which of 165 indexes to keep needed one index_status per
guess: the facts — does the path exist, how big is the index, when was it
last used, does another indexed project already cover it — sit in each
index directory's project.json and state.json and never reached the
client (index_housekeeping R01 §C2–C4). The new store module reads the
store once (40 ms for 165 indexes) and classifies every entry; the store
root is the parent of one `Index:` line rather than a platform path
(R01 §C1, R02: `status` fails on a gone path, so an existing project is
asked). The machine-state roots restate the hook's and a drift test pins
them equal (R01 D4). `list_indexes(stale_only=...)`, `IndexInfo` and
`colgrep://indexes` carry the new fields; the four legacy tokens stay at
the front of each text block.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…wed and cold indexes

Cleaning the store used to cost one confirmed index_clear per project,
dozens of calls on a machine with 65 orphaned indexes (index_housekeeping
R01 §Executive Summary). index_prune classifies the whole store once, lists
the candidates grouped by class as a dry run by default, and with
dry_run=false removes them after confirm=true or an accepted elicitation —
the index_clear flow, now one shared helper. It never runs `colgrep clear`:
that command exits 1 on a gone path (R02) and clears the folding ancestor
on a shadowed one, so the removal is a directory delete guarded by the
directory being a direct child of the store and its project.json naming
the candidate at deletion time (R01 §C5). `cold` is opt-in because a live
but idle project is a judgement call, not dead weight (R01 D8). Both
READMEs' tool tables gain the row so test_readme stays green.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
doctor is the self-check an agent runs when something looks off, so it is
where a stale store gets noticed without a per-session hook — hooks stay
at bare-interpreter cost and carry the search rule only (harness_wiring
R01 §C3; index_housekeeping R01 §C6, D9). The verdict is a `hints` entry,
never a `problems` one: a bloated store does not make the environment
not-ok. Shadowed indexes are listed by list_indexes but not nagged about;
a machine with no index at all gets no hint; a store whose every project
is gone gets INDEX_STORE_UNKNOWN because nothing can be asked for its
`Index:` line (R01 §C1).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…prune and check

The tools alone still leave an agent to invent the order; the prompt fixes
it as list_indexes(stale_only) → index_prune dry run → the confirmed prune
with exactly the reviewed classes → list again (index_housekeeping R01
§C7), with `cold` explicitly a judgement to opt into. It takes no `path`,
so the shared `complete_path` completion is untouched.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…de and readmes

The tools and the prompt exist; the skill is what makes an agent reach for
them, and the guide is where it reads the classes and the confirmation
flow before a non-trivial call (index_housekeeping R01 §C7). The guide's
new section also states why index_prune never runs `colgrep clear` (R02),
so an agent does not "help" by clearing a shadowed path and take the
ancestor project's index with it. The architecture report's C5 wording
now names `store.remove_index_dir`, where the removal actually landed.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
`gh pr merge --subject "<step subject> (PR #N)"` repeats the step's
subject on the merge commit, so v0.3.0 and v0.4.0 each listed one entry
twice; in v0.3.0 GitHub had also copied the PR title into the merge
body, which commitizen parses as a third line. commitizen applies
`changelog_pattern` with `re.match` to the whole message, so the negative
lookahead stops at the first newline rather than `$` — a `$`-anchored
first attempt passed the subject-only form and changed nothing.

Oracle (`maintainer-policy` §Drift tests): `cz changelog --dry-run
0.3.0..0.4.0` with incremental mode off, before and after. Before
reproduces the committed CHANGELOG.md byte for byte; the diff after
removes exactly the `(PR #7)` line of v0.4.0 and the `(PR #6)` merge's
two lines of v0.3.0, nothing else. `probe_cz_check.sh` still meets every
expectation; `test_changelog.py` gains a guard that fails against the
v0.4.0 pattern and against the `$`-anchored attempt.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…nd the hook gate

The read-only dogfood of the classifier on the maintainer's store showed
/private/tmp — indexed with 2 434 units, the project every session
scratchpad folds into — as a live project: `tempfile.gettempdir()` is
`/var/folders/.../T` on macOS, so nothing named /tmp as machine state.
`/tmp` is a system temp directory on every POSIX system whatever the
per-user one is; it joins the roots in both the hook and the server, so
the drift test that pins the two lists equal stays green and the hook's
grep gate fails open under /tmp the way it already does under the
per-user temp directory.

The new test failed against the old roots on macOS (gettempdir and /tmp
differ there); on Linux the two coincide and it passed before the fix.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…in its reports index

The topic README is where the next cycle reads what landed, what was
measured and what was left undone; this fills its Status section before
the merge so the branch carries its own record.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@LittleCoinCoin
LittleCoinCoin merged commit 2a3a4f7 into main Sep 13, 2026
5 checks passed
LittleCoinCoin added a commit that referenced this pull request Sep 13, 2026
…0.5.0 release (PR #10)

Records the landing (PR #9), the release cut by the agent from the main checkout, the publish.yml run and the two end-to-end checks of the new pin.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant