feat(index): incremental reindexing — reuse unchanged files' chunks, vectors, and BM25 postings

.please/docs/decisions/0005-per-file-hash-manifest-incremental-reindex.md

@@ -0,0 +1,78 @@
+# ADR 0005 — Incremental reindexing keyed on a per-file content hash (not `mtime_ns`)
+
+- **Status**: Accepted
+- **Date**: 2026-09-04
+- **Deciders**: csp maintainers
+- **Context**: [Issue #84](https://github.com/pleaseai/code-search/issues/84) — parity with upstream semble [#225](https://github.com/MinishLab/semble/pull/225) (partial / incremental reindexing)
+- **Builds on**: [ADR 0002](0002-index-storage-cache-model.md) (global `~/.csp/index/` content-hash cache)
+
+## Context
+
+Upstream semble #225 added **partial reindexing**: when a cached index is stale, only the files
+that changed since it was built are re-chunked and re-embedded; unchanged files keep their
+chunks, their rows in the vector matrix, and their BM25 postings. Upstream detects change per
+file with `mtime_ns` recorded in a `files` manifest (`{mtime_ns, start, count}`), rebuilt the
+sparse side around an own incremental `BM25` class (`add_document` / `remove_document` /
+`set_doc_order`, JSON persistence of per-document term counts + document order), and
+validates the cached artifacts' alignment in `load_previous_for_incremental` before reusing them.
+
+Before this decision csp rebuilt the **whole** index whenever the whole-tree content hash in the
+manifest mismatched the live tree (ADR 0002). csp never used mtimes: the cache-validity oracle
+is a sha256 over `(path, bytes)` of every indexable file, chosen in ADR 0002 because mtimes are
+unreliable across `git checkout`, copies, CI caches, and container mounts.
+
+## Decision
+
+Port the incremental machinery from #225, but key the per-file manifest on a **per-file content
+hash** instead of `mtime_ns`:
+
+- `IndexManifest.files: {indexed_path → {hash, start, count}}` where `hash` is the sha256 (hex)
+  of the file bytes at index time (`indexing::cache::sha256_hex`). `start`/`count` are the
+  file's chunk range in the global chunk list, exactly as upstream.
+- The whole-tree `contentHash` stays as the **fast path**: when it matches, the cache is loaded
+  as before with no per-file work. Only on a mismatch does `load_or_build_index` call
+  `load_previous_for_incremental` and seed `CspIndex::from_path_with_previous` with the
+  previous chunks / vectors / manifest / BM25 index.
+- `create_index_from_path` reads every file once (it already had to for the whole-tree hash),
+  hashes it, and reuses the previous rows when the hash matches the manifest entry; otherwise it
+  re-chunks, re-embeds, and replaces that file's BM25 postings. Files missing from the new walk
+  have their postings removed. Rows are **moved** out of the previous index (no vector copies),
+  and reused rows are not re-normalised, so they stay bit-identical.
+- `Bm25Index` becomes the id-keyed incremental index from upstream `bm25.py` with stable chunk
+  ids `"{indexed_path}:{slot}"` (`indexing::types::make_chunk_id`). `bm25.json` now persists
+  `{version: 2, documents, docOrder}`; postings are rebuilt on load, and a document order that
+  does not describe exactly the persisted documents is rejected.
+- `INDEX_SCHEMA_VERSION` is bumped to **2**. `load_from_disk` rejects any other version (it
+  already did) and additionally rejects component count mismatches between chunks, vectors, and
+  the BM25 document order. `load_previous_for_incremental` fails closed on any structural
+  inconsistency (missing/empty `files`, non-tiling or overlapping ranges, chunk paths that do
+  not match their range, BM25 order ≠ manifest-derived ids, model/chunk-size/schema mismatch),
+  so a full rebuild is always the fallback.
+
+## Alternatives considered
+
+1. **`mtime_ns` like upstream** — rejected: it contradicts the ADR 0002 oracle and would make
+   the per-file decision disagree with the whole-tree decision (a `git checkout` that restores
+   identical bytes bumps mtimes and would force needless re-embedding; a same-second edit could
+   be missed). One oracle, one answer.
+2. **Reuse the whole-tree hash only (status quo)** — rejected: any single-file edit re-embeds
+   the entire repository, which is the cost #225 exists to remove.
+3. **Store per-file hashes only, drop the whole-tree hash** — rejected: the whole-tree hash is a
+   single string compare on the hit path and avoids loading chunks/vectors/BM25 at all when
+   nothing changed; keeping both costs one extra hex string per file in the manifest.
+
+## Consequences
+
+- A stale cache now costs roughly `O(changed files)` embedding work plus one hash pass over the
+  tree, instead of a full re-embed. The hit path is unchanged.
+- Existing v1 caches are rebuilt once (schema bump), then benefit from incremental reuse.
+- The BM25 **scoring** is intentionally unchanged from the previous csp implementation (Lucene
+  IDF with the `(k1+1)` numerator, de-duplicated query terms). Upstream's own class dropped the
+  `(k1+1)` factor and weights repeated query terms by their query frequency. Only the first is
+  rank-neutral: `(k1+1)` is a global constant, but the query-frequency weight varies per term,
+  so for a query with a repeated token the two implementations can order documents differently.
+  That is a pre-existing parity gap, out of scope here and tracked in
+  `.please/docs/references/semble.md` §6.1 item 10.
+  Recorded as an intentional adaptation in `.please/docs/references/semble.md` §6.1.
+- Git sources (`from_git`) are URL+ref keyed and never take the incremental path, matching
+  upstream (which only seeds `from_path`).

.please/docs/decisions/index.md

@@ -8,3 +8,4 @@
 | [0002](0002-index-storage-cache-model.md) | Index Storage & Caching Model: Global `~/.csp/index/` Content-Hash Cache | 2026-06-18 | Accepted |
 | [0003](0003-rewrite-in-rust.md) | Rewrite `@pleaseai/csp` from TypeScript/Bun to Rust | 2026-06-18 | Proposed |
 | [0004](0004-rust-grammar-coverage-language-pack.md) | Rust grammar coverage via `tree-sitter-language-pack` (downloaded parsers) | 2026-06-20 | Accepted |
+| [0005](0005-per-file-hash-manifest-incremental-reindex.md) | Incremental reindexing keyed on a per-file content hash (not `mtime_ns`) | 2026-09-04 | Accepted |

.please/docs/references/semble.md

@@ -77,10 +77,11 @@ The full ranking in `ranking::{boosting,penalties}` is now **wired** into `searc
 | `index/file_walker.py` | `csp/src/indexing/file_walker.rs` | ported (`.cspignore`) | gitignore-aware recursive walk (`ignore` crate idioms) |
 | `index/files.py` | `csp/src/indexing/files.rs` | ported | ext→language map, content-type sets, file status checks |
 | `index/dense.py` | `csp/src/indexing/dense.rs` | ported (real + stub) | `Model` enum, `embed_chunks`, `SelectableBasicBackend` cosine |
-| `index/sparse.py` | `csp/src/indexing/sparse.rs` | ported | `Bm25Index`, `enrich_for_bm25`, selector→mask |
-| `index/create.py` | `csp/src/indexing/create.rs` | ported | build BM25 + dense + chunks from a path |
+| `index/sparse.py` + `index/bm25.py` | `csp/src/indexing/sparse.rs` | ported (incremental) | id-keyed incremental `Bm25Index` (add/remove/doc order, `bm25.json` v2), `enrich_for_bm25`, selector→mask |
+| `index/types.py` | `csp/src/indexing/types.rs` | adapted (hash, not mtime) | `FileManifestEntry {hash,start,count}`, `PreviousIndex::try_new` alignment checks, `make_chunk_id` |
+| `index/create.py` | `csp/src/indexing/create.rs` | ported (incremental) | build BM25 + dense + chunks + `files` manifest from a path, reusing a `PreviousIndex`'s unchanged files |
 | `index/index.py` | `csp/src/indexing/index.rs` | ported | `CspIndex` orchestrator (from_path/from_git/search/find_related/save/load) + `load_or_build_index` |
-| `cache.py` | `csp/src/indexing/cache.rs` | adapted | content-hash cache at `~/.csp/index/` (ADR-0002), 0700 perms |
+| `cache.py` | `csp/src/indexing/cache.rs` + `cache_orchestrator.rs` | adapted | content-hash cache at `~/.csp/index/` (ADR-0002), 0700 perms; `try_reuse` + `load_previous_for_incremental` (ADR-0005) |
 | `search.py` | `csp/src/search.rs` | ported (ranking wired) | hybrid RRF + alpha blend; trait seams |
 | `ranking/weighting.py` | `csp/src/ranking/weighting.rs` | ported | adaptive alpha |
 | `ranking/boosting.py` | `csp/src/ranking/boosting.rs` | ported (wired) | query-type detection + definition/stem/embedded boosts |
@@ -185,24 +186,43 @@ Same contract as semble `tokens.py`:
 ### 4.7 `indexing/sparse.rs` — BM25 + enrichment
 
 - `enrich_for_bm25(chunk)` → `"{content} {stem} {stem} {dir[-3:]}"` — stem repeated twice to
-  up-weight path matches; last 3 parent dir components. `Bm25Index` ports the BM25 scoring
-  (`get_scores(tokens, weight_mask)`). `selector_to_mask(selector, size)` → `Vec<u8>` mask.
+  up-weight path matches; last 3 parent dir components. `selector_to_mask(selector, size)` →
+  `Vec<u8>` mask.
+- `Bm25Index` ports upstream's own incremental `BM25` class (`index/bm25.py`, #225): documents
+  are keyed on a stable chunk id (`"{path}:{slot}"`), `add_document` rejects duplicates,
+  `remove_document` drops a document's postings, and `set_doc_order(ids)` fixes the global
+  chunk-list order that `get_scores(tokens, weight_mask)` output is aligned to (ids not in the
+  corpus score 0). Internally ids are interned to `u32` slots (recycled on removal) and postings
+  are `term → {slot → tf}` so a removal is `O(terms in doc)`. `bm25.json` (v2) persists
+  `{documents: {id → {term → tf}}, docOrder}`; postings are rebuilt on load, and an order that
+  does not describe exactly the persisted documents is rejected. `build(docs)` remains as a
+  positional-id convenience for fixtures.
 
 ### 4.8 `indexing/create.rs` — index construction
 
-`create_index_from_path`: walk → per file: `detect_language`, size/empty gate, read text, store
-path **relative to `display_root`**, `chunk_source`. Then `embed_chunks` → dense matrix; build
-`Bm25Index` over `tokenize(enrich_for_bm25(chunk))`; wrap dense in `SelectableBasicBackend`.
-Empty → error.
+`create_index_from_path(path, options, previous)`: walk → per file: `detect_language`, size
+gate, read bytes, **sha256 the bytes**, store path **relative to `display_root`**. If `previous`
+has a manifest entry for the path with the same hash, the file's chunks and (already-normalised)
+vector rows are **moved** out of the previous index; otherwise the file is decoded, `chunk_source`d,
+its old BM25 slots removed and new ones added (`reindex_file`), and its chunks embedded. Paths in
+the previous manifest that the walk no longer yields have their postings removed. Finally
+`set_doc_order(chunk_ids)` and the rows are wrapped via `SelectableBasicBackend::from_normalized`.
+Returns chunks + BM25 + dense + a `files: FileManifest` (`{hash, start, count}` per file; a valid
+file that yields no chunks gets `count = 0`). Empty → error. Divergence from upstream: the reuse
+key is the content hash, not `mtime_ns` (ADR-0005); the "same vector layout → mutate in place"
+special case is unnecessary because rows are moved, never copied.
 
 ### 4.9 `indexing/index.rs` — `CspIndex` orchestrator
 
 The public façade (parallels `SembleIndex`):
-- `from_path`, `from_git` (git clone into a tempdir; repo-relative chunk paths), `search`
-  (`QueryOptions`), `find_related` (semantic kNN on the seed, same-language, excludes seed),
-  `save` / `load_from_disk` (persists chunks + bm25 + semantic + metadata).
+- `from_path` (= `from_path_with_previous(path, options, None)`), `from_git` (git clone into a
+  tempdir; repo-relative chunk paths), `search` (`QueryOptions`), `find_related` (semantic kNN
+  on the seed, same-language, excludes seed), `save` / `load_from_disk` (persists chunks + bm25 +
+  semantic + `manifest.json` incl. the per-file `files` manifest; `INDEX_SCHEMA_VERSION = 2`;
+  load rejects other versions and chunk/vector/BM25-order count mismatches).
 - `load_or_build_index` (`LoadOrBuildOptions`) — the cache-aware entry the CLI/MCP use: load from
-  `~/.csp/index/<hash>` on a validated hit, else build and persist.
+  `~/.csp/index/<hash>` on a validated hit; on a miss for a local path, seed the rebuild with
+  `load_previous_for_incremental` so unchanged files are reused (ADR-0005), then persist.
 - Builds file→indices and language→indices maps for selectors and stats.
 
 ### 4.10 `search.rs` — hybrid retrieval & fusion
@@ -277,7 +297,11 @@ Ported faithfully (`LazyLock<Regex>` for the static patterns, `RefCell<HashMap>`
   `chunk_size` equals the current `DESIRED_CHUNK_LENGTH_CHARS` (a manifest predating the field
   → `None` → rebuild) **and**, for local sources, the live source-file content hash matches.
   This mirrors upstream `_metadata_matches`, which gained a `chunk_size` check so the 1500→750
-  change auto-invalidates stale caches.
+  change auto-invalidates stale caches. The same `manifest_compatible` check (schema version,
+  chunk size, model id, model kind) gates `load_previous_for_incremental`, which then loads
+  chunks + vectors + BM25 and runs `PreviousIndex::try_new` alignment checks (counts agree,
+  manifest ranges tile the chunk list, chunk paths match their range, BM25 order == manifest-
+  derived ids). Any failure → `None` → full rebuild (mirrors upstream fail-closed behaviour).
 - **Divergence from upstream**: semble uses the OS cache dir (`~/Library/Caches/semble`, XDG,
   `%LOCALAPPDATA%`) + `SEMBLE_CACHE_LOCATION`; csp fixes a global `~/.csp/index/` per ADR-0002.
 
@@ -367,6 +391,20 @@ Clean two-layer split:
 7. **Storage** — fixed `~/.csp/index/` (0700) + `~/.csp/savings.jsonl` (ADR-0002), not the OS
    cache dir / `SEMBLE_CACHE_LOCATION`. `.cspignore` (not `.sembleignore`).
 8. **CLI** — clap; `init` (not `install`/`uninstall`); explicit `mcp` subcommand; adds `index`.
+9. **Incremental reindex keyed on per-file content hash** (ADR-0005) — upstream #225 records
+   `mtime_ns` per file; csp records the sha256 of the file bytes so the per-file decision uses
+   the same oracle as the whole-tree `contentHash` fast path (ADR-0002).
+10. **BM25 scoring unchanged** — csp keeps the Lucene `(k1+1)` numerator and de-duplicates
+    query terms; upstream's own `BM25` (#225) dropped `(k1+1)` and multiplies each term's
+    contribution by the query term frequency. The `(k1+1)` factor is a single global constant,
+    so dropping it is rank-neutral. The query-tf factor is **not** — it reweights terms against
+    one another whenever a query tokenises to a repeated token, which the identifier-aware
+    tokenizer makes common (`getUserById getUser` repeats `get` and `user`). Ranks can
+    therefore differ from upstream: for query tokens `[a, a, b]` with per-term contributions
+    `a → 1.0` (doc1) and `b → 1.8` (doc2), upstream scores doc1 `2.0` > doc2 `1.8` while csp
+    scores doc1 `1.0` < doc2 `1.8`. This is a **live parity gap**, not a rank-neutral
+    adaptation, and it predates #225; it is tracked here rather than fixed in the incremental
+    port.
 
 ### 6.2 Open stubs & gaps (verify before claiming runtime parity)
 
@@ -389,6 +427,11 @@ Clean two-layer split:
   manifest as `chunk_size` and validated in `try_reuse`, so the change auto-invalidates stale
   caches (mirrors upstream's added metadata field + cache check). The TS source still uses 1500,
   but per the current direction Python upstream — not TS — is the source of truth.
+- **Partial (incremental) reindexing (#225, `204ae4e`) — ported** ([#84](https://github.com/pleaseai/code-search/issues/84),
+  ADR-0005): per-file `files` manifest, id-keyed incremental `Bm25Index`, `PreviousIndex`
+  reuse in `create_index_from_path`, `load_previous_for_incremental` in the orchestrator,
+  `INDEX_SCHEMA_VERSION` 1 → 2. See §4.7 / §4.8 / §4.9 / §4.14 and §6.1 items 9–10 for the
+  two intentional deviations (hash vs mtime, scoring scale).
 
 ---
 

README.ko.md

@@ -405,7 +405,7 @@ csp find-related src/auth.ts 42 ./my-project
 
 `--content`는 `code` (기본), `docs`, `config`, `all`을 받습니다. `path`를 생략하면 현재 디렉터리를 사용합니다. git URL도 받습니다. `csp`가 `$PATH`에 없다면 `bunx @pleaseai/csp`로 대체하세요.
 
-`csp search`나 `csp find-related`를 `--index` 없이 실행하면, `csp`는 소스와 콘텐츠 선택을 키로 하여 글로벌 캐시 `~/.csp/index/`에 자동으로 인덱싱·캐시합니다. 다음 실행 때 캐시를 재사용하며, 소스 파일이 바뀌면 콘텐츠 해시로 자동 무효화되므로 수동으로 다시 인덱싱할 필요가 없습니다. `--index <경로>`를 지정하면 그 경로를 그대로 사용하고 자동 캐시를 우회합니다. `csp index -o <경로>`는 명시적 영속화 전용(`-o` 필수)이며 자동 캐시와는 독립적입니다.
+`csp search`나 `csp find-related`를 `--index` 없이 실행하면, `csp`는 소스와 콘텐츠 선택을 키로 하여 글로벌 캐시 `~/.csp/index/`에 자동으로 인덱싱·캐시합니다. 다음 실행 때 캐시를 재사용하며, 소스 파일이 바뀌면 콘텐츠 해시로 자동 무효화되므로 수동으로 다시 인덱싱할 필요가 없습니다. 무효화된 캐시는 증분으로 다시 만들어집니다. 내용이 바뀐 파일만 다시 청킹·임베딩하고, 바뀌지 않은 파일은 기존 청크·벡터·BM25 포스팅을 그대로 유지합니다. `--index <경로>`를 지정하면 그 경로를 그대로 사용하고 자동 캐시를 우회합니다. `csp index -o <경로>`는 명시적 영속화 전용(`-o` 필수)이며 자동 캐시와는 독립적입니다.
 
 <details>
 <summary>토큰 절약량 보기</summary>

README.md

@@ -405,7 +405,7 @@ csp find-related src/auth.ts 42 ./my-project
 
 `--content` accepts `code` (default), `docs`, `config`, or `all`. `path` defaults to the current directory when omitted; git URLs are accepted. If `csp` is not on `$PATH`, use `bunx @pleaseai/csp` in its place.
 
-When you run `csp search` or `csp find-related` **without** `--index`, `csp` automatically indexes and caches the source in a global cache at `~/.csp/index/`, keyed by the source and content selection. The cache is reused on the next run and invalidated automatically when the source files change (by content hash), so you do not need to reindex manually. Passing `--index <path>` uses that exact path instead and bypasses the auto-cache. `csp index -o <path>` is for explicit persistence only (`-o` is required) and is independent of the auto-cache.
+When you run `csp search` or `csp find-related` **without** `--index`, `csp` automatically indexes and caches the source in a global cache at `~/.csp/index/`, keyed by the source and content selection. The cache is reused on the next run and invalidated automatically when the source files change (by content hash), so you do not need to reindex manually. A stale cache is rebuilt incrementally: only files whose content changed are re-chunked and re-embedded, and unchanged files keep their existing chunks, vectors, and BM25 postings. Passing `--index <path>` uses that exact path instead and bypasses the auto-cache. `csp index -o <path>` is for explicit persistence only (`-o` is required) and is independent of the auto-cache.
 
 <details>
 <summary>Savings</summary>

crates/csp/src/indexing/cache.rs

@@ -200,6 +200,14 @@ pub(crate) fn compute_content_hash_from_paths(mut files: Vec<(String, PathBuf)>)
     to_hex(&hasher.finalize())
 }
 
+/// sha256 (hex) of raw bytes — the per-file hash recorded in the index's file
+/// manifest for incremental reindexing.
+pub(crate) fn sha256_hex(bytes: &[u8]) -> String {
+    let mut hasher = Sha256::new();
+    hasher.update(bytes);
+    to_hex(&hasher.finalize())
+}
+
 fn update_content_hash(hasher: &mut Sha256, path: &str, content: &[u8]) {
     let len16 = path.encode_utf16().count();
     hasher.update(format!("{len16}:{path}").as_bytes());