Context
Follow-up from #84 / #91. The incremental Bm25Index keeps a Doc { terms: Vec<(String, u32)>, length } per chunk so remove_document can subtract postings. Every term string is owned per document, so a corpus with N chunks × T terms allocates N×T Strings in memory and duplicates them in bm25.json v2 (documents: {id → {term → tf}}).
Proposal
- Intern terms:
terms: Vec<(TermId, u32)> with a shared Vec<String> / HashMap<String, TermId> table on the index.
- Persist the table once in
bm25.json (schema bump to v3 with a v2 loader kept for one release).
Acceptance
- Memory and
bm25.json size measured on a mid-size repo before/after (CodSpeed benchmark added for Bm25Index::load).
- Scores are bit-identical to v2 (existing
sparse/tests.rs pass unchanged).
Context
Follow-up from #84 / #91. The incremental
Bm25Indexkeeps aDoc { terms: Vec<(String, u32)>, length }per chunk soremove_documentcan subtract postings. Every term string is owned per document, so a corpus with N chunks × T terms allocates N×TStrings in memory and duplicates them inbm25.jsonv2 (documents: {id → {term → tf}}).Proposal
terms: Vec<(TermId, u32)>with a sharedVec<String>/HashMap<String, TermId>table on the index.bm25.json(schema bump to v3 with a v2 loader kept for one release).Acceptance
bm25.jsonsize measured on a mid-size repo before/after (CodSpeed benchmark added forBm25Index::load).sparse/tests.rspass unchanged).