perf(sqlite): rebuild search_index family indexes as partial (schema v20) - #944
Merged
Conversation
…ly, WIP) Every search_index row uses one value-column family for its parameter type, but all 15 secondary indexes were updated per insert — NULL columns included, so a token row still paid the date/number/quantity/string/uri B-trees. Write-path instrumentation measured those inserts at 53% of total import time (0.40ms/row). The 9 family indexes become partial (WHERE value_X IS NOT NULL; OR over system/code for token and identifier-type), so each row maintains only the structures relevant to its type. Measured on the 10k inline benchmark (all parameters + FTS): 302s -> 181s, 1.67x, reproduced twice, identical index-row counts. WIP: fresh-create path only. Still owed before a PR: the migration for existing databases, EXPLAIN QUERY PLAN verification that search queries imply the predicates (value comparisons do; audit :missing paths), and the search suite + live battery as the gate.
…v20) Every search_index row populates one value-column family for its parameter type, but each INSERT maintained all fifteen secondary indexes, NULL columns included. Write-path instrumentation measured those inserts at 53% of bulk-import time (0.40ms/row). The nine family indexes become partial (WHERE value_X IS NOT NULL; OR over system/code for token and identifier-type), in fresh creation and as a v19->v20 migration, so each row maintains only its own family's structures. Measured on the 10k inline benchmark (all parameters + FTS): 302s -> 181s, 1.67x, reproduced twice, identical row counts. No configuration and no user-facing behavior change. Search plans verified with EXPLAIN QUERY PLAN: family predicates (value_date >= ?) imply the partial predicates and use their indexes; :missing resolves by entry presence, a (tenant, type, param) prefix probe that idx_search_string_folded serves — it deliberately stays full-width for that role. A dedicated presence index was tried and rejected: its highly-duplicated keys cost ~60s of the 120 saved (242s measured, reproduced), and string_folded already covers the probe.
Code reviewNo issues found. Checked for bugs and CLAUDE.md compliance. |
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
smunini
approved these changes
Sep 4, 2026
This was referenced Sep 4, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Write-path instrumentation on the bulk-import benchmark measured index-row INSERTs at 53% of total import time (0.40 ms/row):
search_indexcarries 15 secondary indexes and every row updated all of them — including the family indexes for value columns that are NULL on that row (a token row still paid the date/number/quantity/string/uri B-trees). This is the configuration-free follow-up promised when #932 was closed, and the "review whether all ~15 secondary indexes earn their write cost" item from @edson's analysis (Lever 4).What
The nine per-family indexes become partial (
WHERE value_X IS NOT NULL; OR over system/code for token and identifier-type), in fresh creation and as a v19→v20 migration. Each row now maintains only its own family's structures. No settings, no user-facing behavior change, identical row counts.Measured
10k inline benchmark (all parameters + FTS, worst case): 302 s → 181 s (1.67×), reproduced twice. Also shortens SQLite write-lock holds, mitigating the
database is lockedfailure at high fan-out (#942).Query-plan verification
value_date >= ?,value_token_code = ?) imply the partial predicates — EXPLAIN QUERY PLAN confirms they keep using their indexes.:missingresolves by entry presence (a(tenant, type, param)prefix probe) served byidx_search_string_folded, which deliberately stays full-width for that role.Gate