perf(sqlite): skip the per-entry FTS table scan on bulk ingest (4.1x) - #949
Merged
Conversation
delete_search_index runs before every indexing pass, including each entry of a bulk import. Its second statement deletes the resource's row from resource_fts — an FTS5 virtual table, where a WHERE on plain columns cannot use an index and scans the entire table. On a fresh bulk load the resource never existed, the DELETE removes nothing, and the scan grows with every inserted row: instrumented on the real 31 GB manifest, it consumed 311 of 366 seconds at 25,000 resources (12.4 ms per entry, ~72% of wall time) and made ingest throughput decay as the table grew. A resource with no search_index rows was never FTS-indexed — every indexed resource carries at least its _id row — so the FTS delete now runs only when the main delete removed something. Same window on the same manifest: 56/s before, 232/s after (4.1x), with the delete cost falling from 311 s to 5 s and throughput no longer decaying. Updates of existing resources still pay one FTS scan; replacing that with an external-content FTS5 table is tracked in #947.
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
This was referenced Sep 5, 2026
Merged
smunini
approved these changes
Sep 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bug
delete_search_indexruns before every indexing pass — including each entry of a bulk import. Its second statement removes the resource's row fromresource_fts, an FTS5 virtual table, where a WHERE on plain columns cannot use an index:On a fresh bulk load the resource never existed, the DELETE removes nothing — and the scan grows with every inserted row, making ingest quadratic in the FTS table size. This is the cause of the throughput decay observed on SQLite imports, and it was the bulk of the "84% unattributed" per-entry cost in #947: instrumented on the real 31 GB manifest, the delete consumed 311 of 366 seconds at 25,000 resources (12.4 ms/entry, ~72% of wall time).
The fix
A resource with no
search_indexrows was never FTS-indexed (every indexed resource carries at least its_idrow), so the FTS delete now runs only when the main delete actually removed something. Three lines, no behavior change: any resource that could have an FTS row still gets it deleted.Measured
Identical 7-minute window, same real manifest, single worker, fresh database:
With the scan gone the per-entry budget is fully attributed: batch commit ≈ 1.9 ms and index-row INSERTs ≈ 1.7 ms lead, everything else is sub-0.3 ms. Updates of existing resources still pay one FTS scan; the structural fix (external-content FTS5) is tracked in #947.
Full sqlite persistence suite green (344).