perf(sqlite): bulk-ingest write-path defaults — 64 MiB page cache + 10k WAL autocheckpoint (+65%) - #950
Open
angela-helios wants to merge 2 commits into
Open
perf(sqlite): bulk-ingest write-path defaults — 64 MiB page cache + 10k WAL autocheckpoint (+65%)#950angela-helios wants to merge 2 commits into
angela-helios wants to merge 2 commits into
Conversation
SQLite's default page cache is 2 MiB per connection. Once the search_index B-trees outgrow it — immediately, on any real dataset — every index INSERT and batch commit evicts and rewrites hot pages, so bulk ingest spends its time shuffling the cache instead of building the trees. Measured on the real 31 GB bulk-submit manifest (single worker, identical 7-minute windows on a fresh database): 232/s -> 289/s (+25%), with index-row INSERT cost falling 1.71 -> 1.24 ms per entry and batch commit 1.95 -> 1.72 ms. A larger batch size was also tried and measured slower (205/s at 5,000 entries): bigger transactions overflow the cache and grow the commit-time checkpoint, which is exactly what the larger cache absorbs. The limit is per pooled connection (pool of 10), but a connection's cache only grows with the pages it actually touches, so idle and read-only connections stay small.
Code reviewNo issues found. Checked for bugs and CLAUDE.md compliance. |
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
A bulk-ingest batch writes far more than the default 4 MiB wal_autocheckpoint threshold, so every batch commit also ran a checkpoint and paid the WAL-to-database copy inline — commit cost was 1.72 ms per entry, ~50% of the remaining ingest budget. Fewer, larger checkpoints move the same bytes sequentially: measured on the same real-manifest window, 289/s -> 382/s (+32%), with batch-commit cost falling to 0.96 ms per entry. The -wal file now grows to ~40 MiB between checkpoints.
angela-helios
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two internal SQLite defaults on the write path, measured independently on identical 7-minute windows of the real 31 GB bulk-submit manifest (single worker, fresh database, on top of #949):
1. Page cache: 2 MiB → 64 MiB per connection
SQLite's default page cache is 2 MiB. The
search_indexB-trees outgrow that immediately, so bulk ingest spends its time evicting and rewriting hot pages.A larger batch size was tried first and measured slower (205/s at 5,000 entries): bigger transactions overflow the small cache and grow the commit-time checkpoint. Recorded so it isn't re-run.
2. WAL autocheckpoint: 1,000 pages → 10,000 pages
A bulk batch writes far more than the default 4 MiB threshold, so every batch commit also ran a checkpoint and paid the WAL→database copy inline.
Cost: the
-walfile grows to ~40 MiB between checkpoints.Memory ceiling
The cache limit is per pooled connection (pool of 10 → ~640 MiB theoretical worst case), but a connection's cache only grows with pages it actually touches — in practice the write connection fills it and idle/read-only connections stay small. No configuration is added; both are internal defaults in the same spirit as WAL mode itself.
Stacked
With #944 + #949 + this PR, the full 954,288-entry import measured 51 minutes end to end before the checkpoint change (comment on #947); the +32% here projects to ~42 minutes, single worker, out of the box. Full sqlite persistence suite green (344).