Summary
With file fan-out (HFS_BULK_SUBMIT_FILE_CONCURRENCY, #933) set to 8 on the SQLite backend and the full search-parameter registry (no pruning), a bulk import does not just run slowly — it aborts:
Failed to update manifest counts: database is locked
Found and reproduced deterministically by @edson's hoist-prototype campaign (see the "Cutting Bulk Import to 30 Minutes" analysis, §2.2): full registry, 8 concurrent workers, default busy_timeout = 30000. Each batch holds SQLite's exclusive write lock for its whole extraction + ~25-INSERT span; queued workers wait behind up to 7 other holds; the wait exceeds the busy timeout, and the manifest bookkeeping write is the one that gives up, aborting the manifest.
PostgreSQL is not affected (MVCC, row locks). The original #933 benchmarks never hit this because they were always run with search-parameter pruning active — the one configuration that shortens lock holds enough to stay under the timeout.
Why this is a correctness bug, not tuning
An operator who sets HFS_BULK_SUBMIT_FILE_CONCURRENCY=8 on SQLite gets a failed import, not a slower one. Since #932 (the pruning allowlist) was closed by design decision, there is currently no supported configuration that makes 8 workers safe on SQLite with the full registry.
Proposed fixes (combinable)
- Backend-aware effective concurrency: on SQLite, clamp the effective file concurrency (e.g. to 2) regardless of the configured value, with a startup log line explaining why. PostgreSQL keeps the configured value.
- Make the bookkeeping write resilient: the per-file
update_manifest_progress / counts write should retry on SQLITE_BUSY rather than abort the manifest — a progress update is idempotent and losing the whole ingest to a contended counter update is disproportionate.
- Shorten the lock holds structurally: the partial-index work on
search_index (branch perf/partial-search-indexes, measured 1.67× on the same inline benchmark) reduces per-batch time inside the lock, which shrinks the window in which the timeout can be exceeded — mitigation, not a substitute for 1–2.
Repro
Per §3.2 of the analysis: proto/hoist-extraction harness, extraction in-lock arm, 8 workers, no allowlist → fails; same with allowlist or with extraction hoisted → completes.
Summary
With file fan-out (
HFS_BULK_SUBMIT_FILE_CONCURRENCY, #933) set to 8 on the SQLite backend and the full search-parameter registry (no pruning), a bulk import does not just run slowly — it aborts:Found and reproduced deterministically by @edson's hoist-prototype campaign (see the "Cutting Bulk Import to 30 Minutes" analysis, §2.2): full registry, 8 concurrent workers, default
busy_timeout = 30000. Each batch holds SQLite's exclusive write lock for its whole extraction + ~25-INSERT span; queued workers wait behind up to 7 other holds; the wait exceeds the busy timeout, and the manifest bookkeeping write is the one that gives up, aborting the manifest.PostgreSQL is not affected (MVCC, row locks). The original #933 benchmarks never hit this because they were always run with search-parameter pruning active — the one configuration that shortens lock holds enough to stay under the timeout.
Why this is a correctness bug, not tuning
An operator who sets
HFS_BULK_SUBMIT_FILE_CONCURRENCY=8on SQLite gets a failed import, not a slower one. Since #932 (the pruning allowlist) was closed by design decision, there is currently no supported configuration that makes 8 workers safe on SQLite with the full registry.Proposed fixes (combinable)
update_manifest_progress/ counts write should retry onSQLITE_BUSYrather than abort the manifest — a progress update is idempotent and losing the whole ingest to a contended counter update is disproportionate.search_index(branchperf/partial-search-indexes, measured 1.67× on the same inline benchmark) reduces per-batch time inside the lock, which shrinks the window in which the timeout can be exceeded — mitigation, not a substitute for 1–2.Repro
Per §3.2 of the analysis:
proto/hoist-extractionharness, extraction in-lock arm, 8 workers, no allowlist → fails; same with allowlist or with extraction hoisted → completes.