perf(db): add hot-path indexes for sequence matching, latest-bbox and shared frame lookups

[Acruve15]

Closes #663.

Three query shapes on the detection hot path had no index behind them, so each one seq-scanned a table that keeps growing (~2.6M rows in detections, ~56k in sequences today).

CREATE INDEX CONCURRENTLY ix_sequences_camera_pose_last_seen ON sequences (camera_id, pose_id, last_seen_at);
CREATE INDEX CONCURRENTLY ix_detections_sequence_id_created_at ON detections (sequence_id, created_at);
CREATE INDEX CONCURRENTLY ix_detections_bucket_key ON detections (bucket_key);

Query	Call site	Without index	With index
Recently-seen sequences of a pose (camera_id, pose_id, last_seen_at >)	spatial matching and continuity, on every POST /detections	8 ms	0.07 ms
Latest real bbox of a sequence (sequence_id, created_at DESC, limit 1)	spatial matching, once per candidate sequence per detection	128 ms	0.08 ms
Sibling rows by bucket_key	shared-frame check in DELETE /detections/{id}	48 ms	0.10 ms

Attribution: rows 1 and 3 are the figures from #663 and I have not independently reproduced them. Row 2's query shape is the one I did measure, in more detail below. Happy to benchmark the other two if that is worth it before merge.

The (sequence_id, created_at) index, measured by table size

Index scan vs forced seq scan (enable_indexscan=off) on byte-identical data, so the two are directly comparable, for a 1000-frame sequence:

detections rows	On disk	Seq scan	Index scan	Speedup
200k	29 MB	7.6 ms	0.45 ms	17x
1M	140 MB	18.0 ms	0.53 ms	34x
2M (roughly production today)	266 MB	34.8 ms	0.42 ms	83x
5M	672 MB	63.6 ms	1.14 ms	56x

The seq-scan side grows linearly with the whole table while the index scan stays flat, so this gain keeps widening as detections accumulate. Note this index serves two independent hot paths: the per-detection latest-bbox lookup described in #663, and the sequence reads the player pages through.

One measurement caveat worth recording, because it moves the number by nearly 3x: a first attempt showed 160x, because the test sequence's 1000 rows had been inserted consecutively and so occupied only 14 heap pages (~71 rows/page). Real detections arrive interleaved with the rest of the fleet's writes, so the sequence was rebuilt 1-in-24 across a 24k-row window (321 pages, ~3 rows/page). The table above uses the interleaved layout.

Implementation notes

CONCURRENTLY, in an autocommit block. detections is the highest-write table, and a plain CREATE INDEX holds ACCESS EXCLUSIVE against camera ingest for the whole build. CONCURRENTLY cannot run inside a transaction and env.py wraps the migration run in one, hence op.get_context().autocommit_block().

The migration self-heals. A cancelled CONCURRENTLY build (deploy timeout, dropped connection) leaves an INVALID index behind. if_not_exists would then see the relation and skip creating it, the revision would stamp, and the upgrade would report success while the planner ignored the invalid index, silently leaving these queries on seq scans. So each index is checked in pg_index and dropped first if invalid. The existence check joins pg_class by name rather than casting to regclass, since that cast raises when the relation is absent (the common case) instead of returning no rows.

Declared in models.py too. Per the issue, __table_args__ declarations keep create_all-built test databases matching production.

Verification

• 630 tests pass on the rebased branch, ruff lint and format clean.
• The concurrent build was verified on the real asyncpg path, which was the one genuine unknown: env.py drives an async engine through run_sync, so autocommit_block() flips isolation on a greenlet-backed sync facade. All three indexes come out indisvalid = true after alembic upgrade head.
• The self-heal was verified end to end, not just read: forced all three to indisvalid = false, re-ran the migration, and confirmed all three came back valid.
• The new sync tests were mutation-checked in both directions. My first attempt at this test was tautological and I want to flag it, since it is an easy trap: asserting the indexes exist in pg_indexes passes whether or not models.py declares them, because the test database is migrated. So there are now two tests pinning each side against one canonical list. Removing an index from models.py fails the declaration test; removing it from the migration fails the database test. Both were confirmed to fail.

Coordination with #661

#661 (detection sampling for the player, issue #660) originally carried ix_detections_sequence_id_created_at on its own, since the player's sequence reads need exactly that index. It has been rebased to drop that migration and now depends on this PR, so the index has a single owner and stays independently revertable, which is the point of having split it out of #624.

Worth noting the shared index is probably more valuable here than there: on this path it runs several times per POST /detections, once per candidate sequence.

Sequencing

#624 has now merged, so the dependency the issue mentions is satisfied. That also raises the value of this PR: the continuity pass doubles how often the (camera_id, pose_id, last_seen_at) query runs, and get_latest_with_bbox (added by #624, src/app/crud/crud_detection.py) is exactly the (sequence_id, created_at DESC, limit 1) shape the second index serves, so it is now unindexed on every detection request.

Rebased onto main after #624; no conflicts, and down_revision is still the current head (c4e9f1a2b3d5, since #624 added no migration).

One deployment decision needs a human call: the container start command runs alembic upgrade head, so on the first deploy the concurrent builds run at boot and delay the healthcheck in proportion to table size. Pre-creating the three indexes manually beforehand makes the migration a no-op (if_not_exists), which may be the calmer path for production.

[Acruve15]

Split out of #661, which originally carried ix_detections_sequence_id_created_at on its own for the player's sequence reads. Same index, same name, so it now has one owner here and stays independently revertable.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.73%. Comparing base (5c73e72) to head (1f0a9b7).

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #664   +/-   ##
=======================================
  Coverage   93.73%   93.73%           
=======================================
  Files          59       59           
  Lines        3143     3146    +3     
=======================================
+ Hits         2946     2949    +3     
  Misses        197      197           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.