Skip to content

perf: Speed up media_names_for_ankihub_deck - #1380

Merged
abdnh merged 6 commits into
mainfrom
media-sync-perf
Sep 2, 2026
Merged

perf: Speed up media_names_for_ankihub_deck#1380
abdnh merged 6 commits into
mainfrom
media-sync-perf

Conversation

@abdnh

@abdnh abdnh commented Sep 2, 2026

Copy link
Copy Markdown
Member

Related issues

This came up after my investigation of this community report: https://community.ankihub.net/t/i-accidently-deleted-the-anking-media-folder-from-my-laptop/605714

Proposed changes

This implements some small performance improvements for media sync:

  • Replace _media_referenced_by_notes (which extracts media references from the live Anki collection) with a call to ankihub_db.media_names_for_ankihub_deck().
  • Refactor media_names_for_ankihub_deck to avoid holding all note rows in memory.

How to reproduce

Media sync should work normally. This is mostly a performance improvement (but see note below).

Further comments

  • Locally deleted notes now count and their media gets downloaded. I think this is acceptable considering that the sync process restores deleted notes if they get updates.
  • This is unlikely to be the cause of the community report (it's most likely an IO issue because removing the .exists() check apparently unblocked the sync).

@abdnh
abdnh marked this pull request as ready for review September 2, 2026 12:01
@abdnh
abdnh requested a review from RisingOrange September 2, 2026 12:01
@RisingOrange

RisingOrange commented Sep 2, 2026

Copy link
Copy Markdown
Collaborator

The db.py refactor is a good change on its own and the tests are stronger for dropping the mocks. But swapping the data source in media_sync.py reintroduces INTP-338, so I'd split this PR: land the db.py work and drop the media_sync.py change.

_media_referenced_by_notes was added by #1230 for that ticket. Protected fields (and tag-based protection) only exist locally, so any user who protects a media-heavy field gets its media downloaded into an empty field again — not just the batch-edit repro in the ticket. Passing the protected-fields config in doesn't rescue it, since a field protected after it was populated still holds content that does need downloading. Same root cause as the locally-deleted-notes case you mention: media need is a property of the user's state, not AnkiHub's. Server-side filtering can't distinguish those cases either.

None of which the code says anywhere, which is what made that function look safe to swap out — so it's worth adding a comment on it recording why it reads the collection rather than ankihub_db.

The db.py half is worth landing on its own though: on a synthetic 20k-note, 10-field collection (~157 MB of field text), media_names_for_ankihub_deck goes from 0.714s / 193 MB peak to 0.657s / 0.7 MB, and menu.py:508 gets that on the upload path for free.

And for what it's worth, keeping the collection scan costs almost nothing: on that same collection the existing col.get_note() loop measures 0.674s / 0.7 MB, against 0.657s for the ankihub_db version.

@abdnh abdnh changed the title perf: Avoid scanning collection on media sync perf: Speed up media_names_for_ankihub_deck Sep 2, 2026
@abdnh

abdnh commented Sep 2, 2026

Copy link
Copy Markdown
Member Author

Thanks. I reverted the change and added a comment.

Originally I made the change because I suspected a deadlock issue caused by non-serialized collection access, but I couldn't confirm that.

@abdnh
abdnh merged commit 478b9bd into main Sep 2, 2026
12 of 13 checks passed
@abdnh
abdnh deleted the media-sync-perf branch September 2, 2026 18:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants