Skip to content

Don't cache BGG's transient "please try again" collection response - #139

Open
MSGuzy wants to merge 1 commit into
EmilStenstrom:masterfrom
MSGuzy:fix/bgg-cache-transient-message
Open

Don't cache BGG's transient "please try again" collection response#139
MSGuzy wants to merge 1 commit into
EmilStenstrom:masterfrom
MSGuzy:fix/bgg-cache-transient-message

Conversation

@MSGuzy

@MSGuzy MSGuzy commented Jul 21, 2026

Copy link
Copy Markdown

Summary

CachedHttpClient.get() in scripts/gamecache/http_client.py caches every successful HTTP response unconditionally - including BGG's async "please try again later" placeholder message that the collection endpoint returns while it prepares a stats=1 export.

The bug

  1. BGG's /collection endpoint returns HTTP 202 with:
    <message>Your request for this collection has been accepted and will be processed. Please try again later for access.</message>
    while it builds the export. urlopen doesn't treat 202 as an error, so make_http_request returns this message as a normal response body.
  2. CachedHttpClient.get() writes it into the SQLite cache with status_code = 200 and the full expire_after TTL (24h via Downloader).
  3. BGGClient._make_request sees the "has been accepted" message and retries via sleep_with_backoff_and_jitter - but every retry hits the same cache key, which now holds the placeholder, so it just keeps replaying "still processing" instead of ever re-checking BGG.
  4. Because the retry backoff itself grows unboundedly (10 * 2^tries), this can look like download_and_index.py --cache_bgg has hung - in practice I saw it loop for over an hour, while a direct request to BGG (bypassing the local cache) was returning HTTP 200 with the full collection the whole time.

Fix

Skip the cache write when the response body is BGG's transient message, so every retry actually reaches BGG instead of arguing with its own cache.

Testing

Reproduced the hang locally (collection stuck retrying against a cached placeholder for well over an hour), applied this fix, cleared the cache, and confirmed a fresh run picks up BGG's real response immediately once it's ready, instead of re-caching the placeholder.

BGG's collection endpoint returns HTTP 202 with a message like:

    Your request for this collection has been accepted and will be
    processed. Please try again later for access.

while it prepares a stats-enriched export asynchronously. Since
urlopen doesn't treat 202 as an error, make_http_request returns this
message as a normal response body, and CachedHttpClient.get() was
caching it unconditionally with the full expire_after TTL (24h via
Downloader).

Once cached, every retry in BGGClient._make_request hits the same
cache key and just replays the "still processing" placeholder instead
of re-checking BGG - even after BGG finishes preparing the real
export seconds/minutes later. In practice this can make
download_and_index.py --cache_bgg hang for the entire cache TTL,
because the process is arguing with its own stale cache instead of
BGG.

Fix: skip the cache write when the response body is BGG's transient
message, so every retry actually reaches BGG.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant