Don't cache BGG's transient "please try again" collection response - #139
Open
MSGuzy wants to merge 1 commit into
Open
Don't cache BGG's transient "please try again" collection response#139MSGuzy wants to merge 1 commit into
MSGuzy wants to merge 1 commit into
Conversation
BGG's collection endpoint returns HTTP 202 with a message like:
Your request for this collection has been accepted and will be
processed. Please try again later for access.
while it prepares a stats-enriched export asynchronously. Since
urlopen doesn't treat 202 as an error, make_http_request returns this
message as a normal response body, and CachedHttpClient.get() was
caching it unconditionally with the full expire_after TTL (24h via
Downloader).
Once cached, every retry in BGGClient._make_request hits the same
cache key and just replays the "still processing" placeholder instead
of re-checking BGG - even after BGG finishes preparing the real
export seconds/minutes later. In practice this can make
download_and_index.py --cache_bgg hang for the entire cache TTL,
because the process is arguing with its own stale cache instead of
BGG.
Fix: skip the cache write when the response body is BGG's transient
message, so every retry actually reaches BGG.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
CachedHttpClient.get()inscripts/gamecache/http_client.pycaches every successful HTTP response unconditionally - including BGG's async "please try again later" placeholder message that the collection endpoint returns while it prepares astats=1export.The bug
/collectionendpoint returns HTTP 202 with:urlopendoesn't treat 202 as an error, somake_http_requestreturns this message as a normal response body.CachedHttpClient.get()writes it into the SQLite cache withstatus_code = 200and the fullexpire_afterTTL (24h viaDownloader).BGGClient._make_requestsees the "has been accepted" message and retries viasleep_with_backoff_and_jitter- but every retry hits the same cache key, which now holds the placeholder, so it just keeps replaying "still processing" instead of ever re-checking BGG.10 * 2^tries), this can look likedownload_and_index.py --cache_bgghas hung - in practice I saw it loop for over an hour, while a direct request to BGG (bypassing the local cache) was returning HTTP 200 with the full collection the whole time.Fix
Skip the cache write when the response body is BGG's transient message, so every retry actually reaches BGG instead of arguing with its own cache.
Testing
Reproduced the hang locally (collection stuck retrying against a cached placeholder for well over an hour), applied this fix, cleared the cache, and confirmed a fresh run picks up BGG's real response immediately once it's ready, instead of re-caching the placeholder.