fix: don't KeyError on statistics when begin() ran on another thread (#507) - #700
Open
Mukller wants to merge 2 commits into
Open
fix: don't KeyError on statistics when begin() ran on another thread (#507)#700Mukller wants to merge 2 commits into
Mukller wants to merge 2 commits into
Conversation
Mukller
commented
Aug 23, 2026
Mukller
left a comment
Author
There was a problem hiding this comment.
Independently verified this on Windows/CPython 3.13:
- Reverted only the
__init__.pyhunk and ran the new test against unmodified source - it fails exactly as reported (KeyError path), passes with the fix. - Full suite green here too: 142 passed + 15 subtests.
- Read-path audit: lines with direct
statistics[...]reads are only the two fixed innext_action; thedelay_since_first_attemptwrite further down is a plain dict assignment, which is safe on an empty per-thread dict. So the fix covers all read-modify-write sites.
One thing to fix before merge: the reno note says "Initialize the per-thread statistics dict with the full key set instead of an empty dict", but that is the approach this PR deliberately rejected (it would break the documented empty-until-begin contract and the existing pickle assertion). The actual change keeps the empty default and makes next_action read with defaults. As written, the release notes describe a behavior that does not ship.
Suggested replacement for the note body:
fixes:
- |
Don't raise KeyError in ``next_action`` when ``begin()`` ran on a different
thread than the retry loop (for example under Temporal's replay worker).
Each thread's lazily created ``statistics`` view now starts counting from
defaults instead of assuming the keys already exist. See issue #507.Everything else looks good to me.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #507
Root cause
statisticsis a lazily initialized thread-local dict (self._local).begin()populates the four keys (start_time,attempt_number,idle_for,delay_since_first_attempt) on whichever thread starts the call. Under Temporal, the workflow can be replayed so thatbegin()runs on one thread while the retry loop (iter()→next_action()) is re-executed on another. The second thread's_local.statisticsstarts as an empty dict, and:This matches the reporter's traceback exactly and their suspicion ("a different thread is calling next_action than the one that initializes").
I first tried initializing the property with a full default key set, but that breaks the documented contract that statistics "will be empty when the controller has never been ran" (and an existing pickle test asserts
restored.statistics == {}), so this PR keeps the empty-until-begin semantics and makes the write path defensive instead.Changes
next_action()now reads with defaults before incrementing:TestStatisticsKeys::test_iter_on_fresh_thread_after_begin_elsewhere: callsbegin()on the main thread, then drives the publiciter()from a worker thread with a failed outcome.Testing
KeyError: 'idle_for'; passes on this branch.pytest tests/test_tenacity.py -q→ 142 passed, 15 subtests passed (CPython 3.13 / Windows).ruff check/ruff format --checkclean; reno note added underreleasenotes/notes/.