Skip to content

fix: don't KeyError on statistics when begin() ran on another thread (#507) - #700

Open
Mukller wants to merge 2 commits into
jd:mainfrom
Mukller:fix/statistics-thread-local-defaults
Open

fix: don't KeyError on statistics when begin() ran on another thread (#507)#700
Mukller wants to merge 2 commits into
jd:mainfrom
Mukller:fix/statistics-thread-local-defaults

Conversation

@Mukller

@Mukller Mukller commented Aug 23, 2026

Copy link
Copy Markdown

Fixes #507

Root cause

statistics is a lazily initialized thread-local dict (self._local). begin() populates the four keys (start_time, attempt_number, idle_for, delay_since_first_attempt) on whichever thread starts the call. Under Temporal, the workflow can be replayed so that begin() runs on one thread while the retry loop (iter()next_action()) is re-executed on another. The second thread's _local.statistics starts as an empty dict, and:

self.statistics["idle_for"] += sleep   # KeyError: 'idle_for'

This matches the reporter's traceback exactly and their suspicion ("a different thread is calling next_action than the one that initializes").

I first tried initializing the property with a full default key set, but that breaks the documented contract that statistics "will be empty when the controller has never been ran" (and an existing pickle test asserts restored.statistics == {}), so this PR keeps the empty-until-begin semantics and makes the write path defensive instead.

Changes

  • next_action() now reads with defaults before incrementing:
    stats = self.statistics
    stats["idle_for"] = stats.get("idle_for", 0) + sleep
    stats["attempt_number"] = stats.get("attempt_number", 1) + 1
    A fresh per-thread view therefore starts counting from attempt 1 — the same values a normal run would produce at that point.
  • Regression test TestStatisticsKeys::test_iter_on_fresh_thread_after_begin_elsewhere: calls begin() on the main thread, then drives the public iter() from a worker thread with a failed outcome.

Testing

  • New test fails on unmodified source with exactly the reported KeyError: 'idle_for'; passes on this branch.
  • Full suite: pytest tests/test_tenacity.py -q → 142 passed, 15 subtests passed (CPython 3.13 / Windows).
  • ruff check / ruff format --check clean; reno note added under releasenotes/notes/.

@Mukller Mukller left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independently verified this on Windows/CPython 3.13:

  • Reverted only the __init__.py hunk and ran the new test against unmodified source - it fails exactly as reported (KeyError path), passes with the fix.
  • Full suite green here too: 142 passed + 15 subtests.
  • Read-path audit: lines with direct statistics[...] reads are only the two fixed in next_action; the delay_since_first_attempt write further down is a plain dict assignment, which is safe on an empty per-thread dict. So the fix covers all read-modify-write sites.

One thing to fix before merge: the reno note says "Initialize the per-thread statistics dict with the full key set instead of an empty dict", but that is the approach this PR deliberately rejected (it would break the documented empty-until-begin contract and the existing pickle assertion). The actual change keeps the empty default and makes next_action read with defaults. As written, the release notes describe a behavior that does not ship.

Suggested replacement for the note body:

fixes:
  - |
    Don't raise KeyError in ``next_action`` when ``begin()`` ran on a different
    thread than the retry loop (for example under Temporal's replay worker).
    Each thread's lazily created ``statistics`` view now starts counting from
    defaults instead of assuming the keys already exist. See issue #507.

Everything else looks good to me.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Statistics can be incorrectly initialized inside Temporal contexts

1 participant