Skip to content

Unify caching mechanisms and separate their management from Interpretations - #743

Closed
eb8680 wants to merge 13 commits into
masterfrom
eb-cache-typeof
Closed

Unify caching mechanisms and separate their management from Interpretations#743
eb8680 wants to merge 13 commits into
masterfrom
eb-cache-typeof

Conversation

@eb8680

@eb8680 eb8680 commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Addresses #633

This PR implements a new weakref-based caching system that unifies and generalizes those introduced in #726 and #594

Specifically, it adds a new context manager effectful.internals.runtime.cache such that within a cache()-delimited block, repeated calls to evaluate with the same expression or data structure and active interpretation are cached.

This new mechanism is backed by a new standalone utility module effectful.internals.weak, based on a similar utility module torch.utils.weak in PyTorch, which implements a couple of different cache data structures generalizing weakref.WeakKeyDictionary:

  • WeakIdKeyDictionary behaves like WeakKeyDictionary (including forbidding keys which cannot be weakref.refed, like tuples and ints) but uses object identity rather than object equality/hashing to discriminate between keys. (This is the main thing provided by torch.utils.weak)
  • IdKeyDictionary, which stores strong references to keys that it discriminates between using identity
  • AutoIdKeyDictionary, which extends WeakIdKeyDictionary and IdKeyDictionary's identity-based semantics by attempting to use a weak reference for a new key and falling back to a strong reference when that is disallowed by the key type, and ensuring that weak reference and strong reference keys cannot conflict with one another.
  • weak_memoize, a helper function that can use these data structures to memoize a single-argument function analogous to functools.cache

The PR also includes a number of small caching-related semantics-preserving performance fixes, and a benchmark derived from a RoboTL issue illustrating both asymptotic and constant-factor improvements on a representative workload.

@eb8680 eb8680 linked an issue Aug 3, 2026 that may be closed by this pull request
@eb8680
eb8680 marked this pull request as ready for review August 4, 2026 16:04
@eb8680
eb8680 requested a review from jfeser August 4, 2026 16:04
@eb8680

eb8680 commented Aug 4, 2026

Copy link
Copy Markdown
Contributor Author

Splitting into two separate PRs for easier review

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Clarify evaluation semantics of shared terms

1 participant