fix: snapshot CoW containers so inherited methods see real data - #153
Open
araujof wants to merge 1 commit into
Open
fix: snapshot CoW containers so inherited methods see real data#153araujof wants to merge 1 commit into
araujof wants to merge 1 commit into
Conversation
CopyOnWriteDict/CopyOnWriteList kept their parent dict/list storage empty until the first write, so every inherited method they did not override read an empty container. Reads looked correct because those were overridden; serialization, copying and most operators silently saw nothing, and deepcopy duplicated contents. Copy the wrapped container into real storage at construction and track written/deleted keys separately, so the introspection API is preserved while inherited behaviour is correct by construction. Closes: #152 Signed-off-by: Frederico Araujo <frederico.araujo@ibm.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
CopyOnWriteDictandCopyOnWriteListkept their parentdict/liststorage empty until the first write. Reads looked correct because those methods were overridden — but every inherited method that wasn't saw an empty container. This fixes the class of defect rather than another individual operator.The critical case is silent data loss through serialization.
wrap_payload_for_isolation()is the real production path:That reaches external plugins, which receive payloads via
model_dump()(grpc/client.py:228,unix/client.py:303) — so an out-of-process plugin saw empty containers while an in-process plugin saw real ones.RootModelpayloads such asHttpHeaderPayloadare hit regardless of policy, since_isolate_payloadisolates them unconditionally:model_dump()returned{}, stripping every header.Closes: #152
Approach
Snapshot the wrapped container into real storage at construction, and track written/deleted keys separately so the introspection API (
get_modifications(),get_deleted(),has_modifications()) is preserved.Patching operators one at a time is what left this surface open twice already (#55, then #136). Putting real data in the storage makes inherited behaviour correct by construction — and it is the only option for the critical part: no Python-level override can fix
json.dumps()on a dict subclass (the C encoder reads base storage directly) or Pydantic serialization (fields are annotatedlist/dict, so our type gets no schema hook). I verified both before choosing this design.This also let the read-side overrides be deleted outright, so
memory.pygets simpler rather than more clever: net -60 lines of production code covering strictly more surface.Changes
CopyOnWriteDict: eager snapshot;_modified(insertion-ordered) +_deletedtracking; removed the now-redundant__getitem__/__contains__/__len__/__iter__/get/keys/values/itemsoverrides (inherited versions are now correct, andkeys()/values()/items()return proper dict views instead of generators); added the missingpopitem()and__ior__CopyOnWriteList: eager snapshot;_modifiedflag; removed_materialize/_source/__getitem__/__len__/__iter__/__contains__; added the missing__iadd__/__imul____copy__/__deepcopy__, because the default reconstruction replays items throughappend/__setitem__and would flag an untouched copy as modified_originalattribute — after this change it was written but never read, and retaining it pinned the wrapped object in memory__eq__docstrings that described the removed lazy mechanismFixed behaviours
model_dump()/model_dump_json()/json.dumps()deepcopy()/model_copy(deep=True)/_safe_deepcopy()<<=>>=+*index()count()reversed()+=*=||=popitem()reversed(){}/KeyErrorOne intentional behavior change
Mutating the original container after wrapping is no longer visible through the wrapper. Isolation is now symmetric.
This is unavoidable: fixing serialization requires real data in storage, which is mutually exclusive with reading through to a live original. I believe it is also the correct semantics for an isolation primitive — reading through to a mutating original is a shared-state leak.
The only test encoding the old behaviour was
test_original_dict_mutations_not_reflected, which asserted the opposite of its own name, and justified itself with a comment citing aChainMapthe implementation had long stopped using. I updated it to match its name. Flagging it explicitly since it is a real (if accidental) behaviour change.Performance
Snapshotting up front is modestly more expensive than the lazy wrapper, and I want to be accurate rather than claim a free lunch:
deepcopy(what this avoids)So ~0.1-0.2 us more per wrap, still ~38-45x cheaper than the
deepcopythe class exists to avoid. At a payload-realistic 8 elements it is 0.16 us / 0.24 us. I judged sub-microsecond cost worth correctness on an authorization path; happy to reconsider if that read is wrong.Checks
make lintpasses —ruff check cpexclean,vulturecleanmake testpasses — 1875 passed, 241 skipped (up from 1856; +19 new tests)memory.pyandpolicies.pyCI does not run for
0.1.xPRs (every workflow triggers onbranches: ["main"]only), so the above were verified locally.Notes
19 regression tests added in
TestCopyOnWriteInheritedMethods, deliberately pinning the whole surface — serialization, copying, operators, and both isolation directions — rather than individual operators.