Skip to content

fix: snapshot CoW containers so inherited methods see real data - #153

Open
araujof wants to merge 1 commit into
0.1.xfrom
fix/copyonwrite-inherited-method-data-loss
Open

fix: snapshot CoW containers so inherited methods see real data#153
araujof wants to merge 1 commit into
0.1.xfrom
fix/copyonwrite-inherited-method-data-loss

Conversation

@araujof

@araujof araujof commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Summary

CopyOnWriteDict and CopyOnWriteList kept their parent dict/list storage empty until the first write. Reads looked correct because those methods were overridden — but every inherited method that wasn't saw an empty container. This fixes the class of defect rather than another individual operator.

The critical case is silent data loss through serialization. wrap_payload_for_isolation() is the real production path:

iso = wrap_payload_for_isolation(P(server_id="s", items=["a","b","c"], args={"k":"v"}))

list(iso.items)              # ['a','b','c']                    reads fine
iso.model_dump()             # {'items': [], 'args': {}}        <-- before: data gone
copy.deepcopy(iso).items     # ['a','b','c','a','b','c']        <-- before: DUPLICATED

That reaches external plugins, which receive payloads via model_dump() (grpc/client.py:228, unix/client.py:303) — so an out-of-process plugin saw empty containers while an in-process plugin saw real ones. RootModel payloads such as HttpHeaderPayload are hit regardless of policy, since _isolate_payload isolates them unconditionally: model_dump() returned {}, stripping every header.

Closes: #152

Approach

Snapshot the wrapped container into real storage at construction, and track written/deleted keys separately so the introspection API (get_modifications(), get_deleted(), has_modifications()) is preserved.

Patching operators one at a time is what left this surface open twice already (#55, then #136). Putting real data in the storage makes inherited behaviour correct by construction — and it is the only option for the critical part: no Python-level override can fix json.dumps() on a dict subclass (the C encoder reads base storage directly) or Pydantic serialization (fields are annotated list/dict, so our type gets no schema hook). I verified both before choosing this design.

This also let the read-side overrides be deleted outright, so memory.py gets simpler rather than more clever: net -60 lines of production code covering strictly more surface.

Changes

  • CopyOnWriteDict: eager snapshot; _modified (insertion-ordered) + _deleted tracking; removed the now-redundant __getitem__ / __contains__ / __len__ / __iter__ / get / keys / values / items overrides (inherited versions are now correct, and keys()/values()/items() return proper dict views instead of generators); added the missing popitem() and __ior__
  • CopyOnWriteList: eager snapshot; _modified flag; removed _materialize / _source / __getitem__ / __len__ / __iter__ / __contains__; added the missing __iadd__ / __imul__
  • Both: explicit __copy__ / __deepcopy__, because the default reconstruction replays items through append/__setitem__ and would flag an untouched copy as modified
  • Both: dropped the _original attribute — after this change it was written but never read, and retaining it pinned the wrapped object in memory
  • Corrected __eq__ docstrings that described the removed lazy mechanism

Fixed behaviours

Before After
model_dump() / model_dump_json() / json.dumps() empty container real contents
deepcopy() / model_copy(deep=True) / _safe_deepcopy() duplicated correct
list < <= > >= + * index() count() reversed() wrong / raised correct
list += *= silent no-op correct
dict | |= popitem() reversed() {} / KeyError correct

One intentional behavior change

Mutating the original container after wrapping is no longer visible through the wrapper. Isolation is now symmetric.

This is unavoidable: fixing serialization requires real data in storage, which is mutually exclusive with reading through to a live original. I believe it is also the correct semantics for an isolation primitive — reading through to a mutating original is a shared-state leak.

The only test encoding the old behaviour was test_original_dict_mutations_not_reflected, which asserted the opposite of its own name, and justified itself with a comment citing a ChainMap the implementation had long stopped using. I updated it to match its name. Flagging it explicitly since it is a real (if accidental) behaviour change.

Performance

Snapshotting up front is modestly more expensive than the lazy wrapper, and I want to be accurate rather than claim a free lunch:

lazy (before) eager (after) deepcopy (what this avoids)
100-item list 0.17 us 0.26 us 9.7 us
100-key dict 0.20 us 0.43 us 19.9 us

So ~0.1-0.2 us more per wrap, still ~38-45x cheaper than the deepcopy the class exists to avoid. At a payload-realistic 8 elements it is 0.16 us / 0.24 us. I judged sub-microsecond cost worth correctness on an authorization path; happy to reconsider if that read is wrong.

Checks

  • make lint passes — ruff check cpex clean, vulture clean
  • make test passes — 1875 passed, 241 skipped (up from 1856; +19 new tests)
  • Doctests pass on memory.py and policies.py
  • CHANGELOG updated

CI does not run for 0.1.x PRs (every workflow triggers on branches: ["main"] only), so the above were verified locally.

Notes

19 regression tests added in TestCopyOnWriteInheritedMethods, deliberately pinning the whole surface — serialization, copying, operators, and both isolation directions — rather than individual operators.

CopyOnWriteDict/CopyOnWriteList kept their parent dict/list storage empty
until the first write, so every inherited method they did not override
read an empty container. Reads looked correct because those were
overridden; serialization, copying and most operators silently saw
nothing, and deepcopy duplicated contents.

Copy the wrapped container into real storage at construction and track
written/deleted keys separately, so the introspection API is preserved
while inherited behaviour is correct by construction.

Closes: #152
Signed-off-by: Frederico Araujo <frederico.araujo@ibm.com>
@araujof
araujof marked this pull request as ready for review August 4, 2026 21:26
@araujof
araujof requested review from jonpspri and terylt as code owners August 4, 2026 21:26
@araujof araujof added bug Something isn't working 0.1.x labels Aug 4, 2026
@araujof araujof added this to CPEX Aug 4, 2026
@github-project-automation github-project-automation Bot moved this to Backlog in CPEX Aug 4, 2026
@araujof araujof moved this from Backlog to In review in CPEX Aug 4, 2026
@araujof araujof added this to the 0.1.3 milestone Aug 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

0.1.x bug Something isn't working

Projects

Status: In review

Development

Successfully merging this pull request may close these issues.

[BUG]: CopyOnWrite containers lose or duplicate data in un-overridden inherited methods (serialization, deepcopy, operators)

2 participants