Hash object-dtype storage by content, not PyObject pointers

PROGRESS.md

@@ -1,3 +1,68 @@
+# Issue #907 — population `_storage_parts` hashes object dtype by pointer
+
+Lane: `fix-907-population-stamp-object-storage`, branched from
+`origin/main` at `295130c9f901e08db11457f16dbdee4e2349c5ba` on 2026-09-11.
+Report: `/Users/maxghenis/PolicyEngine/_recovered/scratch-backup/893/lanes/out-907-build-r1.md`.
+
+## State
+
+Implementation complete on `fix-907-population-stamp-object-storage`; ten
+commits, nothing pushed, no new branch, no stash. `decl.py`, `kernel.py`,
+`docs/graph-interface.lock`, every `test_acceptance_*`, `uv.lock`, spec pins
+and evidence JSON are untouched — `shasum -a 256` on `decl.py`/`kernel.py`
+still matches the lock byte for byte.
+
+## Done
+
+- Read `CLAUDE.md` and `docs/shared-constants.md`; confirmed the lane rules.
+- Reproduced #907 directly, and end to end: two independently constructed
+  equal object-dtype Series give different `_storage_parts` value bytes
+  (PyObject addresses), which on `origin/main` surfaces as a **spurious**
+  `PopulationError: Structural node 'n1' changed carried storage in
+  household.tenure_type` for content that did not change.
+- Landed the red regression in `packages/microcosm-graph/tests/`
+  (`test_graph_population.py`, an already-tracked file, so
+  `tools/ci_test_groups.py` needed no change). Final tests against the pre-fix
+  source: 34 failed, 24 passed, 46 deselected; direct exit 1.
+- Fixed `_storage_parts` by routing any materialized object array through a
+  length-prefixed encoding whose body is `store._encode_object_scalar` — the
+  graph package's existing object-leaf codec, the one `ContentStore` writes
+  and reads back. A second, parallel vocabulary would have left a column
+  unequal to its own persisted-and-reloaded self for `pd.NA`, `pd.NaT` and
+  NumPy scalars. Unsupported leaves raise `PopulationError` under the static
+  code `storage-object-leaf` instead of being `repr()`-ed.
+- Kept the masked, numeric and `StringDtype` branches byte-identical, verified
+  by a pre-fix/post-fix byte diff over 20 dtypes and pinned by hex-literal
+  characterization tests. Only object, Categorical, DatetimeTZ, Period and
+  Interval move — every one of them a dtype whose old bytes were addresses.
+- Documented at `storage_equal` what the parts do and do not seal, with
+  regressions pinning the masked half, the numpy-normalization, the new
+  refusals, and a real `ContentStore` frame round trip.
+- Adversarial review (three lenses) found two real defects, both fixed: the
+  refusal missed `UnicodeEncodeError` from a lone-surrogate `str`, and two
+  deliberate normalizations were undocumented.
+- Green: `pytest packages/microcosm-graph` 399 passed; the nine graph-adjacent
+  build/frame/fit/calibrate files 578 passed; `test_graph_population.py` 119
+  passed; `ruff check .` 0; `ruff format --check` on both changed files 0;
+  `tools/ci_test_groups.py --verify` 0; `tools/spec_engine_coverage.py
+  --check` 0 (42156/42156 fields, 41/41 inventory); `tools/graph_acceptance_
+  burndown.py --verify` 0 (green 41, red 0, missing 0).
+
+## Next
+
+- Human review. One item this lane could not complete: the issue asked for a
+  note at `_population_stamp` in `us_runtime/survey_atomic_geography.py`, a
+  module that exists only at the tip of the unmerged
+  `origin/microcosm-us-launch-integration-20260909`. The verified wording —
+  corrected, because the issue's own phrasing was incomplete — is in the lane
+  report for whoever owns that branch.
+- No pinned digest moves. The three `*_population_sha256` stamps on that
+  branch will take new values once this merges; their old values were process
+  addresses, they are recomputed on both sides of every comparison, and
+  `survey_origin_budget` already excludes them from the persisted identity.
+
+---
+
 # F1 portable worker identity — CI crawl fix
 
 ## State

changelog.d/907-object-storage-hashing.fixed.md

@@ -0,0 +1 @@
+Hash object-dtype graph storage by content instead of PyObject pointers, so a population column stays storage-equal to its own persisted-and-reloaded self.

packages/microcosm-graph/src/microcosm/graph/population.py

@@ -26,6 +26,7 @@
     StructuralDelta,
 )
 from .kernel import KernelResult
+from .store import _encode_object_scalar
 
 __all__ = [
     "MassRecord",
@@ -1036,7 +1037,26 @@ def storage_equal(
     right: pd.Series,
     positions: np.ndarray | pd.Series | None = None,
 ) -> bool:
-    """Compare physical values and nullable masks exactly, including float bits."""
+    """Compare physical values and nullable masks exactly, including float bits.
+
+    Dense, string and object leaves are compared by content, so a column stays
+    equal to its own persisted-and-reloaded self.  Object leaves are compared
+    at the ContentStore's own leaf resolution, which is what makes that round
+    trip a fixed point: a NumPy floating leaf compares at ``float64`` width and
+    a NumPy integer leaf by value, because that is what the store decodes back.
+    A leaf outside that vocabulary raises :class:`PopulationError` rather than
+    comparing addresses, so for object columns this predicate is not total.
+
+    Masked storage is not compared by content: it reads ``_data`` under the
+    null mask, where pandas leaves whatever the construction route happened to
+    put, so two content-equal ``Int64`` columns built by different routes can
+    legitimately differ.  A digest folded from these parts is therefore an
+    in-process seal — a statement about what the digest can identify, not about
+    where it may be stored.  Build a cross-reconstruction pin from a content
+    identity of the values instead, together with whatever population-level
+    state the pin has to cover; a frame identity alone does not carry the
+    version, owners, weight kinds, mass ledger or design weights.
+    """
 
     if left.dtype != right.dtype or len(left) != len(right):
         return False
@@ -2671,6 +2691,51 @@ def _rebuild_frame(frame: Frame, tables: Mapping[str, pd.DataFrame]) -> Frame:
     )
 
 
+def _object_storage_values(values: np.ndarray) -> bytes:
+    """Encode an object array as length-prefixed, type-tagged content bytes.
+
+    An object array holds PyObject pointers, so ``tobytes()`` on one
+    serializes addresses: equal content re-materialized in a second array
+    hashes differently, and a comparison across a store round trip or a
+    rebuilt frame can never agree.  Each leaf therefore contributes an 8-byte
+    little-endian body length followed by the ContentStore's own object-scalar
+    body, mirroring the ``StringDtype`` framing above.  Reusing that encoder is
+    what keeps a column equal to its own persisted-and-reloaded self: it is the
+    store's definition of an object leaf's bytes, and its tags keep ``1``,
+    ``1.0``, ``True``, ``"1"`` and ``b"1"`` distinct.  (The executor's
+    kernel-context digest keeps a separate leaf vocabulary in
+    ``executor._update_scalar``, with a ``repr()`` fallback; storage hashing
+    deliberately does not share it.)  Every body
+    carries a tag, so a body is never empty and a zero length stays reserved.
+
+    What it deliberately does not keep distinct is a NumPy integer or float
+    scalar from its Python counterpart: the encoder normalizes ``np.int32(1)``
+    to ``1`` and ``np.float32(1.0)`` to ``1.0`` because the store's decoder
+    hands back the Python form, so a leaf-type-only difference inside an
+    object column is not a storage change.  Refusing it would mean a column
+    could never equal its own persisted-and-reloaded self, which is the defect
+    being fixed.  ``np.timedelta64`` and ``np.datetime64`` leaves are refused
+    outright: timedelta64 is a signedinteger subclass at runtime, and encoding
+    it as an integer would drop the unit and collide with a plain int.
+    """
+
+    payload = bytearray()
+    for value in values:
+        try:
+            body = _encode_object_scalar(value)
+        except (TypeError, UnicodeEncodeError) as error:
+            # Both ways that encoder declines a leaf: TypeError for a type
+            # outside its vocabulary, UnicodeEncodeError for a str that is not
+            # encodable (a lone surrogate).  Fail closed rather than repr() an
+            # unvetted leaf: a repr is neither guaranteed injective nor
+            # guaranteed stable across reconstructions, and calling it can
+            # itself raise.
+            raise PopulationError(f"storage-object-leaf: {error}") from error
+        payload.extend(len(body).to_bytes(8, "little"))
+        payload.extend(body)
+    return bytes(payload)
+
+
 def _storage_parts(series: pd.Series, selected: np.ndarray) -> tuple[bytes, bytes]:
     nulls = series.isna().to_numpy(dtype=np.bool_, copy=False)[selected]
     array = series.array
@@ -2693,6 +2758,11 @@ def _storage_parts(series: pd.Series, selected: np.ndarray) -> tuple[bytes, byte
                 payload.extend(encoded)
         return bytes(payload), np.ascontiguousarray(nulls).tobytes()
     values = series.to_numpy(copy=False)[selected]
+    if values.dtype == object:
+        return (
+            _object_storage_values(values),
+            np.ascontiguousarray(nulls).tobytes(),
+        )
     return (
         np.ascontiguousarray(values).tobytes(),
         np.ascontiguousarray(nulls).tobytes(),

packages/microcosm-graph/src/microcosm/graph/store.py

@@ -329,6 +329,13 @@ def _encode_object_scalar(value: object) -> bytes:
         return bytes([_TAG_PD_NAT])
     if isinstance(value, (bool, np.bool_)):
         return bytes([_TAG_TRUE if bool(value) else _TAG_FALSE])
+    if isinstance(value, (np.timedelta64, np.datetime64)):
+        # timedelta64 subclasses signedinteger at runtime; letting it reach the
+        # integer branch would drop the unit and collide with a plain int.
+        raise TypeError(
+            "Object columns may not carry numpy datetime64 or timedelta64 leaves; "
+            f"found {type(value).__name__}."
+        )
     if isinstance(value, (int, np.integer)):
         return bytes([_TAG_INTEGER]) + str(int(value)).encode("ascii")
     if isinstance(value, (float, np.floating)):