Skip to content

Population stamps hash object-dtype storage by pointer, so they never agree across reconstructions #907

Description

@MaxGhenis

Observed

microcosm.graph.population._storage_parts (packages/microcosm-graph/src/microcosm/graph/population.py, the last branch) hashes a series with no masked-array storage and no StringDtype as np.ascontiguousarray(series.to_numpy(copy=False)[selected]).tobytes(). For an object-dtype series that byte string is the array of PyObject* pointers, not the values. Every consumer that folds it into a digest — survey_atomic_geography._population_stamp and the other _population_stamp helpers in us_runtime — therefore produces a stamp that changes whenever the same content is re-materialized (a fresh reconstruction, a store reload, another process), even though _frame_identity, the mass ledger, owners and weights are all equal.

Reproduced on the invented nineteen-node fixture (test_us_graph_atomic_survey_financial.known_financial_run): two back-to-back reconstruct_atomic_survey_geography passes over the same run give different _population_stamp values for 27 of 88 stamped populations; the only differing component is the strata axis (dtype=object) storage parts. Every in-repo use compares the same objects inside one call (the graph runs end to end and the warm resume="require" replay passes), so nothing in the repository is currently wrong.

Why it matters

The native pilot harness for #893 pinned geography stages with _population_stamp across an export pass and a final verification pass and failed its own seal after a complete 5,615 s CPU run. Anyone persisting that stamp (manifest, receipt, cross-process replay check) will hit the same thing.

Proposal (not done here)

Either (a) document _population_stamp as an in-process mutation seal only and give the object-dtype branch of _storage_parts a content encoding (it already does that for StringDtype: length-prefixed UTF-8 plus the null bitmap), or (b) keep the pointer semantics deliberately and name it so (_in_process_storage_parts) with a content-based sibling for pins. Option (a) changes existing stamp values in one place; whichever way, add a regression that stamps two reconstructions of the same object-dtype strata and expects equality (or inequality, if (b)).

Found while continuing #893 (see its PR body, "11 September, later").

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions