Skip to content

Fix PandasArrayExtensionDtype comparison against strings - #8606

Open
cpruijsen wants to merge 1 commit into
huggingface:mainfrom
cpruijsen:fix/issue-8467
Open

Fix PandasArrayExtensionDtype comparison against strings#8606
cpruijsen wants to merge 1 commit into
huggingface:mainfrom
cpruijsen:fix/issue-8467

Conversation

@cpruijsen

Copy link
Copy Markdown

Summary

Implement construct_from_string on PandasArrayExtensionDtype: parse array[<dtype>] into a PandasArrayExtensionDtype, and raise TypeError for anything else (how pandas spells "not my dtype" in __eq__). Provenance: pandas extension-dtype docs, and the reproduction on main in #8467 (pandas 3.0.5).

PandasArrayExtensionDtype.name is an instance property (array[float64]), so the inherited ExtensionDtype.construct_from_string asserts on a property object. Pandas compares dtypes to strings throughout its internals (is_string_dtype, astype_is_view), so any Array2D/3D/4D/5D column that reaches pandas raises AssertionError on dtype == "string", is_string_dtype, and df.astype(object).

This PR does not change _metadata from "value_type" to ("value_type",). That is #8375 / #8464, a distinct contract bug.

Landing construct_from_string and the _metadata tuple in one PR would also stop dtype == "array[float64]" (the dtype's own name) from raising. With only this change, dtype == "string" is False and df.astype(object) works; own-name comparison still hits the _metadata AttributeError. Can fold the tuple in if you would rather land them together.

Fixes #8467

Test plan

  • tests/features/test_array_xd.py construct-from-string round-trip (array[int32] / array[bool] / array[float64])
  • construct_from_string raises TypeError for string, int64, array[], array[not_a_dtype], not_an_array[int32]
  • test_table_to_pandas_dtype_compares_to_string: dtype == "string" is False, is_string_dtype is False, df.astype(object) works
  • The 11 new cases fail on main (AssertionError: (PandasArrayExtensionDtype, <class 'property'>)) and pass here; full tests/features/test_array_xd.py is 114 passed

Pandas routes dtype == "string" through construct_from_string, which
asserts that cls.name is a str. name is an instance property here, so
ordinary pandas operations on Array2D columns raised AssertionError.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

PandasArrayExtensionDtype has no construct_from_string, so comparing the dtype to any string raises AssertionError

1 participant