Fix PandasArrayExtensionDtype comparison against strings - #8606
Open
cpruijsen wants to merge 1 commit into
Open
Conversation
Pandas routes dtype == "string" through construct_from_string, which asserts that cls.name is a str. name is an instance property here, so ordinary pandas operations on Array2D columns raised AssertionError.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implement
construct_from_stringonPandasArrayExtensionDtype: parsearray[<dtype>]into aPandasArrayExtensionDtype, and raiseTypeErrorfor anything else (how pandas spells "not my dtype" in__eq__). Provenance: pandas extension-dtype docs, and the reproduction onmainin #8467 (pandas 3.0.5).PandasArrayExtensionDtype.nameis an instance property (array[float64]), so the inheritedExtensionDtype.construct_from_stringasserts on apropertyobject. Pandas compares dtypes to strings throughout its internals (is_string_dtype,astype_is_view), so any Array2D/3D/4D/5D column that reaches pandas raisesAssertionErrorondtype == "string",is_string_dtype, anddf.astype(object).This PR does not change
_metadatafrom"value_type"to("value_type",). That is #8375 / #8464, a distinct contract bug.Landing
construct_from_stringand the_metadatatuple in one PR would also stopdtype == "array[float64]"(the dtype's own name) from raising. With only this change,dtype == "string"isFalseanddf.astype(object)works; own-name comparison still hits the_metadataAttributeError. Can fold the tuple in if you would rather land them together.Fixes #8467
Test plan
tests/features/test_array_xd.pyconstruct-from-string round-trip (array[int32]/array[bool]/array[float64])construct_from_stringraisesTypeErrorforstring,int64,array[],array[not_a_dtype],not_an_array[int32]test_table_to_pandas_dtype_compares_to_string:dtype == "string"is False,is_string_dtypeis False,df.astype(object)worksmain(AssertionError: (PandasArrayExtensionDtype, <class 'property'>)) and pass here; fulltests/features/test_array_xd.pyis 114 passed