Fix Databricks tests for GEOGRAPHY and GEOMETRY columns - #1484
Merged
Conversation
ibis reads the first GEOGRAPHY/GEOMETRY type parameter as a geometry subtype (PostGIS's GEOGRAPHY(POINT, 4326)), but Databricks declares only the SRID, GEOGRAPHY(4326), so reflecting the table raised KeyError: '4326' and every check of the model failed - including checks on unrelated columns. Patch ibis's Databricks type mapper to read a lone numeric parameter as the SRID, and type any column ibis still cannot represent as Unknown so the rest of the model is still checked, as the pyspark path already does. Also drop the databricksType custom property from `import unity`: it always duplicated physicalType, and the export it fed already falls back to physicalType. So that the export stays lossless, keep varchar(n)/char(n) lengths and pass through parameterized types with no databricks mapping. Fixes #1483
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1483
datacontract testfailed every check of a Databricks model containing aGEOGRAPHYorGEOMETRYcolumn, including checks on unrelated columns:Root cause
ibis's
_from_sqlglot_GEOGRAPHYreads the first type parameter as a geometry subtype (the PostGIS spellingGEOGRAPHY(POINT, 4326)), but Databricks declares only the SRID,GEOGRAPHY(4326), so the subtype lookup raisesKeyError: '4326'. ibis reflects the whole table in one pass, socon.table(...)throws and_run_modelfails all specs of that model.Fix
New
datacontract/engines/ibis/connections/databricks_patch.py, applied in_databricks_connect(following the existingoracle_patch/redshift_patchconvention):GEOGRAPHY/GEOMETRYwith a lone numeric parameter hand it to ibis as the SRID. The subtype spellings ibis already understood keep working, and only the Databricks type mapper is touched.Unknownfor that column alone, with a warning naming it, so the rest of the model is still checked. This mirrors_pyspark_table_unconvertible_as_unknownon the Spark-session path.Also: drop the
databricksTypecustom propertyimport unitywrotedatabricksTypeas a byte-for-byte duplicate ofphysicalType(and only on top-level columns, never on nested ones). No other native importer does this, and the export it fed already falls back tophysicalTypeverbatim (convert_to_sql_type). Two converter fixes keepexport sql --server databrickslossless without it:varchar(n)/char(n)keep their length instead of collapsing toSTRINGmap<string,bigint>,geography(4326)) passes through instead of loggingCannot map typeExport of the fixture contracts before/after differs only in canonical spelling (
int->INT,varchar(255)->VARCHAR(255)).Tests
tests/test_test_databricks.py: the SRID parse, the subtype/no-parameter spellings still working, one unrepresentable column degrading tounknownwhileidkeeps!int32, and that connect applies the patch. A fixture restores ibis's originals so the process-global patch does not leak between tests.🤖 Generated with Claude Code