Skip to content

Fix Databricks tests for GEOGRAPHY and GEOMETRY columns - #1484

Merged
jochenchrist merged 2 commits into
mainfrom
fix/databricks-geography-type
Aug 5, 2026
Merged

Fix Databricks tests for GEOGRAPHY and GEOMETRY columns#1484
jochenchrist merged 2 commits into
mainfrom
fix/databricks-geography-type

Conversation

@jochenchrist

Copy link
Copy Markdown
Contributor

Fixes #1483

datacontract test failed every check of a Databricks model containing a GEOGRAPHY or GEOMETRY column, including checks on unrelated columns:

│ failed │ Check that field 'id' is present │ id │ Could not read model 'geotable': '4326' │

Root cause

ibis's _from_sqlglot_GEOGRAPHY reads the first type parameter as a geometry subtype (the PostGIS spelling GEOGRAPHY(POINT, 4326)), but Databricks declares only the SRID, GEOGRAPHY(4326), so the subtype lookup raises KeyError: '4326'. ibis reflects the whole table in one pass, so con.table(...) throws and _run_model fails all specs of that model.

Fix

New datacontract/engines/ibis/connections/databricks_patch.py, applied in _databricks_connect (following the existing oracle_patch / redshift_patch convention):

  1. GEOGRAPHY / GEOMETRY with a lone numeric parameter hand it to ibis as the SRID. The subtype spellings ibis already understood keep working, and only the Databricks type mapper is touched.
  2. Anything else ibis cannot convert becomes Unknown for that column alone, with a warning naming it, so the rest of the model is still checked. This mirrors _pyspark_table_unconvertible_as_unknown on the Spark-session path.

Also: drop the databricksType custom property

import unity wrote databricksType as a byte-for-byte duplicate of physicalType (and only on top-level columns, never on nested ones). No other native importer does this, and the export it fed already falls back to physicalType verbatim (convert_to_sql_type). Two converter fixes keep export sql --server databricks lossless without it:

  • varchar(n) / char(n) keep their length instead of collapsing to STRING
  • a parameterized type with no mapping (map<string,bigint>, geography(4326)) passes through instead of logging Cannot map type

Export of the fixture contracts before/after differs only in canonical spelling (int -> INT, varchar(255) -> VARCHAR(255)).

Tests

tests/test_test_databricks.py: the SRID parse, the subtype/no-parameter spellings still working, one unrepresentable column degrading to unknown while id keeps !int32, and that connect applies the patch. A fixture restores ibis's originals so the process-global patch does not leak between tests.

🤖 Generated with Claude Code

ibis reads the first GEOGRAPHY/GEOMETRY type parameter as a geometry subtype
(PostGIS's GEOGRAPHY(POINT, 4326)), but Databricks declares only the SRID,
GEOGRAPHY(4326), so reflecting the table raised KeyError: '4326' and every
check of the model failed - including checks on unrelated columns.

Patch ibis's Databricks type mapper to read a lone numeric parameter as the
SRID, and type any column ibis still cannot represent as Unknown so the rest
of the model is still checked, as the pyspark path already does.

Also drop the databricksType custom property from `import unity`: it always
duplicated physicalType, and the export it fed already falls back to
physicalType. So that the export stays lossless, keep varchar(n)/char(n)
lengths and pass through parameterized types with no databricks mapping.

Fixes #1483
@jochenchrist
jochenchrist merged commit f75bff3 into main Aug 5, 2026
17 checks passed
@jochenchrist
jochenchrist deleted the fix/databricks-geography-type branch August 5, 2026 16:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Fail to validate table with GEOGRAPHY type in Databricks

1 participant