Skip to content

chore: remove v0.11.0–v0.12.1 Spark-repr migration notice #1465

Description

@jschoedl

Summary

The temporary Spark-repr migration shim added in PR #1192 (April 2026) has reached its July 2026 expiry date. The in-code comment on _LEGACY_SPARK_REPR_RE explicitly says "remove after July 2026" — that date has passed (today is 2026-08-01).

What to remove

In datacontract/export/sql_type_converter.py:

  1. The _LEGACY_SPARK_REPR_RE constant and its preceding comment (lines 11–13):

    # invalid physicalType from `import spark` in v0.11.0–v0.12.1 (#1048) (e.g. StringType() instead of string)
    # remove after July 2026
    _LEGACY_SPARK_REPR_RE = re.compile(r"^(?:[A-Z][a-zA-Z]*)?Type\(.*\)$")
  2. The legacy-note branch inside _warn_cannot_map_type (lines 30–34):

    if isinstance(raw_type, str) and _LEGACY_SPARK_REPR_RE.match(raw_type):
        warning_message += (
            "\nNote: `datacontract import spark` emitted invalid physicalType values in v0.11.0–v0.12.1 (December 2025 until June 2026). "
            "Re-import datacontracts from these versions to fix."
        )

    Only remove this if block — the rest of _warn_cannot_map_type (the logger.warning call etc.) must stay.

  3. The import re line at the top — but only if re is no longer used anywhere else in the file. Run grep -n '\bre\b' datacontract/export/sql_type_converter.py to verify before removing it.

Context

  • Shim was introduced to help users who imported data contracts with versions v0.11.0–v0.12.1, which emitted invalid Spark physicalType values like StringType() instead of string.
  • The shim appended a user-facing note to the "cannot map type" warning so affected users could re-import to fix their contracts.
  • Enough time has passed (8+ months since v0.12.1) that the migration notice is no longer necessary.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions