Remove Spark and the JVM from the CLI

[jochenchrist]

Nothing the CLI can reach needs PySpark any more, so no install of it needs Java. datacontract-cli[all] resolves without pyspark, and the Docker image ships no JRE.

The remaining Spark server types (dataframe, Spark-mode databricks) run against a session the caller passes to DataContract(spark=...), which a CLI container cannot supply. PySpark is therefore left to the caller, who cannot have built that session without it, and declaring it was only a way for pip to shadow the build a Databricks Runtime or EMR cluster provides.

Kafka without Spark

spark-sql-kafka-0-10 did the consuming, from_avro / from_json the decoding, and checks ran through ibis.pyspark against a temp view. None of that needed a cluster — a CLI run is single-process either way — while pyspark pulled ~320 JARs and a JDK into every install.

The topic is now consumed with confluent-kafka, decoded in Python (fastavro for Avro), and loaded into DuckDB, which the check engine already queries for every file-based source.

Two behaviours change deliberately, both noted in the changelog:

• An Avro union of more than one non-null type is an error rather than a struct of the union's members. fastavro returns the bare value with no tag, and a column has one type.
• Messages with no value (compaction tombstones) are skipped rather than checked as a row of nulls.

Because messages are held in memory rather than aggregated across Spark partitions, DATACONTRACT_KAFKA_MAX_MESSAGES and DATACONTRACT_KAFKA_TIMEOUT are added. A truncated read says so in the run output.

Exporters without pyspark

spark_exporter built real pyspark.sql.types objects and walked them back into source text. The objects were never used as objects, so the mapping now produces its own type tree and renders from that. to_spark_dict() still returns real schemas via the single function that imports pyspark.

This fixes datacontract export spark and datacontract export great-expectations --engine spark, which failed with a ModuleNotFoundError on a base install since neither is behind a pyspark-carrying extra.

to_spark_schema(), to_struct_type(), to_struct_field() and to_spark_data_type() now return SparkDataType rather than pyspark objects. to_spark_dict() is unchanged; to_pyspark_schema() is new for callers wanting a real StructType.

import unity

StructField.fromJson was only a JSON parser, so datacontract/imports/spark_type_json.py reads the format directly. Verified against pyspark itself: simple_string() matches DataType.simpleString() on all 20 type shapes tested, and the ODCS logical types match the existing mapping on 18 of 20 — the two differences are tinyint and char(5), which make the pyspark path raise AttributeError on PySpark 3.5 because it falls through to a VariantType branch that does not exist before 4.0.

The same class of bug in to_pyspark() is now guarded with a message naming the installed version and the way around it, rather than a bare AttributeError.

Tests and CI

The tests that built a SparkSession are removed: test_test_dataframe.py, and the session-based cases of test_import_spark.py. Two tests in the latter never touched Spark and moved to test_export_sodacl.py, where they belong.

This leaves the dataframe server type and import spark without test coverage. Both still ship and still work; nothing verifies that. Worth a deliberate decision rather than discovering it later.

The JDK setup is gone from both CI jobs, and the Java sections from README.md, AGENTS.md and contributing.md. test_test_delta.py and test_export_spark.py were listed there as needing Java and never did.

Verification

The full suite runs with a meta-path hook that makes every pyspark import raise: 1805 passed, 21 skipped. The four Kafka testcontainer tests pass unchanged, and they no longer need the skipif(is_xdist_worker) guard that Spark's Java client required — 11s under -n 4, down from 49s serially.

Image sizes are measured, by building main and this branch: 2.47 GB to 1.28 GB. The built image was smoke-tested with --version and export spark.