Add Apache DataFusion Comet 1.1.0 release blog post

[andygrove]

Draft blog post for the upcoming Comet 1.1.0 release.

Rendered post

Built from the 379 commits on apache/main since the 1.0.0 tag (2026-08-04), 40 distinct authors, plus the iceberg-writes.md, tuning.md, and spark-versions.md user guides on main.

Native Iceberg writes (experimental) is the headline: the AQE problem, the split-operator plan (IcebergWrite / IcebergCommit), the iceberg-rust native write, fidelity-by-design (manifest metrics re-derived on the JVM with iceberg-java's own code; eligibility as an allowlist), failure/abort semantics, and the accepted divergences. Memory management is the second focus: native allocation accounting, the executor memory usage log for sizing spark.executor.memoryOverhead, the fair_unified share and overcommit fixes, and the simplified on-heap mode. Then Iceberg reads (V3 deletion vectors, native system functions), native Parquet writes on Spark 4.0+, Celeborn native RSS, performance, expanded coverage, plan preview, and upgrade notes.

Open items before this can merge

• Date is a placeholder. Currently 2026-09-25 in the filename and frontmatter — update once the release date is set.
• Spark 3.4 removal. The 1.0.0 post promised removal in 1.1.0, but main still builds and publishes 3.4 binaries and spark-versions.md now says "removal in a future release." Written here as still-deprecated-not-removed to match the code. JDK 11 was removed (#5897), so that half landed.
• No performance numbers. The 1.0.0 post cited the AWS Labs EKS benchmark and 0.17.0 cited "TPC-DS 1TB ~9% faster"; there is nothing comparable here yet. The Iceberg write benchmark (#6038) landed the harness, not results. TPC-DS numbers vs 1.0.0, or native-vs-iceberg-java write timings, would strengthen both the Performance section and the Iceberg pitch.
• Changelog link points at docs/source/changelog/1.1.0.md, which does not exist yet.
• Celeborn framing. Shipped as an unlabeled 8-part series with no dedicated user-guide page, so it is described plainly with its requirements rather than labeled experimental. Confirm whether it should carry an experimental label.
• Commit/contributor counts are from git log 1.0.0..apache/main as of 2026-09-25 and will need refreshing at release time.

[hsiang-c]

👍

[dwsmith1983]

@andygrove the 1.1.0 post covers the Iceberg residual fix and the memory pool fixes. 1.1.0 also closes a long list of other correctness bugs that 1.0.0 users could hit, including wrong decimal SUM results, an ABFS container read returning another container's data, and exchange reuse merging plans that differ. Here is a proposed "Correctness Fixes" section for after "More Iceberg Improvements" or before "Upgrading to 1.1.0". It lists only fixes that affected 1.0.0 users and leaves the rest to the change log.

## Correctness Fixes

1.1.0 fixes a long list of cases where Comet returned different results from Spark, failed where Spark
succeeds, or accepted input that Spark rejects. The Iceberg residual and memory pool fixes are described
above. These are the others most likely to affect 1.0.0 users. The [change log] has the full list.

### Wrong results

- Decimal `SUM` returned NULL, or raised an overflow error under ANSI, when an intermediate sum overflowed
  but the final result fit ([#6041], [@dwsmith1983]).
- After a late shuffle fallback, `avg` could return NULL and `collect_list` / `collect_set` could produce
  mismatched buffers ([#5421], [@sunchao]).
- Exchange reuse could share one shuffle between plans that differ, such as `COUNT(*) + 1` and `COUNT(*) - 1`,
  semi and anti joins, or `explode` and `explode_outer` ([#5470], [@sunchao] and [#5828], [@ErikBPF]).
- Two ABFS containers in the same storage account shared a cached object store, so a read could return the
  other container's data ([#5053], [@peterxcli]).
- Dictionary-encoded values hashed differently from the same values decoded, and a null struct hashed its
  fields, which affected joins, aggregates and shuffle partitioning ([#5757] and [#5754], [@viirya]).
- Parquet field names containing non-ASCII characters that differ only in case read as NULL ([#5602],
  [@comphead]).
- `IN`, `InSet`, nested `=`, `arrays_overlap` and `array_position` now treat `-0.0` and `0.0`, and every NaN
  encoding, as Spark does ([#6073], [@mizulun], [#5235], [@divyankshah] and [#5472], [@sunchao]).
- Decimal to double and float casts were off by one unit in the last place for most `DECIMAL(38,18)` values
  ([#5684], [@peterxcli]).
- String to timestamp casts now follow Spark's parsing rules for short fields, time zones and signed years
  ([#5682] and [#5858], [@peterxcli]).

### Errors Spark raises that Comet did not

- Casts and expressions routed through codegen dispatch could skip ANSI errors raised inside a constant
  subexpression ([#5623], [@andygrove]).
- Rejected `TIMESTAMP_NTZ` casts returned NULL under ANSI instead of raising `CAST_INVALID_INPUT` ([#5752],
  [@peterxcli]).
- Out-of-range Parquet `TIMESTAMP_MILLIS` values, top-level or nested, silently wrapped ([#5177] and [#5740],
  [@peterxcli]).
- Nested Parquet struct, list and map fields now follow Spark's conversion rules instead of returning NULL on
  overflow or accepting values Spark rejects ([#5681], [@peterxcli]).

### Query failures and crashes

- `collect_list` and `collect_set` over nested arguments failed with "column types must match schema types"
  ([#5159], [@andygrove]).
- Native shuffle failed with a 2 GB task serialization error on jobs with very many partitions ([#5392],
  [@parthchandra]).
- A Scala UDF from a user jar failed with a `ClassCastException` ([#5282], [@andygrove]).
- Queries failed with a `ClassCastException` when `spark.shuffle.manager` named Comet's shuffle manager only in
  the session configuration ([#6195], [@andygrove]).
- `rpad` and `lpad` panicked on a NULL length ([#5680], [@peterxcli]).
- Structs with duplicate field names failed the task in native shuffle, and panicked in the native Parquet
  scan ([#5866], [@dwsmith1983] and [#5786], [@ErikBPF]).

### Hangs and resource use

- The JVM hung on exit when an application returned from `main` without calling `spark.stop()` ([#5748],
  [@zhangfengcdt]).
- The native scan busy-polled while waiting on S3 or HDFS reads, keeping one core per task at 100% ([#6219],
  [@mixermt] and [@andygrove]).
- One task could force-spill another task's shuffle buffers, and a failed shuffle write leaked its memory
  reservation ([#5493] and [#5461], [@peterxcli]).

<!-- New reference links. [@andygrove], [@sunchao], [@ErikBPF], [@peterxcli], [@comphead], [@dwsmith1983],
[@parthchandra] and [@zhangfengcdt] are already defined in the post. -->

[@viirya]: https://github.com/viirya
[@mizulun]: https://github.com/mizulun
[@divyankshah]: https://github.com/divyankshah
[@mixermt]: https://github.com/mixermt

[#6041]: https://github.com/apache/datafusion-comet/pull/6041
[#5421]: https://github.com/apache/datafusion-comet/pull/5421
[#5470]: https://github.com/apache/datafusion-comet/pull/5470
[#5828]: https://github.com/apache/datafusion-comet/pull/5828
[#5053]: https://github.com/apache/datafusion-comet/pull/5053
[#5757]: https://github.com/apache/datafusion-comet/pull/5757
[#5754]: https://github.com/apache/datafusion-comet/pull/5754
[#5602]: https://github.com/apache/datafusion-comet/pull/5602
[#6073]: https://github.com/apache/datafusion-comet/pull/6073
[#5235]: https://github.com/apache/datafusion-comet/pull/5235
[#5472]: https://github.com/apache/datafusion-comet/pull/5472
[#5684]: https://github.com/apache/datafusion-comet/pull/5684
[#5682]: https://github.com/apache/datafusion-comet/pull/5682
[#5858]: https://github.com/apache/datafusion-comet/pull/5858
[#5623]: https://github.com/apache/datafusion-comet/pull/5623
[#5752]: https://github.com/apache/datafusion-comet/pull/5752
[#5177]: https://github.com/apache/datafusion-comet/pull/5177
[#5740]: https://github.com/apache/datafusion-comet/pull/5740
[#5681]: https://github.com/apache/datafusion-comet/pull/5681
[#5159]: https://github.com/apache/datafusion-comet/pull/5159
[#5392]: https://github.com/apache/datafusion-comet/pull/5392
[#5282]: https://github.com/apache/datafusion-comet/pull/5282
[#6195]: https://github.com/apache/datafusion-comet/pull/6195
[#5680]: https://github.com/apache/datafusion-comet/pull/5680
[#5866]: https://github.com/apache/datafusion-comet/pull/5866
[#5786]: https://github.com/apache/datafusion-comet/pull/5786
[#5748]: https://github.com/apache/datafusion-comet/pull/5748
[#6219]: https://github.com/apache/datafusion-comet/pull/6219
[#5493]: https://github.com/apache/datafusion-comet/pull/5493
[#5461]: https://github.com/apache/datafusion-comet/pull/5461