Bump linkedin/iceberg 1.5 version to 1.5.2.17 - #647
Merged
Conversation
… e2e tests Picks up com.linkedin.iceberg v1.5.2.17 (linkedin/iceberg#254), which propagates the delete file replication factor to ORC delete files. Adds spark-3.5 sparkitest coverage exercising the configuration through the full OpenHouse stack via SQL: the write.delete-file-replication table property round-trips through the tables service, and merge-on-read DELETE on an ORC table produces ORC position delete files through the replication-aware write path, for both the table property and the spark.sql.iceberg.delete-file-replication session conf routes. Verification is metadata-based (delete_files manifests) because the itest classpath mixes shaded and unshaded iceberg, which breaks ORC position delete reads with a TypeDescription ClassCastException. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…oopFS verification - Remove explicit "initial-default": null from evolved_dummy_healthy_schema.json test resource; Iceberg 1.5.2.17's SchemaParser.defaultFromJson NPEs on explicit null defaults. This was root-causing 15+ cascaded failures in TablesServiceTest and TablesControllerTest. - Replace metadata-only assertions in DeleteFileReplicationTestSpark with real HadoopFS-based verification that delete files are actually created with the configured replication factor, not just that the property round-trips through table metadata. - Add ReplicationTrackingLocalFileSystem, a custom fs.file.impl override that disables checksums and records the replication factor requested at file-create time (stock RawLocalFileSystem/ChecksumFileSystem silently discard it), since local disks have no native replication concept. - Isolate these two tests into a new deleteFileReplicationTest Gradle task with its own JVM/SparkContext, since fs.file.impl is fixed at SparkContext-creation time and must not leak into the shared catalogTest suite. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Replace the standalone ReplicationTrackingLocalFileSystem with a small RawLocalFileSystem subclass nested in the test. Registering the raw filesystem directly as fs.file.impl keeps ChecksumFileSystem out of the path entirely, so no checksum disabling is needed; the getFileStatus synthesis and setReplication override were unused by the assertions and are dropped. Both public create overloads are still overridden (they are independent entry points on RawLocalFileSystem), and getScheme() must be provided since the raw filesystem does not implement it and Iceberg's locality check calls it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The dedicated deleteFileReplicationTest task is not needed: every
catalogtest class opens its SparkSession in try-with-resources, which
stops the SparkContext when the class finishes, so the replication
tests always build a fresh context and their fs.file.impl override
applies and is discarded with it. Verified by running the full
catalogTest task in one JVM: 64/64 pass including both replication
tests. If a leaked session ever precedes them, the failure is loud
("No file creation was captured"), not a silent false pass.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
cbb330
approved these changes
Jul 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Picks up com.linkedin.iceberg v1.5.2.17 (linkedin/iceberg#254), which propagates the delete file replication factor to ORC delete files.
Adds spark-3.5 sparkitest coverage exercising the configuration through the full OpenHouse stack via SQL: the write.delete-file-replication table property round-trips through the tables service, and merge-on-read DELETE on an ORC table produces ORC position delete files through the replication-aware write path, for both the table property and the spark.sql.iceberg.delete-file-replication session conf routes. Verification is metadata-based (delete_files manifests) because the itest classpath mixes shaded and unshaded iceberg, which breaks ORC position delete reads with a TypeDescription ClassCastException.
Changes
For all the boxes checked, please include additional details of the changes made in this pull request.
Testing Done
For all the boxes checked, include a detailed description of the testing done for the changes made in this pull request.
Additional Information
For all the boxes checked, include additional details of the changes made in this pull request.