Add support to configure replication factor of delete files. by shanthoosh · Pull Request #219 · linkedin/iceberg

shanthoosh · 2025-12-01T23:20:08Z

Changes

Adds support to configures the replication factor during the creation of the delete files. Enables the spark jobs to set the HDFS delete replication as part of the writer configuration.

Tests

Added the unit tests for all the introduced changes.
Verified that existing FileIO implementations continue to work.
./gradlew clean && ./gradlew build

sumedhsakdeo

Left a few comments.

sumedhsakdeo

lgtm

teamurko

Thanks @shanthoosh.

/linkedin#229) into sql-wire-in # Conflicts: # spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/TestSparkWriteConf.java

…es (v3.5/1.10) Root cause: the linkedin#219/linkedin#229 delete-file replication override was correctly plumbed all the way to the delete OutputFileFactory (SparkPositionDeltaWrite threads deleteFileReplication into the delete factory; OutputFileFactory puts file-replication-factor into the properties map; HadoopFileIO/HadoopOutputFile read it). But the actual data/delete file writers BYPASS HadoopOutputFile.create() (the only place the factor reaches fs.create): - Parquet: ParquetIO.file(HadoopOutputFile) returns parquet's native HadoopOutputFile.fromPath(path, conf), which opens the file at the file system default replication. - ORC: ORC.newFileWriter -> OrcFile.createWriter(path, options); ORC's PhysicalFsWriter uses fs.getDefaultReplication(path). So on real HDFS the position-delete files silently took dfs.replication regardless of the override. This is a pre-existing gap (present in openhouse-1.5.2 too), only surfaced by real-HDFS validation — the LocalFileSystem test harness ignores replication entirely. Fix (guarded by replication > 0, i.e. zero impact when the feature is unused): - HadoopOutputFile: expose getReplication(). - ParquetIO.file(...) both overloads: when a custom replication factor is set, route through ParquetOutputFile so iceberg's HadoopOutputFile.create() (fs.create with the explicit replication) is used. - ORC.newFileWriter: after creating the (empty) writer, apply the requested replication to the file via fs.setReplication so the blocks written use it. Validated on real HDFS (Hadoop 3.2.1, Spark 3.5.2) via the oh-hadoop-spark docker stack, dfs.replication=3: ORC session delete-file-replication=1 -> delete file %r = 1 ORC TBLPROPERTY write.delete-file-replication=2 -> delete file %r = 2 ORC UPDATE (MoR) session=1 -> delete file %r = 1 Parquet session=1 -> delete file %r = 1 Data files continue to follow dfs.replication (%r = 3), no regression. 1.10 / Spark-3.5 only; core orc/ and parquet/ modules are shared, so the same patch should be replayed onto the 1.11 branch. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LLnVvLYnxjTuH2b5nAHa4A

…es (v3.5/1.10) Root cause: the linkedin#219/linkedin#229 delete-file replication override was correctly plumbed all the way to the delete OutputFileFactory (SparkPositionDeltaWrite threads deleteFileReplication into the delete factory; OutputFileFactory puts file-replication-factor into the properties map; HadoopFileIO/HadoopOutputFile read it). But the actual data/delete file writers BYPASS HadoopOutputFile.create() (the only place the factor reaches fs.create): - Parquet: ParquetIO.file(HadoopOutputFile) returns parquet's native HadoopOutputFile.fromPath(path, conf), which opens the file at the file system default replication. - ORC: ORC.newFileWriter -> OrcFile.createWriter(path, options); ORC's PhysicalFsWriter uses fs.getDefaultReplication(path). So on real HDFS the position-delete files silently took dfs.replication regardless of the override. This is a pre-existing gap (present in openhouse-1.5.2 too), only surfaced by real-HDFS validation — the LocalFileSystem test harness ignores replication entirely. Fix (guarded by replication > 0, i.e. zero impact when the feature is unused): - HadoopOutputFile: expose getReplication(). - ParquetIO.file(...) both overloads: when a custom replication factor is set, route through ParquetOutputFile so iceberg's HadoopOutputFile.create() (fs.create with the explicit replication) is used. - ORC.newFileWriter: after creating the (empty) writer, apply the requested replication to the file via fs.setReplication so the blocks written use it. Validated on real HDFS (Hadoop 3.2.1, Spark 3.5.2) via the oh-hadoop-spark docker stack, dfs.replication=3: ORC session delete-file-replication=1 -> delete file %r = 1 ORC TBLPROPERTY write.delete-file-replication=2 -> delete file %r = 2 ORC UPDATE (MoR) session=1 -> delete file %r = 1 Parquet session=1 -> delete file %r = 1 Data files continue to follow dfs.replication (%r = 3), no regression. 1.10 / Spark-3.5 only; core orc/ and parquet/ modules are shared, so the same patch should be replayed onto the 1.11 branch. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LLnVvLYnxjTuH2b5nAHa4A (cherry picked from commit 06e05cc)

github-actions Bot added SPARK CORE API labels Dec 1, 2025

CI Builder added 2 commits December 2, 2025 10:01

Adding replication as a output file argument

d8e5425

Fix spotless apply-build failure.

364e96a

shanthoosh force-pushed the configure_replication_during_creation branch from 19cada3 to 364e96a Compare December 2, 2025 18:11

Fix checkstyle issues.

9e078c8

shanthoosh force-pushed the configure_replication_during_creation branch from 07b4998 to 9e078c8 Compare December 2, 2025 21:30

CI Builder added 3 commits December 2, 2025 13:36

Fix checkstyle issues.

4f60693

Fix rev-api errors.

9d55eee

Fix rev-api errors.

3596e30

github-actions Bot added BUILD INFRA labels Dec 2, 2025

sumedhsakdeo requested a review from teamurko December 3, 2025 20:40