Spark 3.1: fill ORC initial defaults on the row reader by cbb330 · Pull Request #264 · linkedin/iceberg

cbb330 · 2026-07-23T20:56:58Z

Summary

Opts the Spark 3.1 iterative ORC reader into #263's initial-default projection and supplies Spark's constant conversion to the ID-binding StructReader.

RowDataReader explicitly calls supportsInitialDefaults().
SparkOrcValueReaders converts missing-field defaults into Spark's in-memory representation.
Constant conversion lives in SparkValueConverters, shared with the existing partition-constant path.
ORC scans projecting an initial default are routed away from vectorization until the vectorized reader is implemented.

Scope

Spark 3.1 iterative ORC reads only. The vectorized ORC reader is intentionally deferred; schemas without projected initial defaults continue to vectorize normally.

Other Spark and Flink versions do not opt into #263 and retain their previous projection behavior.

Stack

ORC: bind struct fields by Iceberg field id in generic and Spark 3.1 readers #265 — ID-based StructReader backport
ORC: fill initial defaults for missing id-bound fields #263 — Generic ORC initial-default fill and reader capability gate
This PR — Spark 3.1 iterative ORC initial-default fill

Review this PR's delta against chbush/oh120-orc-default-fill.

Testing Done

Spark 3.1 row tests cover declared, undeclared, and nested defaults
Spark scan tests cover mixed old/new ORC files and physical value/null precedence
Spark filter tests cover top-level and nested defaults, null parents, equality, set, range, and string-prefix predicates
Reader-factory tests verify default projections use iterative reads while ordinary projections remain vectorized
Spotless passes
Spark 3.1 CI passes

mkuchenbecker · 2026-07-31T15:46:23Z

    }
  }
-
-  protected static Object convertConstant(Type type, Object value) {


why is this moved out of base data reater?

mkuchenbecker · 2026-07-31T15:47:34Z

+      // Follow-up: wire initial-default constants into VectorizedSparkOrcReaders. Tables whose
+      // in-memory schema declares defaults must remain on the row reader until this path is wired.


Is there a PR for the vectorized spark orc readers?

mkuchenbecker

the delineation of spark 3.1 and 3.5 is unclear to me why this is implemented in 3.1?

Opt the Spark 3.1 row ORC reader into default filling by supplying Spark's constant converter to the id-binding StructReader, so an absent field that declares an initial-default is materialized in Spark's in-memory representation. Move convertConstant out of BaseDataReader into SparkValueConverters in the data package, mirroring Flink's RowDataUtil. The converter is a value conversion utility rather than reader behavior, and the readers that need it live in the data package. This keeps BaseDataReader package-private instead of widening an internal class to public, and leaves the dependency running source -> data. Mirrors the Generic reader; the visitor call site is unchanged. Vectorized reads are not wired yet, so a table whose schema declares defaults must stay on the row reader. Co-authored-by: Cursor <cursoragent@cursor.com>

mkuchenbecker

Mostly LGTM, some additional testing and I want to undrestand the default behavior. I know it was moved but it seems like a silent failure (correct me if im wrong on that).

mkuchenbecker · 2026-08-12T16:03:45Z

+        }
+
+        return new GenericInternalRow(values);
+      default:


Is the default here correct? Should i.e. i assume this is an error condition.

mkuchenbecker · 2026-08-12T16:06:24Z

+  private static final Schema WRITE_SCHEMA =
+      new Schema(
+          required(1, "id", Types.LongType.get()), optional(2, "data", Types.StringType.get()));
+
+  private static final Schema READ_SCHEMA =
+      new Schema(
+          required(1, "id", Types.LongType.get()),
+          optional(2, "data", Types.StringType.get()),
+          Types.NestedField.optional("country")


Can we exercise all data types?

cbb330 · 2026-08-12T22:03:29Z

Superseded by #267. This was the Spark half of Option A; defaulted scans now stay on the row reader in the #267–#269 stack. Closing.

github-actions Bot added the SPARK label Jul 23, 2026

cbb330 force-pushed the chbush/oh120-orc-default-fill branch from a888bee to 4db9002 Compare July 25, 2026 01:04

cbb330 force-pushed the chbush/oh120-spark31-default-fill branch from 93e4737 to b22ffd9 Compare July 25, 2026 01:04

This was referenced Jul 25, 2026

ORC: bind struct fields by Iceberg field id in generic and Spark 3.1 readers #265

Merged

ORC: fill initial defaults for missing id-bound fields #263

Closed

cbb330 force-pushed the chbush/oh120-orc-default-fill branch from 4db9002 to cac8ff0 Compare July 28, 2026 02:35

cbb330 force-pushed the chbush/oh120-spark31-default-fill branch from b22ffd9 to 748e269 Compare July 28, 2026 02:35

cbb330 force-pushed the chbush/oh120-orc-default-fill branch from cac8ff0 to 1115244 Compare July 28, 2026 03:11

cbb330 force-pushed the chbush/oh120-spark31-default-fill branch 2 times, most recently from 5de2a95 to 4f928f2 Compare July 28, 2026 07:13

cbb330 force-pushed the chbush/oh120-orc-default-fill branch from 1115244 to 0e7b860 Compare July 28, 2026 07:13

cbb330 force-pushed the chbush/oh120-spark31-default-fill branch from 4f928f2 to 5549db5 Compare July 29, 2026 06:08

mkuchenbecker reviewed Jul 31, 2026

View reviewed changes

cbb330 force-pushed the chbush/oh120-spark31-default-fill branch 2 times, most recently from 459d91f to b6b7d89 Compare August 3, 2026 01:38

cbb330 force-pushed the chbush/oh120-orc-default-fill branch from 3fd044f to 5175788 Compare August 3, 2026 01:58

cbb330 force-pushed the chbush/oh120-spark31-default-fill branch from b6b7d89 to 77082fc Compare August 3, 2026 01:58

cbb330 force-pushed the chbush/oh120-orc-default-fill branch from 5175788 to 540604c Compare August 3, 2026 02:03

cbb330 force-pushed the chbush/oh120-spark31-default-fill branch 2 times, most recently from af3ada9 to 2880ffb Compare August 3, 2026 05:47

cbb330 force-pushed the chbush/oh120-orc-default-fill branch from 540604c to 60115f6 Compare August 3, 2026 06:42

cbb330 force-pushed the chbush/oh120-spark31-default-fill branch from 2880ffb to b5e1ffa Compare August 3, 2026 06:42

cbb330 force-pushed the chbush/oh120-orc-default-fill branch from 60115f6 to d5893e8 Compare August 3, 2026 07:10

cbb330 force-pushed the chbush/oh120-spark31-default-fill branch from b5e1ffa to fd15d74 Compare August 3, 2026 07:10

cbb330 force-pushed the chbush/oh120-orc-default-fill branch from d5893e8 to ca18b89 Compare August 3, 2026 07:13

cbb330 force-pushed the chbush/oh120-spark31-default-fill branch 2 times, most recently from 73b8758 to c7082d1 Compare August 3, 2026 07:25

cbb330 force-pushed the chbush/oh120-spark31-default-fill branch from c7082d1 to 51dd54c Compare August 3, 2026 21:46

cbb330 mentioned this pull request Aug 10, 2026

ORC/Spark: forward-port default-value reads via idToConstant (LI #76) #267

Open

mkuchenbecker reviewed Aug 12, 2026

View reviewed changes

cbb330 closed this Aug 12, 2026

Navigation Menu

Uh oh!

Spark 3.1: fill ORC initial defaults on the row reader - #264

Spark 3.1: fill ORC initial defaults on the row reader#264
cbb330 wants to merge 1 commit into
chbush/oh120-orc-default-fillfrom
chbush/oh120-spark31-default-fill

cbb330 commented Jul 23, 2026 •

edited

Loading

Uh oh!

mkuchenbecker Jul 31, 2026

Uh oh!

mkuchenbecker Jul 31, 2026

Uh oh!

mkuchenbecker left a comment

Uh oh!

mkuchenbecker left a comment

Uh oh!

mkuchenbecker Aug 12, 2026

Uh oh!

mkuchenbecker Aug 12, 2026

Uh oh!

cbb330 commented Aug 12, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		// Follow-up: wire initial-default constants into VectorizedSparkOrcReaders. Tables whose
		// in-memory schema declares defaults must remain on the row reader until this path is wired.

Uh oh!

Conversation

cbb330 commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Scope

Stack

Testing Done

Uh oh!

mkuchenbecker Jul 31, 2026

Choose a reason for hiding this comment

Uh oh!

mkuchenbecker Jul 31, 2026

Choose a reason for hiding this comment

Uh oh!

mkuchenbecker left a comment

Choose a reason for hiding this comment

Uh oh!

mkuchenbecker left a comment

Choose a reason for hiding this comment

Uh oh!

mkuchenbecker Aug 12, 2026

Choose a reason for hiding this comment

Uh oh!

mkuchenbecker Aug 12, 2026

Choose a reason for hiding this comment

Uh oh!

cbb330 commented Aug 12, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

cbb330 commented Jul 23, 2026 •

edited

Loading