ORC: bind struct fields by Iceberg field id in generic and Spark 3.1 readers - #265
Conversation
bd16f75 to
d39cbfd
Compare
cbb330
left a comment
There was a problem hiding this comment.
Provenance annotations for each code block, categorized as: backported exactly / backported with adaptations / net new / inspiration from shipped code. Reference: apache/iceberg 77e7dbb24 (PR #15776).
These six files are byte-identical to the earlier 1.2.x-raised backport in #256 (same patch ID 74c67920ff84); this PR re-bases that work onto openhouse-1.2.0, which never had ID-bound ORC readers.
Divergence vs
|
Noise-free compares vs
|
…readers Backport the upstream id-based StructReader binding (apache/iceberg 77e7dbb, PR #15776) to the openhouse-1.2.0 ORC reader, adapted for the 1.2.x type system. The ORC StructReader previously bound struct fields positionally, consuming column vectors in projection order. This replaces that with id-based binding: each expected field is matched to its ORC column by Iceberg field id, which is resilient to field reordering and schema evolution. Scope: only GenericOrcReaders and Spark 3.1 SparkOrcValueReaders are converted. The positional constructor/overloads are retained and marked @deprecated because the other engine readers (Spark 2.4/3.2/3.3, Flink) still use them. Row-lineage, UNKNOWN, and VARIANT constructs from the upstream commit are omitted as they do not exist on openhouse-1.2.0. Behavior-preserving for well-formed reads: id-based binding produces identical results to positional binding on all current inputs. Co-authored-by: Cursor <cursoragent@cursor.com>
d39cbfd to
4ca1b1e
Compare
Add direct generic and Spark reader tests that force expected fields to map to different ORC vector positions. Generated-by: pi (GPT-5.6)
Port the test-only follow-up commit from #265 (openhouse-1.2.0) so this 1.5.2 backport is a 1:1 match: - TestGenericOrcReaderIdBinding.testNonIdentityFieldIdMapping: forces expected fields to map to different ORC vector positions. - TestGenericOrcReaderIdBinding.testIdLessFileBindsThroughNameMapping: an id-less (Hive-migrated) ORC file still binds through name mapping, and a field absent from it reads null. - TestSparkOrcReader.nonIdentityFieldIdMapping: same non-identity vector mapping through the Spark 3.5 reader, written in the JUnit 5 / AssertJ style used on this branch. Generated-by: pi (GPT-5.6)
…readers (#257) Backport the upstream id-based StructReader binding (apache/iceberg 77e7dbb, PR #15776) to the openhouse-1.5.2 ORC reader. Companion to #265 on openhouse-1.2.0. The ORC StructReader binds fields by Iceberg field id instead of projection order. Only GenericOrcReaders and Spark 3.5 SparkOrcValueReaders are converted; positional overloads stay @deprecated for unconverted engines. Includes the non-identity field-id mapping tests ported from #265.
Summary
Backports the upstream ID-based ORC
StructReaderbinding (apache/iceberg77e7dbb24, PR #15776) to theopenhouse-1.2.0line, adapted for the 1.2.x type system.What changes: the ORC reader bound struct fields positionally — consuming ORC column vectors in projection order, assuming projected field order matches the file's physical column order. This switches the converted readers to ID-based binding: each expected field is matched to its ORC column by its Iceberg field ID, which is robust to field reordering and schema evolution.
Why it's safe:
buildOrcProjectionalready normalizes column order, so ID-based binding produces identical results to positional binding on all current inputs. It is a behavior-preserving robustness change — and the prerequisite for the default-fill work stacked on top.How ID-based binding works (quick primer)
An ORC file stores columns in a physical order. Iceberg has an expected schema, and every field carries a permanent numeric field ID. Two ways to match expected fields to file columns:
The logic lives in the shared
OrcValueReaders.StructReaderbase class iniceberg-orc; the per-engine reader edits are thin plumbing that thread the ORC file schema (TypeDescription) down to it and opt in.Provenance of each change
OrcValueReaders: ID-binding constructor +readersByFieldId/buildFieldIdToOrcIndex+readInternalselectionGenericOrcReaders4-argstruct+ subclass ctor;GenericOrcReaderrecordthreadingSparkOrcValueReaders/SparkOrcReader(v3.1)@Deprecatedpositional overload that upstream deletedTestGenericOrcReaderIdBindingTestSparkOrcReadMetadataColumns/TestGenericReadProjectionPer-line classification is in the inline review annotations on this PR.
Adaptations vs
apache/main(why it diverges at all)openhouse-1.2.0lacks constructs the upstream commit references, so these are intentionally omitted (porting them verbatim would not compile here):ROW_ID/LAST_UPDATED_SEQUENCE_NUMBERbranches, thehandleRowIdField/handleLastUpdatedSeqFieldhelpers, and theRowIdReader/LastUpdatedSeqReaderclasses.UNKNOWNtype — the|| field.type().typeId() == Type.TypeID.UNKNOWNclause in the metadata fallback.VARIANT— the variant reader/visitor.Everything else is aligned to upstream verbatim (method ordering, Javadoc, whitespace, no defensive null-guards).
Scope
Only
GenericOrcReadersand Spark 3.1SparkOrcValueReadersare converted. The positional constructor/overloads are retained and@Deprecatedbecause the unconverted readers (Spark 2.4/3.2/3.3, Flink 1.14/1.15/1.16) still call them — those readers keep identical positional behavior.Reviewing the divergence from upstream
77e7dbb24apache/main: apache/iceberg@main...cbb330:iceberg-apache:compare/orc-id-binding-vs-apache-main (if GitHub says "taking too long to generate," hit Retry — a large-repo cross-fork rendering quirk)apache/main" comment below.git fetch apache main && git diff apache/main -- orc/src/main/java/org/apache/iceberg/orc/OrcValueReaders.javaThese
compare/orc-id-binding-*branches are disposable comparison artifacts, not intended for merge. They remain accurate for this PR: the six files here are byte-identical to the earlier 1.2.x-based backport in #256 (same patch ID74c67920ff84).Companion PR
The identical change is backported to the 1.5.x line (targeting Spark 3.5) in #257. The ORC core is byte-identical between the two backports — proof (renders "the branches are identical"): cbb330/iceberg-apache@compare/orc-id-binding-shared-core...compare/orc-id-binding-shared-core-1.5.x
History
This supersedes #256, which raised the same change against
1.2.x— a near-vanilla apache branch that is not the OpenHouse dev line. #256's closing note incorrectly stated thatopenhouse-1.2.0already had ID-bound ORC readers; it did not (OrcValueReaders.javathere was byte-identical to vanilla 1.2.x). This PR restores that work on the correct base. Note that shanthoosh's #250 (theNestedFieldcolumn-default API, already merged here) has zero file overlap with this change.Stack
StructReaderbackport (base:openhouse-1.2.0)The default-fill PRs build on this binding so physical columns and absent-field default constants are both associated by Iceberg field ID.
Testing Done
TestGenericOrcReaderIdBindingcovers type promotion (int→long, float→double, decimal widening), reordered projection,_pos/_deletedmetadata columns, andidToConstantNext steps
This PR lands only the binding mechanism and deliberately does not change read output. The motivating follow-up is column default values: once fields are bound by ID, the stacked PRs use those bound IDs to resolve an Iceberg field's initial-default for fields present in the expected schema but absent from the ORC file (which today read as
null).Made with Cursor