ORC: omit initial-defaults only for complete embedded field IDs

[cbb330]

Summary

• Tighten PR1's omit path: a defaulted field is omitted from the ORC projection only when the reader opted in (supportsInitialDefaults), the file uses EMBEDDED Iceberg field IDs, and the file has a complete id set.
• Name-mapped or partially annotated files cannot prove a column was never written, so they keep the null-synthesizing path instead of filling a default.
• Stacked on #267 (Raymond forward-port).

Testing Done

• Unit tests added/updated (TestBuildOrcProjection, TestORCSchemaUtil.testHasAllIds, Spark reader partial-ID cases)
• Integration tests pass
• Manual testing performed

export JAVA_HOME=$(/usr/libexec/java_home -v 11)
./gradlew -DsparkVersions=3.1 -DscalaVersion=2.12 -DhiveVersions= -DflinkVersions= \
  :iceberg-orc:test --tests 'org.apache.iceberg.orc.TestBuildOrcProjection' \
  --tests 'org.apache.iceberg.orc.TestORCSchemaUtil' \
  :iceberg-spark:iceberg-spark-3.1_2.12:test \
  --tests 'org.apache.iceberg.spark.data.TestSparkOrcReaderForFieldsWithDefaultValue' \
  -x generateGitProperties

Dependencies

• Requires #267.

[cbb330]

@mkuchenbecker carrying over your notes from #263. they apply here more than on the closed PR (that was a discarded strategy. it was net-new code and too complex. here we are building off a previous implementation that landed in prod for hive, and forward porting the mechanism)

TLDR; we do not fill defaults on name-mapped files. We do that usingFieldIdSource which represents where the IDs already on the read schema came from:

• EMBEDDED: the file wrote iceberg.id attributes. A missing ID means that column was never written, so it is safe to omit and fill.
• NAME_MAPPED: IDs were applied at read time by the existing Iceberg name mapping (e.g for Hive-migrated / no headers). These are not safe. e.g. After a rename the mapping does not cover it, a column can look absent while its data is still in the file. Filling a default there would overlay real data, so we never omit. We fall back to same null-synthesizing projection as today.

When we skip name mapping entirely, those files would still read the way they do today. We just would not fill defaults. That matches the restriction you suggested: only fill when the file already has complete embedded field IDs. Reading does not require a rewrite. Filling does, the file has to have been written (or rewritten) with Iceberg ORC headers. We can call that out in the as a feature contract.

btw we did not throw. Throwing on name-mapped / incomplete-ID files would break existing reads. The correctness fix is that we do not fill unless EMBEDDED + hasAllIds. Otherwise keep today's null column. The Spark row reader that opted in, on a file with complete IDs, gets the default rather than null.

[cbb330]

Inline pointer for the name-mapping / rewrite notes from #263 — full answer is in the PR conversation.

[cbb330]

@mkuchenbecker this is the site for your name-mapping / Kyoto / rewrite notes from #263.

We are not adding a mapping. NAME_MAPPED means IDs were applied at read time by the existing Iceberg name mapping, and we never omit/fill on that path (a rename the mapping misses can look absent while data is still in the file).

Fill only for EMBEDDED + complete IDs. Files without headers stay readable as today (null, not default). Full write-up: #268 (comment)