Skip to content

ORC: don't push file filters for omitted initial-default fields - #269

Open
cbb330 wants to merge 5 commits into
chbush/oh120-orc-defaults-raymond-pr2from
chbush/oh120-orc-defaults-raymond-pr3
Open

ORC: don't push file filters for omitted initial-default fields#269
cbb330 wants to merge 5 commits into
chbush/oh120-orc-defaults-raymond-pr2from
chbush/oh120-orc-defaults-raymond-pr3

Conversation

@cbb330

@cbb330 cbb330 commented Aug 12, 2026

Copy link
Copy Markdown
Collaborator

Summary

When a defaulted column is omitted from the ORC file projection (#267), Spark may still filter on it (WHERE country = 'US'). Pushing that filter into the file fails in two ways:

  1. If the projection is only that omitted column, the read schema is empty and converting the filter crashes.
  2. If the filter is pushed anyway, the file never had the column, so the predicate cannot see the filled default. Spark would filter the wrong thing.

This PR stops pushing file-level predicates for omitted defaulted fields. Spark evaluates those filters on the filled values after read. Empty projections no longer crash.

Also: TypeUtil schema copies (AssignFreshIds / PruneColumns / ReassignIds) now keep initialDefault via NestedField.from(field), so table-create and SELECT round-trips do not drop defaults.

Stacked on #268#267.

Testing Done

export JAVA_HOME=$(/usr/libexec/java_home -v 11)
./gradlew -DsparkVersions=3.1 -DscalaVersion=2.12 -DhiveVersions= -DflinkVersions= \
  :iceberg-api:test --tests 'org.apache.iceberg.types.TestTypeUtil' \
  :iceberg-orc:test --tests 'org.apache.iceberg.orc.TestExpressionToSearchArgument' \
  :iceberg-spark:iceberg-spark-3.1_2.12:test \
  --tests 'org.apache.iceberg.spark.source.TestSparkOrcInitialDefaultScan' \
  --tests 'org.apache.iceberg.spark.data.TestSparkOrcReaderForFieldsWithDefaultValue' \
  -x generateGitProperties
  • Filter on only the omitted defaulted column — no crash, Spark sees the filled value
  • Mixed physical + defaulted predicates
  • Defaulted scans stay on the row reader even with vectorization enabled
  • initialDefault survives TypeUtil copies

@cbb330
cbb330 force-pushed the chbush/oh120-orc-defaults-raymond-pr3 branch from 3fff6c3 to aec1304 Compare August 12, 2026 19:20
@cbb330
cbb330 force-pushed the chbush/oh120-orc-defaults-raymond-pr3 branch from aec1304 to dc60d6a Compare August 12, 2026 20:31
@cbb330
cbb330 force-pushed the chbush/oh120-orc-defaults-raymond-pr3 branch from dc60d6a to 0fc9d35 Compare August 12, 2026 23:06
@cbb330 cbb330 changed the title ORC: disable SARG pushdown for omitted initial-default fields ORC: don't push file filters for omitted initial-default fields Aug 12, 2026
cbb330 added 3 commits August 12, 2026 17:36
Empty read projections no longer crash convert (#118). Predicates on
omitted fields become YES_NO_NULL so Spark can filter filled defaults.
Preserve defaults across TypeUtil schema copies so table create round-trips them.
With vectorization enabled, projections that include an initial-default
must not take the columnar ORC path.
Keep the why (empty projection, SARG disable) without stacking ticket numbers.
@cbb330
cbb330 force-pushed the chbush/oh120-orc-defaults-raymond-pr3 branch from 0fc9d35 to c4b556c Compare August 13, 2026 00:36
cbb330 and others added 2 commits August 12, 2026 18:08
Omitting a nested initial-default can leave struct<loc:struct<>>, which
fails convert() even though the root schema is non-empty. Treat that as
no bindable columns. Cover mixed-file IS NOT NULL/upper/= CA filters and
parent-null nested defaults in Spark SQL.

Co-authored-by: Cursor <cursoragent@cursor.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant