Skip to content

[default values] Implement ReadBridge.apply without an Iceberg fork - #679

Open
cbb330 wants to merge 2 commits into
chbush/read-bridge-sanitize-on-writefrom
chbush/read-bridge-apply
Open

[default values] Implement ReadBridge.apply without an Iceberg fork#679
cbb330 wants to merge 2 commits into
chbush/read-bridge-sanitize-on-writefrom
chbush/read-bridge-apply

Conversation

@cbb330

@cbb330 cbb330 commented Aug 12, 2026

Copy link
Copy Markdown
Collaborator

Summary

Depends on #678 (server handshake + drop), stacked on #674#668#645.

Finish the client half of the column-default read bridge: turn server-stamped config into Iceberg NestedField.initialDefault at metadata load — without forking Iceberg.

#678 lands first so apply is never in tree without the drop. Overlays sit on metadata.schema(); the first write after a bridged load would persist them unless the server already strips them.

Step Where What
Decode ReadBridge.from(config) in doRefresh / before IO Parse openhouse.read-bridge.column-default.<fieldId> (kept from #668, off the retry path)
Apply ReadBridge.apply(raw) in loadMetadata Overlay defaults onto every schema-id, then rebuild metadata

How apply works

Patch initial-default onto schema field objects by field-id in metadata JSON. No Iceberg fork.

Failure policy (mirrors the server encoder):

Case Behavior
No read-bridge keys / inert bridge Return raw unchanged
Field-id absent from a schema Capability gap → leave that field NULL
Default cannot bind to column type Fail loud (IllegalStateException)
Malformed known config entry Fail loud (decode; already unrecoverable for retries via #668)
Unknown openhouse.read-bridge.* key Ignore (forward compatible)

Semantics notes:

  • Overlay hits all schema-ids (time-travel / tags / branches), not only current.
  • Decode keeps validated JSON strings (shade-safe bind at apply time).
  • Current stamp wins — no default history; missing ids on older schemas are gaps, not errors.

Changes

  • New Features — ReadBridge.apply overlays initialDefault
  • Tests — ReadBridgeTest: single/multi-schema overlay, nested fields, gaps, fail-loud bind, inert path; OpenHouseTableOperationsTest: PUT still sends stamped initial-default

Testing Done

  • Added new tests for the changes made.
  • Updated existing tests to reflect the changes made.
  • Local: ./gradlew :integrations:java:iceberg-1.2:openhouse-java-itest:test --tests '*ReadBridgeTest' --tests '*OpenHouseTableOperationsTest'

Additional Information

  • Large PR broken into smaller PRs, and PR plan linked in the description.

Stack

  1. #645 — substrate (seam + encode + decode hook)
  2. #668 — decode off Iceberg’s retry path
  3. #674 — policy / ramp in OpenHouse
  4. #678 — handshake + drop on the server
  5. This PRReadBridge.apply
  6. #681 — Spark catalog itest (overlay)

@cbb330
cbb330 force-pushed the chbush/read-bridge-apply branch 2 times, most recently from 27edc5d to 4f25ef4 Compare August 12, 2026 23:44
@cbb330
cbb330 force-pushed the chbush/read-bridge-sanitize-on-write branch from d101fcd to 693b61b Compare August 12, 2026 23:57
@cbb330
cbb330 force-pushed the chbush/read-bridge-apply branch from 4f25ef4 to e68a724 Compare August 12, 2026 23:57
@cbb330
cbb330 force-pushed the chbush/read-bridge-sanitize-on-write branch from 693b61b to 108b7df Compare August 13, 2026 00:19
@cbb330
cbb330 force-pushed the chbush/read-bridge-apply branch from e68a724 to 171134f Compare August 13, 2026 00:19
@cbb330
cbb330 force-pushed the chbush/read-bridge-sanitize-on-write branch from 108b7df to dcc7edb Compare August 13, 2026 00:37
@cbb330
cbb330 force-pushed the chbush/read-bridge-apply branch from 171134f to 7829367 Compare August 13, 2026 00:37
@cbb330
cbb330 force-pushed the chbush/read-bridge-sanitize-on-write branch from dcc7edb to ff60798 Compare August 13, 2026 02:36
@cbb330
cbb330 force-pushed the chbush/read-bridge-apply branch from 7829367 to a79c316 Compare August 13, 2026 02:36
@cbb330
cbb330 force-pushed the chbush/read-bridge-sanitize-on-write branch from ff60798 to a5571f8 Compare August 13, 2026 02:48
@cbb330
cbb330 force-pushed the chbush/read-bridge-apply branch 3 times, most recently from 01a1468 to 3793274 Compare August 13, 2026 05:29
@cbb330
cbb330 force-pushed the chbush/read-bridge-sanitize-on-write branch from f05903f to 9d1bb96 Compare August 13, 2026 05:42
@cbb330
cbb330 force-pushed the chbush/read-bridge-apply branch 2 times, most recently from 4c6f354 to a64ea0f Compare August 13, 2026 05:52
@cbb330
cbb330 force-pushed the chbush/read-bridge-sanitize-on-write branch from 253129c to 7c005b2 Compare August 13, 2026 06:10
@cbb330
cbb330 force-pushed the chbush/read-bridge-apply branch from a64ea0f to 4be6d59 Compare August 13, 2026 06:10
@cbb330
cbb330 changed the base branch from chbush/read-bridge-sanitize-on-write to chbush/read-bridge-policy-ramp August 13, 2026 22:27
@cbb330
cbb330 force-pushed the chbush/read-bridge-apply branch 2 times, most recently from 08551a4 to 145297f Compare August 13, 2026 22:36
@cbb330
cbb330 changed the base branch from chbush/read-bridge-policy-ramp to chbush/read-bridge-sanitize-on-write August 13, 2026 22:36
try {
return MAPPER.readTree(json);
} catch (JsonProcessingException e) {
throw new IllegalStateException("read-bridge: invalid json", e);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

any reason we are using sneaky throws vs typed?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

quick answer: i believe this is because we catch illegalstateexception at a higher level so that it doesn't trigger retries. i will confirm in a bit.

@cbb330
cbb330 force-pushed the chbush/read-bridge-sanitize-on-write branch from 4ad3f14 to b8a8a8b Compare August 13, 2026 23:56
@cbb330
cbb330 force-pushed the chbush/read-bridge-apply branch from 145297f to 17b3d21 Compare August 13, 2026 23:56
@cbb330
cbb330 force-pushed the chbush/read-bridge-apply branch from 17b3d21 to d7edc72 Compare August 14, 2026 04:43
cbb330 added 2 commits August 14, 2026 13:40
Overlay stamped column defaults onto schema field objects by field-id using the sanitize JSON helpers.
Apply puts overlays on the in-memory schema; constructMetadataRequestBody
must not strip them. Skip-reload still sends the bound overlay. Unstamped
writer defaults stay on the wire.
@cbb330
cbb330 force-pushed the chbush/read-bridge-apply branch from d7edc72 to 950f3f4 Compare August 14, 2026 20:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants