Skip to content

[default values] Implement ReadBridge.apply without an Iceberg fork - #675

Closed
cbb330 wants to merge 9 commits into
chbush/read-bridge-policy-rampfrom
chbush/read-bridge-apply
Closed

[default values] Implement ReadBridge.apply without an Iceberg fork#675
cbb330 wants to merge 9 commits into
chbush/read-bridge-policy-rampfrom
chbush/read-bridge-apply

Conversation

@cbb330

@cbb330 cbb330 commented Aug 12, 2026

Copy link
Copy Markdown
Collaborator

Summary

Depends on #674 (policy), stacked on #668#645.

Finish the client half of the column-default read bridge: turn server-stamped config into Iceberg NestedField.initialDefault at metadata load — without forking Iceberg.

Step Where What
Decode ReadBridge.from(config) in doRefresh / before IO Parse openhouse.read-bridge.column-default.<fieldId> (kept from #668, off the retry path)
Apply ReadBridge.apply(raw) in loadMetadata Overlay defaults onto every schema-id, then rebuild metadata

How apply works

  1. For each schema in raw.schemas(), rebuild fields with NestedField.builder + withInitialDefault when a stamped field-id is present and binds.
  2. Replace the schema set on TableMetadata via public TableMetadataParser.toJson / fromJson (patch schemas / schema-id; format-v1 also patches schema). No withSchemaOverlay, no Iceberg fork.
  3. Return the rebuilt metadata to the catalog load path.

Failure policy (mirrors the server encoder):

Case Behavior
No read-bridge keys / inert bridge Return raw unchanged
Field-id absent from a schema Capability gap → leave that field NULL
Default cannot bind to column type Fail loud (IllegalStateException)
Malformed known config entry Fail loud (decode; already unrecoverable for retries via #668)
Unknown openhouse.read-bridge.* key Ignore (forward compatible)

Semantics notes:

  • Overlay hits all schema-ids (time-travel / tags / branches), not only current.
  • Decode keeps validated JSON strings (shade-safe bind at apply time).
  • Current stamp wins — no default history; missing ids on older schemas are gaps, not errors.

Changes

  • New Features — ReadBridge.apply actually overlays initialDefault
  • Internal API Changes — replaceSchemas JSON rebuild helper (public Iceberg APIs only)
  • Tests — ReadBridgeTest: single/multi-schema overlay, nested fields, gaps, fail-loud bind, inert path

Why not fork Iceberg

LinkedIn Iceberg already exposes NestedField.initialDefault / withInitialDefault. The missing piece is replacing schemas on an existing TableMetadata instance. A JSON round-trip through TableMetadataParser is enough for the client overlay and keeps OH off an Iceberg fork. If Iceberg later ships a real overlay helper, swap replaceSchemas only.

Testing Done

  • Added new tests for the changes made.
  • Updated existing tests to reflect the changes made.
  • Local: ./gradlew :integrations:java:iceberg-1.2:openhouse-java-itest:test --tests '*ReadBridgeTest'

Iceberg 1.5 / Spark 3.5 java runtime shares these sources via Gradle sourceSets (same apply path).

Additional Information

  • Large PR broken into smaller PRs, and PR plan linked in the description.

Stack

  1. #645 — substrate (seam + encode + decode hook)
  2. #668 — decode off Iceberg’s retry path
  3. #674 — policy / ramp in OpenHouse
  4. #678 — sanitize-on-write (do not merge apply without this)
  5. This PRReadBridge.apply

Depends on #678. Overlay + sanitize round-trip is covered by sanitizeAfterApplyRestoresOnDiskSchema.

Rollback Plan

Revert this PR. Clients keep decoding config (#668) but stop overlaying initialDefault; readers behave as today (NULL for missing columns). Server stamp (#645/#674) can remain inert for old clients.

@cbb330
cbb330 force-pushed the chbush/read-bridge-apply branch from c84734e to 4f2f134 Compare August 12, 2026 02:24
@cbb330
cbb330 force-pushed the chbush/read-bridge-apply branch 5 times, most recently from a62d4e6 to 3266fa6 Compare August 12, 2026 05:33
@cbb330
cbb330 marked this pull request as ready for review August 12, 2026 16:10
@cbb330
cbb330 force-pushed the chbush/read-bridge-apply branch from 3266fa6 to b3f7c4c Compare August 12, 2026 16:51
cbb330 and others added 8 commits August 12, 2026 10:01
Keep the deployment-specific ColumnDefaultsSource optional and data-only;
OpenHouse owns the read-bridge.column-default feature id, self-serve
enabled property, and fail-open toggle lookup before asking for defaults.
Correct the self-service property name and drop the contradictory
"cluster kill switch" claim. Add coverage for a real source that returns
no defaults, and assert table-property opt-in never calls the server
toggle.

Testing Done:
- :services:tables:test --tests '*ReadBridgeConfigResolverTest'
BaseTableFeatureToggle looks up (databaseId, tableId, featureId) exactly;
claiming a * / * fleet kill switch was inaccurate. Also note HTS is only
hit when the self-service property is absent.
Replace essay javadoc with brief ownership, contract, ObjectProvider,
and fail-open notes so the PR description carries the design narrative.
Read-bridge overlays must not persist: a later write would stamp
initial-default onto disk and survive ramp-off. Restore default slots
for field-ids that existed at load; keep writer defaults on new ids so
V2 schema evolution still works.
Overlay server-stamped column defaults onto every schema-id at metadata
load using NestedField.initialDefault and a public TableMetadataParser
JSON rebuild, keeping decode on the from()/apply() split from #668.
@cbb330
cbb330 force-pushed the chbush/read-bridge-apply branch from b3f7c4c to b85fd39 Compare August 12, 2026 22:59
@cbb330

cbb330 commented Aug 12, 2026

Copy link
Copy Markdown
Collaborator Author

Restacking onto #678 (sanitize-on-write). GitHub would not change this PR's base because it is part of a stack, so apply moves to a new PR with the correct base.

@cbb330 cbb330 closed this Aug 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant