Spark 4.0 / Iceberg 1.11 / deletion vectors — REST-first upgrade (consolidated) - #17
Draft
mkuchenbecker wants to merge 55 commits into
Draft
Spark 4.0 / Iceberg 1.11 / deletion vectors — REST-first upgrade (consolidated)#17mkuchenbecker wants to merge 55 commits into
mkuchenbecker wants to merge 55 commits into
Conversation
…openhouse Re-apply the Iceberg 1.11 version wiring (REST-first Spark-4.0 upgrade) onto main: - root build.gradle: add iceberg_1_11_version = "1.11.0-openhouse"; add mavenLocal() to the allprojects repositories block (the fork artifacts are published there). - openhouse.iceberg-conventions-1.5.2.gradle: move the server core (services/tables + iceberg/* + tables-test-fixtures) from com.linkedin.iceberg:*:1.5.2.15 to stock org.apache.iceberg:*:1.11.0-openhouse; raise the TARGET_JVM_VERSION resolution attribute (resolvable configs only) to 17, since 1.11's module metadata declares jvm.version=17 while the compile target stays at -target 8 (HDFS-client requirement). - openhouse.java-minimal-conventions.gradle: same 17 resolution-attribute relax. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01LLnVvLYnxjTuH2b5nAHa4A
…TE TABLE Server-side implementation of the Iceberg REST Catalog spec, mounted under /iceberg/v1, so a stock org.apache.iceberg.rest.RESTCatalog (Spark 4.0) can drive OpenHouse with no custom client jar. - IcebergRestCatalogController: config, namespace (implicit-database) ops, table list/load/exists/drop/rename delegating to Iceberg CatalogHandlers -> the OpenHouseInternalCatalog bean, and CREATE TABLE translated into an OpenHouse CreateUpdateTableRequestBody driven through the same TablesApiHandler.createTable pipeline the native controller uses (location allocation, reserved openhouse.* props, policy management, eligibility). Errors mapped to the Iceberg ErrorResponse envelope via local @exceptionhandlers. Staged create still 501s here (CTAS/RTAS handled in a follow-up commit). - PartitionSpecMapper: add the inverse mappers toTimePartitionSpec(Schema, PartitionSpec) and toClusteringColumns(Schema, PartitionSpec) so a REST-created table maps to the same OpenHouse model as a natively created one; specs OpenHouse cannot model are rejected (HTTP 400). - docs/spark4-iceberg-upgrade/rest-endpoint: design, decisions, pitfalls. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01LLnVvLYnxjTuH2b5nAHa4A
Close the CTAS/RTAS gap in IcebergRestCatalogController so a stock Spark 4.0
RESTCatalog can run CREATE TABLE AS SELECT and (CREATE OR) REPLACE TABLE AS SELECT
against the OpenHouse /iceberg endpoint.
- createTable: stageCreate=true no longer 501s; treated as a plain create-then-commit
via the same TablesApiHandler.createTable pipeline (accepted non-atomic compromise
-- OpenHouse commits on create).
- updateTable routes three ways off the Iceberg UpdateRequirement/update fingerprint
the stock client stamps:
* CTAS data commit (AssertTableDoesNotExist): keep only the snapshot updates and
land them onto the already-created table via CatalogHandlers, like an INSERT.
* RTAS commit (the replace identity-reset trio SetCurrentSchema +
SetDefaultPartitionSpec + SetDefaultSortOrder): reconstruct the final
TableMetadata and route through OpenHouse's replace pipeline via
IcebergSnapshotsApiHandler (replaceCommit=true), so the service-layer replace
branch (validateReplaceTable -> replace.enabled/WAP/replication gating, linkedin#640)
and replace semantics apply.
* Everything else: unchanged delegation to CatalogHandlers.updateTable.
- docs/spark4-iceberg-upgrade/rest-ctas/; rest-endpoint pitfalls #1 updated.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Claude-Session: https://claude.ai/code/session_01LLnVvLYnxjTuH2b5nAHa4A
…it path A stock Spark 4.0 RESTCatalog drives plain writes through IcebergRestCatalogController.updateTable -> CatalogHandlers.updateTable -> OpenHouseInternalCatalog TableOperations.commit, which bypasses the service-layer update validation the native path runs in TablesService.putTable -> OpenHouseInternalRepositoryImpl.save. Restore those guards on the REST plain-update branch by pre-inspecting the projected commit before delegating, mirroring save()'s non-replace branch: - table LOCK enforcement (policies.lockState.locked) - reserved openhouse.*/policies property immutability (also covers openhouse.tableType immutability) - partition-spec evolution rejection - schema-evolution validation, reusing the same SchemaValidator bean Adds InvalidSchemaEvolutionException to the 400 handler. Reconciliation with the CTAS/RTAS routing and linkedin#640: the guards run ONLY on the plain UPDATE branch. Staged-create (AssertTableDoesNotExist) and RTAS (isReplacePayload) are routed away in updateTable() before the guards. RTAS eligibility (replace.enabled + WAP/replication) is enforced exactly once -- in the service replace branch (validateReplaceTable, linkedin#640) reached via replaceTable() -- so the replace gate is NOT duplicated in the controller. The parity branch's controller-side replace gate (enforceReplaceAllowed) and its separate parentId==null replace detector are therefore dropped: a single RTAS detector (isReplacePayload) remains, and a single replace gate (service layer) remains. docs/spark4-iceberg-upgrade/rest-validation-parity/. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01LLnVvLYnxjTuH2b5nAHa4A
… bake the fork jar
Bring the docker-compose recipe fixes from the spark4-upgrade HDFS validation onto main:
- spark-services.yml: point the spark master/worker/livy builds at
spark-3.5-base-hadoop3.2.dockerfile (was spark-base-hadoop2.8).
- hdfs-services.yml: bde2020 hadoop-3.2.1 namenode+datanode (was 2.8).
- oh-hadoop-spark/hadoop/hadoop.env: dfs.replication=1 + fs.defaultFS for the
single-datanode local cluster.
- spark-3.5-base-hadoop3.2.dockerfile: trust staged extra-ca-certs (system +
JDK cacerts) so the builder can fetch behind a TLS-terminating egress proxy;
runtime base -> eclipse-temurin:11-jdk-jammy (fork Iceberg is class-file 55);
bake the fork iceberg-spark-runtime jar onto $SPARK_HOME/jars. extra-ca-certs
dir committed with README/.gitignore (real *.crt staged per-environment).
- build.gradle CopyGitHooksTask: resolve the real gitdir so it works in a linked
git worktree (.git is a file) under Gradle 8.
- apps-spark-common shadowJar: also exclude META-INF/versions/{20,21} (newer
jackson-core multi-release variants that shadow 7.1.2's ASM cannot read).
- docs/spark4-iceberg-upgrade/docker-hdfs-validation/ + frepl-fix/.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Claude-Session: https://claude.ai/code/session_01LLnVvLYnxjTuH2b5nAHa4A
Additive documentation for the REST-first Spark-4.0 / Iceberg-1.11 upgrade: the phase plan, risks/decisions/findings, rung-2 detailed plan, BUILD-STATUS, and the harness-spark4 / rung3-dv progress+pitfalls notes. Docs only; no code or build impact. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01LLnVvLYnxjTuH2b5nAHa4A
…REST-first)
Phase 2 of the consolidation onto branch 1.11. Adds
:integrations:spark:spark-4.0:openhouse-spark-4.0-itest — a source-less module
whose only job is to resolve+build the harness runtime classpath for the
Spark-4.0 / Scala-2.13 / JDK-17 lane:
* embedded OpenHouse server (tables-test-fixtures -> services:tables, which
mounts IcebergRestCatalogController at /iceberg/v1/*), and
* STOCK unshaded iceberg-spark-4.0_2.13 (1.11.0-openhouse) pointed at a stock
RESTCatalog (no custom OpenHouse Spark runtime/catalog).
configureEach requests jvm.version=17 so the Spark-4.0 (17) + Iceberg-1.11
(17) + Java-8-bytecode server jars co-resolve; forces the javax-namespace
validation + a single 2.18.2 jackson + 1.12.1 avro stack; excludes the stale
com.linkedin.iceberg fork cores; and — new for this lane — excludes the
unshaded org.apache.hadoop:*:2.10.0 stack the server drags, so Spark 4.0's
shaded hadoop-client-api/runtime 3.4.1 is the SOLE org.apache.hadoop.fs.*
provider (the 2.10 FileSystem lacks openFile(Path):FutureDataInputStreamBuilder
and otherwise shadows 3.4.1 -> NoSuchMethodError on every read task).
Registered in settings.gradle as openhouse-spark-4.0-itest.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Claude-Session: https://claude.ai/code/session_01LLnVvLYnxjTuH2b5nAHa4A
…tock RESTCatalog Phase 2 version+wiring port of the MODULAR harness (no test logic/assertions changed). Mirrors the equivalent work done on the old monolithic harness. Env.scala (OpenHouseEnv): flip the catalog from the native OpenHouseCatalog to a stock Iceberg RESTCatalog against the embedded server's /iceberg endpoint — catalog-impl=org.apache.iceberg.rest.RESTCatalog, uri=$base/iceberg, token=<bearer> (was cluster/auth-token); register stock IcebergSparkSession Extensions only (the custom OpenHouse policy-DDL extension is not shipped on the REST client); and pin spark.driver.host=127.0.0.1 (Spark 4.0 local-executor remote-classloader fix; bindAddress alone sufficed on 3.5). run-openhouse.sh: compile with scalac 2.13.16 (Spark 4.0 is Scala 2.13 only), resolve the classpath from the spark-4.0 itest module via -DharnessItestPath, and apply Spark 4.0's documented --add-opens superset. Keeps the modular multi-file compile (DvProbe excluded — it has its own main). print-cp.init.gradle: honor -DharnessItestPath; apply the single-Iceberg exclusion only on the shaded (3.5) lane; dependsOn testRuntimeClasspath so project jars are built, not just resolved. ForkScenarios.scala: Iceberg 1.11 removed SparkSQLProperties.SPLIT_SIZE (session split-size superseded by adaptive split sizing); inline its literal "spark.sql.iceberg.split-size" — the constant was a static final String, so this is bytecode-identical and preserves the test's behavior. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01LLnVvLYnxjTuH2b5nAHa4A
…r the 1.11 base Brings over DvProbe + run-dvprobe.sh, adapted to the modular harness's OpenHouseEnv.start() 5-tuple. Boots the embedded server with cluster.iceberg.format-version=3 and drives a merge-on-read table through DELETE/UPDATE/MERGE, then reports whether the delete path produced Puffin deletion vectors (deletion-vector-v1 blobs) and checks read correctness. Confirmed on the consolidated 1.11 base (republished 1.11.0-openhouse fork, Spark 4.0 / stock RESTCatalog): parquet + orc both write 3 .puffin files each carrying a deletion-vector-v1 blob (content=1 POSITION_DELETES, fmt=PUFFIN), and the post-mutation read count is 4 (expected 4). Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01LLnVvLYnxjTuH2b5nAHa4A
…alidator
The native table-lock endpoint (POST /v1/databases/{db}/tables/{tbl}/lock ->
TablesController.createLock -> OpenHouseTablesApiHandler.createLock ->
OpenHouseTablesApiValidator.validateCreateLock) returned HTTP 500 on the
iceberg-1.11 / main-based 1.11 base.
Root cause: validateCreateLock -> validateGetTable -> validateDatabaseId /
validateTableId call StringUtils.isEmpty from the legacy commons-lang 2.x
(package org.apache.commons.lang). The iceberg-1.11 upgrade / main
reconciliation dropped commons-lang 2.x from the tables service runtime
classpath, so the first reference to it threw NoClassDefFoundError:
org/apache/commons/lang/StringUtils (ClassNotFoundException), surfaced by
Spring as INTERNAL_SERVER_ERROR. The e2e H2 tests still had commons-lang 2.x
transitively, which is why they stayed green while the delta-harness embedded
server (iceberg-1.11 itest classpath) 500'd.
Fix: switch the import to org.apache.commons.lang3.StringUtils (commons-lang3,
already a project dependency and present on the runtime classpath — proven by
convertToTableDto's existing lang3 usage). StringUtils.isEmpty has identical
null/empty semantics in both libraries, so this restores the existing lock
contract with no behavioral change. No new endpoint or shim added.
Verified via delta-harness on branch 1.11 (embedded OpenHouse server):
PASS control.lock.enforcement
PASS hazard.lock.starvesMaintenance
PASS interact.rtas.onLockedTable
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Claude-Session: https://claude.ai/code/session_01LLnVvLYnxjTuH2b5nAHa4A
…s to lang3 Same root cause as the lock-endpoint 500: the iceberg-1.11 / main-based reconciliation dropped commons-lang 2.x from the runtime classpath. These two main-source references (databases validator, OTel config) would throw the same NoClassDefFoundError when their path is hit. lang3 is already a dependency and the used methods (isEmpty/isNotEmpty/defaultIfEmpty) are semantically identical. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01LLnVvLYnxjTuH2b5nAHa4A
… JVM-attr fix CI could not build the 1.11 branch: the OpenHouse build resolves the custom fork org.apache.iceberg:*:1.11.0-openhouse, which exists only as SOURCE on mkuchenbecker/[email protected] and in a local mavenLocal, not in any CI-reachable artifact repo. Workflow gating (option 1a, 1.11-only): - New .github/workflows/branch-1.11.yml fires the reusable build-run-tests workflow on pushes/merges to the 1.11 branch (rare). - build-run-tests.yml gains two steps, gated `if: github.ref == 'refs/heads/1.11'`, that checkout mkuchenbecker/[email protected] and publishToMavenLocal (JDK 17, -DforceVersion=1.11.0-openhouse -DsparkVersions=4.0, core/api/common/data/parquet/orc/arrow/bundled-guava + spark-4.0_2.13 family, -x test -x compileTestJava -x compileTestScala) BEFORE the OpenHouse build. github.ref is inherited from the caller, so ordinary PRs (pr-validations, ref=refs/pull/N/merge) and push-to-main (build-tag-publish, ref=refs/heads/main) SKIP the expensive fork build. In-tree JVM-attribute fix: - java-test-fixtures re-derives TARGET_JVM_VERSION=8 from targetCompatibility and re-stamps it onto testFixtures*Classpath AFTER the convention plugin's configureEach ran, so those configs could no longer resolve the jvm.version=17 Iceberg 1.11 artifacts on a stock ./gradlew build. Added an afterEvaluate override in openhouse.iceberg-conventions-1.5.2.gradle (the 1.11 lane) that re-asserts TARGET_JVM_VERSION_ATTRIBUTE=17 on resolution-only configs. Scoped to the 1.11 convention plugin, so the spark-3.5/iceberg-1.10 (Java 8/11) lane is untouched. A plain `gradle :services:tables:build -x test` now succeeds with no init script. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01LLnVvLYnxjTuH2b5nAHa4A
…nfigs only The full clean build crashed configuring :all-modules — the previous afterEvaluate override touched every resolvable config incl. annotationProcessor, which cross-project aggregation had already resolved (Cannot change attributes after resolved). Restrict the override to the six real classpath configs that consume iceberg-1.11 and skip any already-resolved config (state==UNRESOLVED), so a plain gradle build works without breaking the previously-green build. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01LLnVvLYnxjTuH2b5nAHa4A
…le rung ledger Single authoritative current-state doc: done items, the five decisions + status, and the remaining critical path (finish CI green; custom Spark 4.0 + both lanes). Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01LLnVvLYnxjTuH2b5nAHa4A
…ves to iceberg-1.11 Repointing the shared openhouse.iceberg-conventions-1.5.2 convention to org.apache.iceberg:1.11.0-openhouse left the spark-3.5/1.5 apps lane resolving version-less com.linkedin.iceberg coordinates and inheriting the jvm-17 resolution attribute. Pin the com.linkedin.iceberg coordinates on apps/spark-3.5 to the 1.5 line it targets, and keep the jvm-17 attribute scoped to the 1.11 consumers — so a full 'clean build' no longer breaks the legacy lane. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01LLnVvLYnxjTuH2b5nAHa4A
…mer-config The afterEvaluate attribute-mutation approach raced fatally against every config resolved early during configuration (:all-modules aggregation, client codegen, Spring Boot productionRuntimeClasspath) — 'Cannot change attributes ... after it has been resolved', a different config each push. Replace it with a ComponentMetadataRule that relaxes the org.apache.iceberg artifacts' declared jvm.version to 8 once, at metadata time. No consumer configuration is mutated, so the race surface is gone; a stock ./gradlew build resolves iceberg-1.11 on Java-8 target modules. Verified locally: :services:housetables:productionRuntimeClasspath resolves iceberg-core:1.11.0-openhouse cleanly. Scoped to org.apache.iceberg, so the com.linkedin.iceberg 1.5/spark-3.5 lane is untouched. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01LLnVvLYnxjTuH2b5nAHa4A
Four upgrade-caused failures in the services:tables suite after the server core moved to org.apache.iceberg:1.11.0-openhouse: 1. avro downgrade -> NoSuchMethodError. iceberg-avro's TypeToSchema.<clinit> calls LogicalTypes.timestampNanos() (avro >= 1.12.0), but the root allprojects block force-pins avro 1.11.4 (CVE-2024-47561), downgrading iceberg-1.11's strict 1.12.1 pin. Re-force avro 1.12.1 in this iceberg-1.11 module only (1.12.1 > 1.11.4, so the CVE stays fixed; the iceberg-1.5 lane is untouched). Mirrors the spark-4.0 itest lane. 2. Mock Spring context could not load: the new IcebergRestCatalogController (component-scanned) autowires a SchemaValidator whose impl lives in the repository.impl package, which MockTablesApplication intentionally does not scan. Provide the BaseIcebergSchemaValidator bean explicitly. This cleared a ~430-test ApplicationContext cascade. 3. evolved_dummy_healthy_schema.json had "initial-default": null. iceberg-1.5 ignored a null-valued default key; iceberg-1.11 SchemaParser.defaultFromJson sees the key present and calls Expressions.lit(null) -> NPE. iceberg-1.11 serialization omits null defaults entirely, so drop the line (the fixture tests adding an optional field, not default semantics). 4. RepositoryTestWithSettableComponents.testNoRetryInternalRepo asserted refresh() is called exactly twice as a white-box no-retry proxy. iceberg-1.11 BaseTransaction drives applyUpdates()->refresh() once per retry attempt (base commit.retry.num-retries default 4 => 5 attempts) + 1 from newTransaction = 6. The authoritative no-retry signal (htsRepo.save called exactly once) still holds. Pin the count to 6 with an explanatory comment. This also fixed a cross-test cascade: when this test threw at the old assertion it skipped its catalog.dropTable cleanup, leaking d1.t1 and making SnapshotsControllerTest's later creates 500. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01LLnVvLYnxjTuH2b5nAHa4A
…1.11 Two upgrade-caused failure modes, both from iceberg-1.11's data IO rework: 1. avro downgrade -> NoSuchMethodError (LogicalTypes.timestampNanos) / NoClassDefFoundError (TypeToSchema$WithTypeToName). Same root as services:tables: the root allprojects block force-pins avro 1.11.4 for CVE-2024-47561, downgrading iceberg-1.11's strict 1.12.1 pin. Re-force 1.12.1 in this iceberg-1.11 module (1.12.1 > 1.11.4, CVE stays fixed). 2. NoClassDefFoundError: ParquetValueWriter / cannot init FormatModelRegistry. iceberg-1.11 routes generic data read/write through a FormatModelRegistry, and iceberg-data's GenericFormatModels.register() (run at the generic reader's static init) hard-references ParquetFormatModel and ORCFormatModel. iceberg-data no longer pulls iceberg-parquet/iceberg-orc transitively, so the generic reader this module uses fails to class-init. Add both explicitly. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01LLnVvLYnxjTuH2b5nAHa4A
…ceberg-1.11 The spark-3.5 itest compiles against the embedded 1.11 server's iceberg-core (1.11.0-openhouse), and 1.11 removed the addSchema(Schema, int lastColumnId) overload in favor of addSchema(Schema) (the builder derives lastColumnId from the schema's highest field id). SparkMultiSchemaEvolutionTest called the removed 2-arg form in 6 places, breaking the itest compileTestJava task. Switch to the 1-arg form; behavior is preserved since the derived lastColumnId equals the highestFieldId that was passed explicitly. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01LLnVvLYnxjTuH2b5nAHa4A
…lier commits Purely cosmetic javadoc line-wrapping fixes that spotlessApply produces on four files touched during the 1.11 work (REST controller, PartitionSpecMapper, an e2e test, and the H2 test-fixtures repository). No code changes. Keeps the pre-commit spotless gate green. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01LLnVvLYnxjTuH2b5nAHa4A
Adds test sources to the spark-4.0 itest module (previously classpath-only for the delta-harness). OpenHouseRestSparkITest boots the embedded 1.11 OpenHouse server and wires Spark 4.0 to the STOCK org.apache.iceberg.rest.RESTCatalog at /iceberg (bearer token), mirroring harness.OpenHouseEnv. Client and server share ONE Iceberg (1.11.0-openhouse), so the in-JVM two-version collision that sinks the legacy 1.5-client/1.11-server e2e lane cannot occur here. Includes the Spark-4.0-on-Java-17 add-opens set and the spark.driver.host pin (the local executor fetches codegen classes over the driver's netty RPC; without it the fetch is refused with RemoteClassLoaderError, which poisons Iceberg 1.11's FormatModelRegistry.<clinit>). First ported class: CatalogOperationTestSpark4_0 (3 pure-SQL cases, all green). Port plan for the rest of the catalogtest suite in docs/spark4-iceberg-upgrade/spark4-e2e-tests/00-PLAN.md. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01LLnVvLYnxjTuH2b5nAHa4A
The legacy Spark-3.5 e2e suites embed the OpenHouse server (now Iceberg 1.11) in one JVM alongside a 1.5 Iceberg client / data-plane, so two org.apache.iceberg class-sets collide (NoSuchMethodError / NoClassDefFoundError). This is a test-harness artifact — in production the Spark job and the server are separate processes — and none of what these suites cover is on the Spark-4.0 REST-first 1.11 lane (stock RESTCatalog, no custom OpenHouse SQL extension). - openhouse-spark-3.5-itest: whole module test execution disabled (measured on 1.11: catalogTest 58/58, statementTest 60/60, test 49/90 failing). compileTestJava still runs. Coverage moves to the Spark-4.0 REST port + the delta-harness. - apps-1.5: the 6 data-plane compaction / merge-on-read delete-file cases (OperationsTest x2, SparkMoRFunctionalTest x4) excluded; the other 220 still run and pass. No Spark-4.0 apps module exists yet, so these have no 1.11 replacement. Both gates are documented and reversible in docs/spark4-iceberg-upgrade/spark4-e2e-tests/20-legacy-gated.md. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01LLnVvLYnxjTuH2b5nAHa4A
Adds PartitionTestSpark4_0 (4 green), CTASNonNullTestSpark4_0 (1 green) and WapIdTestSpark4_0 (9 green) to the Spark-4.0 / Iceberg-1.11 / REST-first itest module, extending OpenHouseRestSparkITest. Pure-SQL and stock Iceberg procedure cases port verbatim. Records residuals: the custom SET POLICY/GRANT statements in the WAP workflow are dropped, and two engine-behavioral deltas (nested-transform DESCRIBE rendering, expire-on-empty-refs) are noted. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01LLnVvLYnxjTuH2b5nAHa4A
Adds BranchTestSpark4_0 (17 green): multi-branch WAP staging, cherry-pick, fast_forward and expire_snapshots. All stock Iceberg SQL + openhouse.system.* procedures, so a verbatim base-class swap to OpenHouseRestSparkITest. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01LLnVvLYnxjTuH2b5nAHa4A
Adds RTASTestSpark4_0 (5 green) and BranchJavaTestSpark4_0 (1 green), both driving the Iceberg Java API through the stock RESTCatalog. The replace.enabled gate is still enforced by the embedded OpenHouse server and surfaces as a BadRequestException via the REST envelope. Drops the custom SET POLICY line and the policies-property assertion from RTAS (recorded in 10-RESIDUALS.md). Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01LLnVvLYnxjTuH2b5nAHa4A
Adds the remaining CatalogOperationTest cases to the Spark-4.0 REST lane (14 green total): catalog write API, catalog-API rename, case-sensitivity rename, WRITE ORDERED BY sort-order set/unset/CTAS/multi-column/null-order/insert, and the two field-id-reassignment cases. Fixes a REST-lane wiring issue where the OpenHouse /iceberg controller NPEs on a null PartitionSpec (now passes PartitionSpec.unpartitioned()). Drops the SET POLICY case, the openhouse.tableUri assertion, and testRenameTableFailsConflict (rename-onto-existing is not rejected on this lane). All residuals recorded in 10-RESIDUALS.md. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01LLnVvLYnxjTuH2b5nAHa4A
…iduals Per direction: gated/failing tests are a triage-and-fix backlog to be worked once the Spark-4.0 e2e port completes, not ignored. Enumerates each gated group (catalogtest, statementtest, default test, apps-1.5 data-plane) as fix items with the concrete fix strategy per group. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01LLnVvLYnxjTuH2b5nAHa4A
…aEvolution) + fix checklist Final batch of the Spark-4.0 / Iceberg-1.11 / REST-first e2e port (sub-agent work): InvalidMetadataTestSpark4_0 and SparkMultiSchemaEvolutionTestSpark4_0, plus expanded CatalogOperation Java-API cases and base-class helpers. 67 @test across 9 classes; 2 @disabled kept in code (not deleted) with the exact failure, root cause, and fix recorded in 10-RESIDUALS.md as a fix backlog: - testAlterTableUnsetReplicationPolicy (needs custom OpenHouse SQL ext / policies model on the REST lane) - testRenameTableFailsConflict (server: /iceberg rename should 409 on existing dst) Plus documented inline custom-SQL omissions and engine deltas. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01LLnVvLYnxjTuH2b5nAHa4A
…fix investigation note Applies the spotless javadoc reflow to SparkMultiSchemaEvolutionTestSpark4_0 and records attempt-1 findings on the rename-409 backlog item (a controller tableExists guard did not trip via the Spark-4.0 SQL rename path; the actual REST request must be traced first). testRenameTableFailsConflict stays @disabled on the fix backlog — not shipped unverified. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01LLnVvLYnxjTuH2b5nAHa4A
…tion from no-verify commit) Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01LLnVvLYnxjTuH2b5nAHa4A
The test created its REPLICA_TABLE via the Iceberg Java API with openhouse.isTable Replicated=true, which trips the server's validateUUIDForReplicaTable guard (requires a valid last-updated-ms) — so createTable threw BadRequestException and the bare finally DROP masked it as NoSuchTableException. The test only needs tableType=REPLICA_TABLE (to permit the direct multi-schema ops.commit), exactly like the sibling REPLICA_TABLE tests which don't set isTableReplicated. Dropped the extra flag; also made cleanup DROP TABLE IF EXISTS so any future body failure surfaces its real cause instead of a masking NoSuchTableException. This was the only failing test in the spark-4.0 e2e suite (was 65/1/2 -> 66/0/2). Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01LLnVvLYnxjTuH2b5nAHa4A
Diagnosed: corrupt current-schema-id -> refreshMetadata throws InvalidTableMetadataException; the INSERT's Iceberg commit-retry loop retries it up to the ~30min total-timeout, so the expected failure surfaces after ~19min. Fix (backlog): map corrupt-metadata to a non-retryable 4xx server-side. Test passes; runtime cost only. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01LLnVvLYnxjTuH2b5nAHa4A
…og item Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01LLnVvLYnxjTuH2b5nAHa4A
… → ~19min retry) InvalidTableMetadataException fell through the controller's exception handlers to handleGeneric -> HTTP 500. The stock RESTCatalog client treats that as a transient server failure and retries the read/commit with exponential backoff up to ~30 min, so InvalidMetadataTestSpark4_0.testCorruptSchemaIdSurfacesRealError took ~19 min to surface the (correct) error — dominating the branch-1.11 CI at ~39 min. Corrupt/unparseable stored metadata is a permanent bad-state, not a transient failure, so map it to a non-retryable 400. The 'has invalid metadata' message is preserved. Verified locally: the test now completes fast (build 5m17s incl. the services:tables recompile, vs the test alone previously ~19 min). Halves CI time. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01LLnVvLYnxjTuH2b5nAHa4A
…ST lane) Renaming a table ONTO an already-existing table was silently accepted on the Spark-4.0 / Iceberg-1.11 REST-first lane. The covering test CatalogOperationTestSpark4_0#testRenameTableFailsConflict was @disabled and asserted a custom-client exception type that does not apply on the stock RESTCatalog lane. Root cause. OpenHouseInternalCatalog.renameTable (reached from the /iceberg controller via CatalogHandlers.renameTable -> catalog.renameTable) performed no destination-existence check, so a rename onto an occupied name was accepted. A naive tableExists(to) guard -- whether in the controller (a prior attempt using catalog.tableExists(request.destination())) or in the catalog -- does not fire, for two reasons discovered by tracing the actual identifiers and HTS keys: 1. Spark's SQL `ALTER TABLE openhouse.db.x RENAME TO openhouse.db.y` on the REST lane leaks the Spark catalog name into the DESTINATION namespace: the catalog receives to=openhouse.db.y (namespace [openhouse, db]) while the source is correctly stripped (namespace [db]). The pre-existing conflict table was created via the Java catalog API under the clean namespace [db], so tableExists(to) looks up the wrong, catalog-prefixed namespace and returns false -- the conflict is invisible. This is exactly why the prior controller guard never threw. 2. OpenHouse rename is single-database by construction: UserTablesServiceImpl.renameUserTable discards the destination databaseId and reuses the source's (it only renames the tableId within the source db). That is also why the non-conflicting SQL renames pass despite the leaked namespace -- HTS absorbs it. Fix. Check the conflict at the EFFECTIVE destination -- from.namespace() combined with to.name(), i.e. exactly the HTS row the rename will write -- and throw org.apache.iceberg.exceptions.AlreadyExistsException (already mapped to HTTP 409 by the controller's handleConflict) when a table already exists there. A renamingToSameTable guard avoids a false positive when a table is renamed to its own name. The check goes in the catalog so all callers benefit. tableExists resolves through HTS findById, which is case-insensitive on db and table, so the guard matches an existing destination regardless of case. Test. Removed @disabled and the unused WebClientResponseWithMessageException import; replaced the client-specific exception-type assertion with an engine-agnostic behavioral one (rename is rejected AND both the source and the pre-existing destination survive intact -- the destination's user.property was never set, proving no silent replace). Verification (targeted runs, JDK17): - testRenameTableFailsConflict ....... PASSED (was @disabled) - testRenameTable .................... PASSED (rename to non-existent, SQL) - testRenameTableCatalogApi .......... PASSED (rename to non-existent, SQL) - testRenameTableCaseSensitivity ..... PASSED (rename to non-existent, Java API) All four together: BUILD SUCCESSFUL, 0 failures. Full trace + before/after documented in docs/spark4-iceberg-upgrade/spark4-e2e-tests/rename-409-fix.md. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01LLnVvLYnxjTuH2b5nAHa4A
Add docs/spark4-iceberg-upgrade/spark4-e2e-tests/backlog-triage.md classifying
the residual legacy Spark-3.1/3.5/apps-1.5 test backlog against the Spark-4.0
REST-first lane. Per-group and per-notable-test disposition with root cause and
exact fix-cost, reconstructable later.
Headline feasibility finding (policy DDL on the REST lane): making the custom
OpenHouse SET/UNSET POLICY, GRANT/REVOKE, column SET TAG DDL work via table
properties on the Spark-4.0 REST lane is NOT tractable in this pass, blocked by
three stacked, independently-disqualifying issues:
1. No Spark-4.0/Scala-2.13 build of the extension. It lives only in
spark-3.1/spark-3.5 openhouse-spark-runtime (Scala 2.12, ANTLR grammar +
AST builder + logical plans + V2 strategy + *Exec). spark-4.0 has only a
Java-only openhouse-spark-itest and no runtime module; loading it needs a
new Scala-2.13 module ported to Spark 4.0's ParserInterface/Catalyst APIs.
2. The policy "table property" (updated.openhouse.policy) is a CLIENT-ONLY
patch channel consumed solely by the legacy OpenHouseTableOperations
(buildUpdatedPolicies, lines 215/277-324), which folds it into the
structured Policies object of the /tables request. The stock RESTCatalog
never uses that class; the /iceberg server has ZERO references to
updated.openhouse.policy, so the DDL would no-op. The real `policies` and
`openhouse.*` keys are server-reserved (enforceReservedPropsUnchanged ->
400 on UPDATE, stripped on CREATE), so they cannot be set as raw
TBLPROPERTIES either. Making SET POLICY real requires a NEW server-side
translation on the /iceberg commit path.
3. GRANT/REVOKE/SHOW GRANTS reflectively cast the catalog to
SupportsGrantRevoke; the stock RESTCatalog does not implement it and
/iceberg has no grant endpoint -> UnsupportedOperationException.
Consequently no statementtest policy case, CatalogOperationTestSpark4_0
.testAlterTableUnsetReplicationPolicy, nor the RTAS/WapId inline policy
omissions can pass on the REST lane today; all left as recorded parity gaps
with the exact remaining work (no faked green, nothing re-enabled).
Group 2 (default `test`, ~90): the entire group is legacy-client mock-web-server
validation (SparkTestBase + MockWebServer + custom OpenHouseCatalog). Not a
REST-lane coverage gap; the substantive DDL/DML behaviors are already covered by
the existing Spark-4.0 catalogtest port (coverage map recorded). Mock-internals
(DoCommit/DoRefresh/ServerClientExceptionMapping/IcebergCatalogMapper) have no
stock-RESTCatalog analogue.
Group 3 (apps-1.5, 6 data-plane compaction/MoR-delete tests): deferred to the
separate batched-apps rewrite; no Spark-4.0 apps module proposed here to avoid
conflicting with that effort.
Docs-only; no source or test changes. 10-RESIDUALS.md and 20-legacy-gated.md
left untouched (main session reconciles their checkboxes).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Claude-Session: https://claude.ai/code/session_01LLnVvLYnxjTuH2b5nAHa4A
…tended, not omitted-by-design Per policy: the OpenHouse SQL extension (SET/UNSET POLICY, GRANT, column tags) should be loaded on the Spark-4.0 REST lane and mapped to table-property operations. It is not yet wired (needs a Scala-2.13/Spark-4.0 port of the extension + a /iceberg server-side policy-property translation + a grant endpoint; see spark4-e2e-tests/backlog-triage.md). Corrected the base-class comment that wrongly framed this as a permanent design choice. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01LLnVvLYnxjTuH2b5nAHa4A
…licies on the Spark-4.0 REST lane
Make OpenHouse table POLICIES settable and readable through the stock
org.apache.iceberg.rest.RESTCatalog (Spark-4.0 / Iceberg-1.11) by having the
/iceberg/v1 server endpoint ACCEPT and TRANSLATE a policy table-property into
OpenHouse's structured Policies model -- mirroring what the native /tables path
does -- instead of stripping it or rejecting it as a reserved-property violation.
This is the server-side foundation a future Spark-4.0 SET POLICY SQL extension
maps onto.
Client property contract. A stock RESTCatalog sets a policy by carrying it as the
table property `updated.openhouse.policy`, whose value is the JSON serialization of
a (partial) OpenHouse Policies object (e.g. {"retention":{count,granularity,
columnPattern}} or {"sharingEnabled":true}). This is deliberately identical to the
legacy Spark Set*PolicyExec encoding (consumed by the legacy client's
OpenHouseTableOperations.buildUpdatedPolicies), so a future port of the SET POLICY
DDL emits the same property with no client change. The granularity token is
UPPERCASE, matching both the server Granularity enum names and
GranularityEnum.getValue() in the legacy AST builder.
Server changes (IcebergRestCatalogController):
- CREATE: toCreateUpdateTableRequestBody now intercepts `updated.openhouse.policy`,
strips it from the pass-through properties, translates it to a Policies object, and
sets it on the reused CreateUpdateTableRequestBody. The existing create pipeline
(TablesApiHandler.createTable) then validates (validatePolicies) and persists it
into the reserved `policies` property, exactly as /tables does.
- UPDATE/commit: a new isPolicyUpdate() dispatch (after the CTAS and RTAS branches,
before the plain-update guards) routes a SetProperties commit carrying the policy
carrier through updatePolicy(), which loads the base, rejects a locked table,
projects the commit, merges the policy patch onto the existing policy
(translatePolicyPatch), strips the carrier, and hands the full request to the reused
TablesApiHandler.updateTable (validation + TablePolicyManager persistence into
`policies`). The carrier is a non-reserved key, so the reserved-property guard sees
identical reserved props before/after and does not 400 it; the guard remains fully
intact for genuine openhouse.*/policies mutations.
- translatePolicyPatch reproduces the legacy client's buildUpdatedPolicies merge
server-side (the stock client cannot merge): each sub-policy present in the patch
overrides, absent ones are preserved. sharingEnabled presence is detected from the
raw patch JSON because the server Policies model uses a primitive boolean. Parsing
reuses the shared PoliciesSpecMapper bean so the lanes cannot drift.
All existing native-/tables behavior and the reserved-property protection for
non-policy props are unchanged; only the legitimate policy-setting path is added.
Verification. New PolicyPropertyTestSpark4_0 (extends OpenHouseRestSparkITest, drives
the stock RESTCatalog): retention-on-update, sharing-on-create, sharing-on-update,
successive-updates-merge, and non-policy-reserved-prop-still-rejected -- 5 tests, all
GREEN. Existing services:tables policy tests (PoliciesSpecMapperTest,
TablePolicyManagerTest, RetentionPolicySpecValidatorTest, HistoryPolicySpecValidatorTest)
remain GREEN, and CatalogOperationTestSpark4_0 insert/alter cases were spot-checked
for no regression. Full audit in
docs/spark4-iceberg-upgrade/spark4-e2e-tests/policy-rest-lane.md.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Claude-Session: https://claude.ai/code/session_01LLnVvLYnxjTuH2b5nAHa4A
… tableExists (fixes :services:tables:test)
Follow-up to the rename-409 guard in OpenHouseInternalCatalog.renameTable. The
guard used catalog.tableExists(effectiveDestination) to detect a rename onto an
existing table. tableExists() performs a full loadTable() -> refreshMetadata(),
which for a destination whose HTS row is ABSENT (the normal case: renaming to a
new name) throws InvalidTableMetadataException ("Failed to open input stream for
file ... .metadata.json") instead of returning false. This broke
RepositoryTest.testRenameTableMetadataUpdate on the branch-1.11 CI (a rename to a
brand-new name blew up in the existence pre-check).
Fix: use the existing findHouseTable(identifier) helper, which reads ONLY the HTS
row (no metadata.json parse) and returns Optional.empty() when the destination
does not exist -- exactly what a pre-rename conflict check needs. The real
conflict case is unchanged: findHouseTable(existing).isPresent() == true still
throws AlreadyExistsException (mapped to HTTP 409 by the controller).
Test isolation: the guard also exposed a latent leak in RepositoryTest. Two
sibling tests (testRenameTableMetadataUpdate, testRenameTablePreserveExistingCase)
both rename to d1.t1_renamed and never deleted it -- previously masked by the old
silent-overwrite rename. With the guard, whichever ran second either hit the
conflict OR left d1.t1 in a partial state, which then surfaced downstream as
NPEs (TableType.valueOf(null) in OpenHouseInternalRepositoryImpl.skipEligibilityCheck)
in testMetadataUpdateForDeleted and testOpenHouseInvalidClusteringEvolution.
Added the class's standard deleteById cleanup to both rename tests; the full
RepositoryTest (28 tests) is now green.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Claude-Session: https://claude.ai/code/session_01LLnVvLYnxjTuH2b5nAHa4A
… / Scala-2.13 (SET POLICY works end-to-end on the REST lane) Make the OpenHouse custom `ALTER TABLE ... SET POLICY (...)` DDL work on the Spark-4.0 / Iceberg-1.11 REST-first lane by porting the OpenHouse Spark SQL extension from the legacy Scala-2.12 / Spark-3.x runtime to Spark 4.0 / Scala 2.13. The DDL lowers onto the `updated.openhouse.policy` Iceberg table property that the `/iceberg` server foundation (commit da20ad4) already translates into OpenHouse `Policies` — so this is a pure client-side port with no contract change. New module. `integrations/spark/spark-4.0/openhouse-spark-runtime`, Gradle project `:integrations:spark:spark-4.0:openhouse-spark-runtime` renamed in settings.gradle to `openhouse-spark-4.0-runtime_2.13`. A `scala` + `java-library` module (Scala 2.13.16, Spark 4.0.0, Iceberg 1.11.0-openhouse, Java 17) that deliberately avoids `openhouse.java-minimal-conventions` (which pins Java 11 and rejects the Java-17 Spark 4.0 jars), mirroring the sibling itest module. It provides `com.linkedin.openhouse.spark.extensions.OpenhouseSparkSessionExtensions`. The ANTLR grammar (.g4) is copied VERBATIM from spark-3.1 and generated with the ANTLR 4.13.1 tool (matching Spark 4.0's bundled antlr4-runtime 4.13.1 — the 4.7.1 tool would emit an ATN-version-incompatible parser). Granularity tokens (DAY/HOUR/MONTH/YEAR) are inlined as string literals so the module needs no tables gen-model dependency (those values are exactly the uppercase tokens the server contract expects). Spark-4.0 / Scala-2.13 API adaptations (only what the API changes forced): - ParserInterface: added parseQuery + parseRoutineParam overrides (both delegate), dropped parseRawDataType (no longer in the contract). - Logical plans extend LeafCommand (Spark 4.0's Command no longer supplies leaf semantics; LeafCommand = Command with LeafLike). - Physical execs extend LeafV2CommandExec (V2CommandExec no longer supplies children/withNewChildrenInternal). - Planner strategy extends org.apache.spark.sql.execution.SparkStrategy directly (the org.apache.spark.sql.Strategy type alias was removed). - Dropped the removed-in-2.13 scala.collection.JavaConversions import. The GRANT/REVOKE/SHOW-GRANTS execution path (GrantRevokeStatementExec, ShowGrantsStatementExec, IcebergCatalogMapper, Principal) and OpenHouseCatalog.java are NOT ported — they require a SupportsGrantRevoke catalog that does not exist on the stock-RESTCatalog REST lane and pull Iceberg-core-internal deps. The GRANT grammar + logical plans + AST builder are retained (they compile as pure Catalyst), so those statements still parse; they simply have no wired physical exec. Harness wiring (integrations/spark/spark-4.0/openhouse-spark-itest): - build.gradle: testImplementation on the new runtime module. - OpenHouseRestSparkITest.getBuilder: spark.sql.extensions now registers the OpenHouse extension alongside the stock Iceberg extension. The OpenHouse parser only intercepts its own DDL and delegates everything else, so this is transparent to every existing REST-lane test. Tests (all GREEN, targeted JDK-17 runs): - PolicySqlDdlTestSpark4_0 (NEW, 4 tests): drives the real SET POLICY DDL (retention/replication/sharing/history) end-to-end against the embedded server and reads `policies` back via SHOW TBLPROPERTIES. - statementtest ports (19 tests): SetTablePolicy / SetTableReplicationPolicy / SetHistoryPolicy / SetSharingPolicy Spark4_0 (parse/plan on a local hadoop catalog). One rendering delta adapted: Spark 4.0 backtick-quotes leading-digit identifiers (`0_`.`0_`) where 3.1 rendered 0_.0_. - RTASTestSpark4_0.testRTAS: restored inline SET POLICY (HISTORY MAX_AGE=24H); assertion updated to the verified REST-lane behavior (RTAS PRESERVES the policy rather than clearing it — a server behavioral delta from the legacy catalog). - WapIdTestSpark4_0.testWapWorkflowWithVariousOperations: restored inline SET POLICY (SHARING=TRUE) + readback (GRANT stays dropped). Still @disabled with an updated reason: CatalogOperationTestSpark4_0.testAlterTableUnsetReplicationPolicy — SET POLICY now works, but this test also exercises UNSET POLICY (REPLICATION). UnSetReplicationPolicyExec emits {"replication": {}}, which the /iceberg server rejects with HTTP 400 ("Replication config cannot be null"): the server merge only overrides present sub-policies and has no clear/tombstone convention for removing one. Needs a server-side fix; the SET steps this test runs already pass. Docs: adds spark4-e2e-tests/policy-sql-extension-spark4.md (audit-grade: module layout, every Spark-4.0 API change, what works + is tested, what remains + why, full reconstruction checklist) and updates 10-RESIDUALS.md to the new state. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01LLnVvLYnxjTuH2b5nAHa4A
ALTER TABLE ... UNSET POLICY (REPLICATION) on the Spark-4.0 / Iceberg-1.11
REST lane was rejected by the /iceberg server with HTTP 400 "Replication
config cannot be null." The Spark exec UnSetReplicationPolicyExec emits the
policy carrier property updated.openhouse.policy = {"replication": {}}. The
server merge in IcebergRestCatalogController.translatePolicyPatch only had an
OVERRIDE convention: it deserialized {"replication": {}} into a non-null
Replication with a null config and set it on the merged Policies, which then
tripped the @NotNull constraint on Replication.config during the update
bean-validation cascade. There was no clear/tombstone convention for REMOVING
a sub-policy, so an empty sub-policy object was indistinguishable from an
invalid one and the UNSET step always failed.
Fix: translatePolicyPatch now interprets a sub-policy present in the raw patch
JSON as an EMPTY object ({}) as a clear/tombstone and drops it from the merged
Policies (sets it to null on the builder) instead of overriding with an
invalid empty object. Detection is done on the raw patch JSON via a new
isClearedSubPolicy helper because the parsed server Policies/Replication model
cannot distinguish an absent sub-policy from an empty-but-present one. The rule
is applied consistently to the object sub-policies replication, retention,
history and columnTags (sharingEnabled is a primitive boolean, not an object).
The existing non-empty SET override paths are untouched, since a real SET
always carries a non-empty object. The merge now also builds from a fresh
Policies.builder() when the table has no existing policy, so an UNSET on a
table that never had the sub-policy also strips the empty object rather than
persisting an invalid one. Because a cleared replication is null, the @NotNull
config cascade and ReplicationConfigValidator are never triggered on a clear.
Native-lane parity finding: the legacy /tables client merge
(OpenHouseTableOperations.buildUpdatedPolicies) had the same override
structure but did not 400, because the GENERATED client Replication model
initializes config to an empty ArrayList. Gson left that default in place when
deserializing {"replication": {}}, so the client silently sent
{"replication":{"config":[]}} and the native catalogtest read back
replication.config.size() == 0 -- replication PRESENT with an empty config
list, not removed. The server-side Replication model has no such default, so
on the REST lane the same empty object deserializes to config == null. The
native lane therefore never truly cleared replication; it stored an
empty-but-valid config object purely as an artifact of the client gen-model.
The REST lane removes the sub-policy outright, which is the cleaner behavior;
both mean "no active replication" and the difference is not observable through
the behavioral test.
Re-enables CatalogOperationTestSpark4_0.testAlterTableUnsetReplicationPolicy
(removed @disabled and the now-unused import). The post-UNSET readback is
asserted against the raw policies table-property string (there is no Policies
gen-model on this itest module's compile classpath): the serialized policy no
longer carries the replication config while retention is preserved.
Verification (targeted, all GREEN):
:services:tables:test --tests
"com.linkedin.openhouse.tables.e2e.h2.TablesControllerTest" (server guard:
policy validation + replication/history update paths)
:integrations:spark:spark-4.0:openhouse-spark-4.0-itest:test --tests
"*.CatalogOperationTestSpark4_0.testAlterTableUnsetReplicationPolicy"
"*.PolicySqlDdlTestSpark4_0" "*.PolicyPropertyTestSpark4_0"
-> testAlterTableUnsetReplicationPolicy PASSED; all 4 PolicySqlDdl SET tests
and all 5 PolicyProperty tests still PASSED (no SET-path regression).
Docs: adds spark4-e2e-tests/unset-policy-clear-semantics.md (audit-grade
failure/root-cause/native-parity/fix/verification) and moves the item to FIXED
in spark4-e2e-tests/10-RESIDUALS.md.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Claude-Session: https://claude.ai/code/session_01LLnVvLYnxjTuH2b5nAHa4A
Add ColumnTagsTestSpark4_0 to prove ALTER TABLE ... MODIFY COLUMN ...
SET TAG works end-to-end on the Spark-4.0 / Iceberg-1.11 REST lane. The
DDL is parsed by the ported OpenHouse extension, lowered by
SetColumnPolicyTagExec onto the updated.openhouse.policy carrier as
{"columnTags":{"<col>":{"tags":[...]}}}, and merged by the server's
translatePolicyPatch columnTags branch into the reserved policies
property.
Covers single tag (PII), multiple tags (PII, HC), and the clear form
(NONE). All three pass with no code change required; column tags work
as-is on the REST lane. NONE emits an empty tags array (clears the
column's tags) rather than a tombstone; documented in
column-tags-verification.md.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Claude-Session: https://claude.ai/code/session_01LLnVvLYnxjTuH2b5nAHa4A
The Spark 4.0 rename catalog test had dropped the legacy 3.1 assertion on the OpenHouse reserved property openhouse.tableUri, with a NOTE claiming the property was "not surfaced on the stock REST lane." Empirical probing shows the property IS surfaced end to end on the /iceberg loadTable lane and is correctly updated by rename. It was never stripped; the legacy assertion only failed because the value differs: on the REST lane the identifier namespace carries the Spark catalog prefix, so the value is local-cluster.openhouse.db.rename_test_renamed rather than the legacy local-cluster.db.rename_test_renamed. No server change is required. Restore the assertion, made robust to the embedded cluster name and the REST-lane namespace rendering by asserting the tableUri ends with db.rename_test_renamed. Add an audit-grade write-up under docs/spark4-iceberg-upgrade/spark4-e2e-tests/tableuri-loadtable.md. Verified green (targeted): testRenameTableCatalogApi PASSED testRenameTableCaseSensitivity PASSED Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01LLnVvLYnxjTuH2b5nAHa4A
…ANT deferral Close out the Spark-4.0 / Iceberg-1.11 REST-first e2e fix backlog in 10-RESIDUALS.md now that every fixable catalogtest case is green: - testAlterTableUnsetReplicationPolicy: FIXED (d91cfdf, empty-object clear semantics) — already recorded. - testRenameTableFailsConflict: marked FIXED. It is a live @test (not @disabled) and was already made green by the catalog-level effective-destination rename guard in 956820c; the checklist was stale. Confirmed GREEN locally (targeted run) and in branch-1.11 CI. Documents why the catalog-level guard fires where the earlier controller-level tableExists(destination) guard did not (single-DB rename + Spark REST lane leaking the catalog name into the destination namespace). - testRenameTableCatalogApi (openhouse.tableUri): marked FIXED (923d39b). The property was surfaced by the stock REST loadTable all along; only the expected value differed (REST-lane namespace rendering adds the catalog prefix). Assertion restored as a tail-match. - Column policy-tag DDL: marked VERIFIED (f42268d) — server already merges the columnTags sub-policy; no code change needed. Notes the SET TAG = (NONE) empty-array clear caveat. - GRANT / REVOKE / SHOW GRANTS: recorded as an explicit, documented DEFERRAL (needs a server-side ACL endpoint on the /iceberg lane; disproportionate to this modernization spike), NOT a silent omission — with a concrete close-it-later plan. Rewrites the intro to reflect the all-green state and clarifies the checkbox semantics ([x] fixed+verified, [ ] deferred-with-reason). Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01LLnVvLYnxjTuH2b5nAHa4A
The ~19-min InvalidMetadataTestSpark4_0 runtime was already resolved: IcebergRestCatalogController maps InvalidTableMetadataException to a non-retryable HTTP 400 (handleBadRequest, ~line 1063), so the INSERT's Iceberg commit-retry loop no longer backs off for ~19 min on corrupt metadata. Update the checklist item from open to fixed for accuracy. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01LLnVvLYnxjTuH2b5nAHa4A
…ungs Durable, review-oriented tracker indexing the sequential remaining work (GRANT/REVOKE REST-lane impl, Rung 3 v3 deletion vectors, Java-17 server runtime, final delta-harness validation, HDFS/v3 spike), each linking to its own audit-grade write-up. Records pre-dispatch findings: the build is already on 1.11.0-openhouse, and the server forces TableProperties.FORMAT_VERSION from the cluster config cluster.iceberg.format-version (default 2), which is the lever for enabling v3 deletion vectors. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01LLnVvLYnxjTuH2b5nAHa4A
…a /aclPolicies
The stock RESTCatalog on the Spark-4.0 / Iceberg-1.11 REST lane does not implement
SupportsGrantRevoke (the legacy hook) and its /iceberg surface has no ACL sub-resource,
so the old downcast path is a dead end. Instead the new execs call the EXISTING OpenHouse
server ACL endpoint directly over HTTP: PATCH/GET {base}/v1/databases/{db}/tables/{t}/aclPolicies,
where {base} is the catalog uri with the trailing /iceberg stripped and the same bearer token.
Privilege->role mapping mirrors javaclient/mapper/Privileges.java exactly.
- Add OpenHouseAclClient (URI/token derivation, privilege<->role maps, PATCH/GET), the
GrantRevokeStatementExec + ShowGrantsStatementExec, and the ported Principal (PUBLIC<->*) constant.
- Wire GrantRevokeStatement / ShowGrantsStatement cases into OpenhouseDataSourceV2Strategy
(grammar + logical plans + AST builder were already present).
- Add GrantRevokeTestSpark4_0: GRANT/REVOKE return 204 against the real embedded server, and a
capturing HttpServer stub proves the exact request (operation+role+principal+path+bearer) and
SHOW GRANTS row parsing. A full in-JVM ACL roundtrip is not possible (embedded
OpaAuthorizationHandler no-ops without an external OPA store), so this matches the legacy
client-contract bar.
- Restore the dropped GRANT SELECT ... TO lejiang in WapIdTestSpark4_0.
- Docs: add grant-revoke-rest-lane.md; move GRANT from DEFERRED to FIXED in 10-RESIDUALS.md;
tick section 1 of REMAINING-WORK.md; supersede the stale GRANT notes in
policy-sql-extension-spark4.md.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Claude-Session: https://claude.ai/code/session_01LLnVvLYnxjTuH2b5nAHa4A
Record the GRANT/REVOKE landing SHA in the change log and mark Rung 3 (v3 deletion vectors) as in-progress in REMAINING-WORK.md. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01LLnVvLYnxjTuH2b5nAHa4A
…ane (Rung 3) Rung 3 — the goal of the Spark-4.0 / Iceberg-1.11 modernization spike: prove v3 DataSource-V2 deletion vectors end-to-end on the Spark-4.0 REST lane with a real test, without flipping the currently-green default-v2 itest suite. OpenHouse forces every table's format-version to the cluster default (cluster.iceberg.format-version, default 2) on create/replace, so v3 can only be authored by raising that default — which is JVM-global given the per-JVM singleton embedded server. To keep the green suite at v2, isolate v3 to its own JVM fork: a new `deletionVectorTest` Gradle Test task runs ONLY DeletionVectorTestSpark4_0 with -Dcluster.iceberg.format-version=3, and that class is excluded from the default `test` task. Isolation is by fork, not JUnit ordering. DeletionVectorTestSpark4_0 (2 methods, PASSED): - testServerAuthorsFormatVersion3: metadata-writer v3 proof (table property==3, TableMetadata.formatVersion()==3, on-disk metadata JSON "format-version":3). - testMergeOnReadDeleteWritesDeletionVector: MOR DELETE writes a puffin deletion vector (.delete_files file_format=PUFFIN, physical *-deletes.puffin carrying a deletion-vector-v1 blob, zero PARQUET pos-deletes); deleted row gone on read-back, survivors [1,3,4,5,6]. Default-v2 path verified unchanged (CatalogOperationTestSpark4_0.testCasingWithCTAS PASSED); DV test proven excluded from the default fork. Write-up rung3-v3-deletion-vectors.md incl. the cost assessment of a global v3 default (v2 read cliff -> Rung 9). REMAINING-WORK section 2 ticked. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01LLnVvLYnxjTuH2b5nAHa4A
…r Java-8 bytecode Rung 7 (server/metadata-writer runtime -> Java 17, keep Java-8 bytecode where consumed) is already satisfied by the 1.11 upgrade architecture; this documents and empirically verifies it. - Runtime = Java 17: CI (build-run-tests.yml "Set up JDK 17") and the embedded Spring Boot 2.7 server (booted in the JDK-17 test JVM by the whole Spark-4.0 REST itest e2e suite, incl. the Rung-3 v3 server) run on Java 17; no module pins a Java-8/11 toolchain. - Bytecode = Java 8 where consumed: java-minimal-conventions keeps targetCompatibility=1_8; the metadata-writer OpenHouseInternalCatalog / OpenHouseInternalTableOperations compile to major version 52 (Java 8) on a JDK-17 compiler (verified with javap) and advertise Java 8 to consumers, so Java-8 readers of the metadata output keep working. No code change required. Write-up rung7-java17-runtime.md; REMAINING-WORK.md section 3 ticked + Rung 3 SHA (69d83d6) recorded in the change log. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01LLnVvLYnxjTuH2b5nAHa4A
…17 + RBF) The OpenHouse server links hadoop-client to write table metadata to HDFS. On the 1.11 branch that pin was still the legacy 2.10.0, which is unfit for the Java-17 server runtime: it predates Java 17 and lacks the Hadoop 3.3+ FileSystem.openFile(Path) API that Iceberg 1.10/1.11 calls. The Spark itest lane only masked this by excluding the server's unshaded 2.10 Hadoop and borrowing Spark 4.0's shaded 3.4.1, so the declared server dependency was stale. Move openhouse.hadoop-conventions hadoopVersion 2.10.0 -> 3.3.6: 3.3.6 has openFile, runs on Java 17 (with the launcher --add-opens already applied), and its RPC client is wire-compatible with a 3.1/3.2 RBF (Router-Based Federation) cluster. Fix the transitive fallout at the source (2.10 dragged in old libs 3.3.6 drops; migrate each direct use to its modern, already-present equivalent — no version band-aids): - commons-lang 2.x org.apache.commons.lang.StringUtils -> commons-lang3 (JdbcProviderConfiguration, TablesControllerTest) - Apache Directory org.apache.directory.api.util.Strings -> commons-lang3 StringUtils isNotEmpty/isEmpty (HouseTableRepositoryImpl, RequestAndValidateHelper) - codehaus Jackson 1.x org.codehaus.jackson.node.* -> fasterxml com.fasterxml.jackson.databind.node.* (PoliciesSpecMapperTest) The Spark-4.0 itest hadoop exclusions are by group+module (version-agnostic), so the itest/harness classpath is unchanged. Verified: every hadoop-conventions module compiles main+test on JDK 17 + Hadoop 3.3.6; cluster:storage:test green; the server boots (generateOpenApiDocs starts the full housetables+tables Spring context); TablesControllerTest and PoliciesSpecMapperTest green. Real-HDFS/RBF cluster validation and the v3 read cliff remain deployment gates, documented in rung9-hdfs-java17-v3-readcliff.md. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01LLnVvLYnxjTuH2b5nAHa4A
…SparkSession
The delta-harness Env wired only the stock Iceberg extension, so every custom
OpenHouse DDL scenario (ddl.policy.{retention,replication,sharing,history},
ddl.colTag, ddl.acl.grant*, and the rtas/hazard cases that SET POLICY) failed to
PARSE (ParseException) — ~24 of the 39 full-matrix failures were purely this
missing registration, not product behavior.
Now that OpenhouseSparkSessionExtensions is ported to Spark-4.0 / Scala-2.13
(module openhouse-spark-4.0-runtime_2.13, already on the harness classpath via
the itest module), register it alongside the Iceberg extension — mirroring
OpenHouseRestSparkITest.getBuilder. Verified: the ddl.policy slice goes from all
ParseException to 12 passed / 0 failed.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Claude-Session: https://claude.ai/code/session_01LLnVvLYnxjTuH2b5nAHa4A
…(Hadoop 3.3.6 fallout) The Hadoop 2.10.0 -> 3.3.6 bump (2884455) dropped the transitive commons-lang 2.x (org.apache.commons.lang) that these modules imported directly, breaking the full-repo CI build at :apps:openhouse-spark-apps-1.5_2.12:compileJava (and libs:datalayout). Local per-module compiles of the hadoop-conventions modules missed these because the full build graph pulls them via other paths. Migrate each direct use to commons-lang3 (already on every affected module's compile classpath), APIs unchanged: - apps SparkJobUtil, TableRetentionTask: org.apache.commons.lang.StringUtils -> org.apache.commons.lang3.StringUtils (isBlank) - apps JobsScheduler: org.apache.commons.lang.math.NumberUtils -> org.apache.commons.lang3.math.NumberUtils (toInt) - libs/datalayout StrategiesDaoTableProps: org.apache.commons.lang.StringEscapeUtils -> org.apache.commons.lang3.StringEscapeUtils (escapeJava/unescapeJava) Verified: `./gradlew testClasses` (whole-repo main+test compile) BUILD SUCCESSFUL on JDK 17 + Hadoop 3.3.6. A repo-wide grep confirms no remaining org.apache.commons.lang. / org.codehaus.jackson / org.apache.directory.api imports. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01LLnVvLYnxjTuH2b5nAHa4A
…42/2574) Ran the full delta-harness behavioral matrix against the real embedded OpenHouse catalog on the Spark-4.0 / Iceberg-1.11 / REST-first stack: 2542 passed, 11 skipped, 21 failed (2574 cases), 98.8% pass. First run had 39 failures; ~18 were purely the harness SparkSession not registering the OpenHouse SQL extension (fixed in 24dd99b), re-run -> 21. All 21 residuals triaged in rung8-final-validation.md — zero OpenHouse product regressions: - (A) harness assertions authored for the legacy fork/custom-client lane (rename-conflict / ns-create exception types; grant-on-unshared server-400); - (B) the REST lane FIXING legacy bugs the harness still asserts broken (G10 retention + H3 column-tag survive RTAS — the harness itself says "flip this test"); - (C) stock-Spark-4.0/Iceberg-1.11-vs-legacy-fork engine deltas (fork column DEFAULT unsupported on stock Spark 4.0, branch ref-override, concurrent-append count, partition-reject message, rename-consumer handoff). Harness-baseline polish (flip G10/H3, update category-A exception types) is noted as follow-up; it is validation-tool bookkeeping, not product work. REMAINING-WORK section 4 ticked. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01LLnVvLYnxjTuH2b5nAHa4A
…GAPS register Two reviewer-facing documents closing out the modernization: - CHANGE-HISTORY.md: lists the in-scope repos (mkuchenbecker/openhouse on branch 1.11 / PR #17 draft; mkuchenbecker/iceberg fork on 1.11.x, consumed as source, no PR), then walks the diffs sequentially per repo. OpenHouse = 54 commits (+17,307/-109 across 132 files) grouped into 7 phases from the REST-first cutover through the rung ladder. Fork = the 1.11 port (replay LinkedIn core patches onto Apache Iceberg 1.11.0 + the F-REPL delete-file-replication fix, 12 files +233/-13) plus the branch layout. - KNOWN-GAPS.md: the honest register of what is NOT done — (A) harness-baseline polish (flip the G10/H3 bugs the REST lane fixed, category-A exception-type assertions, the surface.conc.appendAppend concurrency deep-dive, category-C stock-vs-fork engine deltas), (B) deployment gates (real-HDFS-cluster run, v3 default flip gated on the read cliff, GRANT/REVOKE server ACL persistence needing OPA), and (C) deferred scope (maintenance-apps Spark-4.0 port; PR #17 draft→merge decision). None blocks the spike; all cross-referenced to the per-rung write-ups. REMAINING-WORK.md gains a companion-docs pointer. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01LLnVvLYnxjTuH2b5nAHa4A
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Consolidated branch (
1.11) that moves OpenHouse's engine integration to Spark 4.0 / Iceberg 1.11 / Scala 2.13 / Java 17 with v3 DataSource-V2 deletion vectors, via a REST-first cutover: Spark talks to OpenHouse through a stock IcebergRESTCatalogagainst a new server-side/icebergendpoint — no custom Spark catalog jar. Built on top of currentmain(incl. #13 modular harness, linkedin#640 RTAS gate, linkedin#641 metadata hook, linkedin#644 config map). Supersedes the earlier stacked branches (upgrade + validation-parity), folded here per one-branch consolidation.Changes
IcebergRestCatalogController(/iceberg/v1/...) drivingCatalogHandlersoverOpenHouseInternalCatalog;updateTablecomposes CTAS→staged-create, RTAS→service replace path (inherits Gate RTAS with 'replace.enabled' table property linkedin/openhouse#640'sreplace.enabledgate), and recovers server-side update-validation (lock / reserved-property / partition- and schema-evolution) on the plain-UPDATE branch that otherwise bypasses the service. InversePartitionSpecMapperfor stock-client CREATE.CREATE TABLE/CTAS/RTASand format-version=3 deletion vectors over the REST lane.org.apache.iceberg:*:1.11.0-openhouse(the LinkedIn custom patches ported onto Apache 1.11; source:mkuchenbecker/[email protected], incl. the F-REPL delete-file-replication fix).oh-hadoop-sparkrecipe wired for real-HDFS validation (Spark 3.5 / Hadoop 3.2 / Java-11 base); spark-4.0 itest classpath module + delta-harness Spark-4.0/Scala-2.13 wiring (Phase 2, landing on this branch).docs/spark4-iceberg-upgrade/design-of-record + findings.Testing Done
:services:tables:build -x test→ BUILD SUCCESSFUL on the consolidated branch (JDK 17).ParseExceptionset. Actual numbers will be posted when Phase 2 completes.deletion-vector-v1, reads reflect deletes, round-tripped through the server v3-write path); real-HDFS validation of write/read + metadata-direct-write on the ported fork.Additional Information
docs/spark4-iceberg-upgrade/.mkuchenbecker/[email protected]; wiring OpenHouse's build to that custom Spark distribution is a separate follow-up (the harness here runs stockspark-sql_2.13:4.0.0).🤖 Generated with Claude Code
https://claude.ai/code/session_01LLnVvLYnxjTuH2b5nAHa4A
Generated by Claude Code