feat(spark-3.5): [stacked] Add VACUUM SQL extension for Iceberg table maintenance by mkuchenbecker · Pull Request #661 · linkedin/openhouse

mkuchenbecker · 2026-07-24T19:08:42Z

OpenHouse spark-3.5 SQL Extensions Stack

PR	Content
#660	Standalone refactor (decouple spark-3.5 from spark-3.1)
(this)	VACUUM

Summary

Adds a VACUUM command to Spark 3.5 Openhouse SQL Extensions.

VACUUM <table> [REMOVE ORPHAN FILES] [RETAIN n HOURS]

Behaviour:
All files beyond the specified retention that are no longer referenced in the current version are deleted.

Snapshot expiration always runs. Snapshot expiration will delete files with the command.
REMOVE ORPHAN FILES opts into orphan-file deletion,
run before expiration. This is because snapshot expiration can't run on a table out of quota (writes a snapshot), but orphaned file deletion can run. Orphaned file deletion uses a recursive list operation and is expensive. The command may OOM if the Driver is not large enough. We should consider Inventory Lists as an optional input in the future so the operation can be run as a SQL query distributed on the executors.
RETAIN n HOURS bounds both operations via the procedures' older_than argument (resolved to a literal
timestamp in the session time zone, since procedure arguments must be foldable); when omitted, each procedure falls back to its own default retention.

Thin sugar over the Iceberg stored-procedure CALL path: VacuumTableExec validates the
target is an OpenHouse table (openhouse.tableId) and issues
CALL <catalog>.system.expire_snapshots / remove_orphan_files, so procedure resolution
and argument binding reuse the existing CALL path.

Changes

New non-reserved keywords VACUUM / REMOVE / ORPHAN / FILES / RETAIN / HOURS;
existing identifiers with those names still parse.

Testing Done

Added new tests for the changes made.

VacuumStatementTest (real Iceberg, Hadoop catalog), 6 tests: RETAIN 0 HOURS collapses
three snapshots to one with rows intact; REMOVE ORPHAN FILES RETAIN 24 HOURS preserves
live data; default retention; lower-case; non-OpenHouse-table rejection; invalid-syntax
parse error.

./gradlew :integrations:spark:spark-3.5:openhouse-spark-3.5-itest:statementTest --tests '*VacuumStatementTest'
# BUILD SUCCESSFUL — 6/6 pass

Additional Information

Large PR broken into smaller PRs, and PR plan linked in the description.

Adds a VACUUM command to the OpenHouse spark-3.5 SQL extensions: VACUUM <table> [REMOVE ORPHAN FILES] [RETAIN n HOURS] Snapshot expiration always runs; REMOVE ORPHAN FILES opts into orphan-file deletion, run after expiration so it cleans against the settled live-file set. RETAIN n HOURS bounds both operations via the procedures' older_than argument (resolved to a literal timestamp in the session time zone, since procedure arguments must be foldable); when omitted, each procedure applies its own default retention. Implemented as thin sugar over the Iceberg stored-procedure CALL path: VacuumTableExec validates the target is an OpenHouse table (openhouse.tableId property) and issues CALL <catalog>.system.expire_snapshots / remove_orphan_files via sparkSession.sql(...), so procedure resolution and argument binding reuse the existing CALL path. New non-reserved keywords VACUUM/REMOVE/ORPHAN/FILES/RETAIN/ HOURS; existing identifiers with those names still parse. Tested by VacuumStatementTest (real Iceberg, Hadoop catalog): RETAIN 0 HOURS collapses three snapshots to one with rows intact, REMOVE ORPHAN FILES RETAIN 24 HOURS preserves live data, default retention, lower-case, non-OpenHouse-table rejection, and invalid-syntax parse errors. Co-authored-by: Copilot <[email protected]>

Snapshot expiration commits table metadata and therefore cannot run on a table that is out of quota, whereas orphan-file deletion only removes unreferenced files from storage and always can. Running orphan removal first ensures it still executes in that case, and scanning against the pre-expiration referenced-file set means it can never delete a file a still-live snapshot references. Co-authored-by: Copilot <[email protected]>

VACUUM is an Alpha feature and is now opt-in per table. A table must set 'openhouse.vacuum.enabled' = 'true'; otherwise VACUUM throws UnsupportedOperationException explaining how to enable it. - VacuumTableExec: add the openhouse.vacuum.enabled gate + ENABLED_PROP constant. - VacuumStatementTest: enable the property on the vacuumed table and add testVacuumNotEnabledThrows for an OpenHouse table that has not opted in. - Add public documentation (docs/VACUUM.md): behavior, syntax, the Alpha opt-in knob, merge-on-read handling, examples, and caveats. Co-authored-by: Copilot <[email protected]>

Co-authored-by: Mike Kuchenbecker <[email protected]>

When RETAIN is omitted, VACUUM previously fell back to the Iceberg procedure defaults (5-day snapshot expiration, 3-day orphan-file deletion). Wire the cutoffs to the correct OpenHouse table properties instead, with no change to what the command does (it still cleans reclaimed files via the CALL procedures' default behavior): - Snapshot expiration honors the history policy in the 'policies' property: older_than = now - maxAge x granularity, plus retain_last => versions when set (defaults maxAge=3, granularity=DAY, versions=0, matching OpenHouse). - Orphan-file deletion honors 'ofd.one_day_ttl.enabled' (1 day when set, else the 3-day default). An explicit RETAIN n HOURS still overrides both. Pure helpers (ofdRetainDays, parseHistoryRetention, granularityToChrono) are unit-tested in VacuumTableExecTest; VacuumStatementTest.testVacuumHonorsHistoryPolicyVersions proves end-to-end that a history policy of versions=2 keeps exactly 2 snapshots. Co-authored-by: Copilot <[email protected]>

mkuchenbecker mentioned this pull request Jul 24, 2026

feat(spark-3.5): [stacked] Add ANALYZE TABLE ... COMPUTE CLUSTERING QUALITY SQL extension #663

Draft

11 tasks