Align DuckLake catalog preflight with the 1.5.x engine ("1.0" format) and add migration-coverage groundwork#843
Open
sagarswamirao wants to merge 6 commits into
Open
Align DuckLake catalog preflight with the 1.5.x engine ("1.0" format) and add migration-coverage groundwork#843sagarswamirao wants to merge 6 commits into
sagarswamirao wants to merge 6 commits into
Conversation
…ests. The internal list now reflects the catalog format changes, and tests have been modified to accept the "1.0" format while ensuring unsupported versions are correctly rejected. Signed-off-by: Sagar Swami Rao Kulkarni <[email protected]>
…hance related tests. The internal list has been updated to reflect the current catalog format, ensuring that unsupported versions are correctly rejected. New integration tests have been added to validate the behavior of the DuckLake extension with the updated versioning logic. Signed-off-by: Sagar Swami Rao Kulkarni <[email protected]>
…port. Introduced a new test file to characterize the behavior of AUTOMATIC_MIGRATION, ensuring proper handling of data paths and version upgrades. Added necessary catalog and data files to facilitate testing. This enhances the integration test coverage for DuckLake's migration functionality. Signed-off-by: Sagar Swami Rao Kulkarni <[email protected]>
…og fixtures for migration tests. The new script regenerates DuckLake catalog fixtures for versions 0.3, 0.4, and 1.0, ensuring compatibility with the migration tests. This update enhances the integration test coverage for DuckLake's migration functionality by providing necessary data files for each catalog format. Additionally, the migration test suite has been updated to reflect these changes, ensuring accurate validation of data integrity during migrations. Signed-off-by: Sagar Swami Rao Kulkarni <[email protected]>
…talog operations. Updated test logic to prevent sharing violations on Windows by reusing the same connection for attaching and detaching catalogs. Enhanced comments for clarity on connection handling and catalog version retrieval. This improves test reliability and aligns with DuckDB's connection management requirements. Signed-off-by: Sagar Swami Rao Kulkarni <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this addresses
The bundled DuckDB engine moved to the 1.5.x line, which produces and reads the DuckLake "1.0" catalog format. The catalog preflight in
ducklake_version.tsstill only accepted the older 0.3 line, so it rejected the very catalogs the engine generates -- returning a non-retryable 422 ("catalog version 1.0 is newer than this Publisher's extension supports") and blocking DuckLake connection attach and the materialization store. Issue #832.This PR realigns the preflight with what the engine actually does and builds the test infrastructure to keep them aligned as the engine version moves.
Changes
SUPPORTED_CATALOG_VERSIONSis set to the format the baked engine attaches as-is:["1.0"]. The comment now references the actual@duckdb/node-apipin instead of the stale 1.4.4 note. Older formats (0.3, 0.4) are intentionally not attach-as-is; they require a migration step (see below).ducklake_version.spec.ts).ducklake_catalog_version.test.ts): drives an actual@malloydata/db-duckdbconnection against local-file catalogs and pinsSUPPORTED_CATALOG_VERSIONSto observable engine behavior, so the constant cannot silently drift from what the engine does on ATTACH again.ducklake_migration.test.ts) plus committed fixtures, one per historical catalog format:0.3(written by DuckDB 1.4.x),0.4(1.5.0-1.5.1),1.0(1.5.2+).AUTOMATIC_MIGRATION, and asserts the data survives the forward migration to the current format. It also asserts a fixture exists for the format the current engine writes, so a future engine bump that introduces a new format fails the suite until a fixture is added.generate-ducklake-fixtures.sh): regenerates the per-format fixtures with the matching old DuckDB CLI; documents the version-to-format mapping.upgrade-malloy.sh: detects when a Malloy bump moves the DuckDB engine version and prints actionable guidance to regenerate and commit the fixtures, so DuckLake migration coverage stays current on every engine upgrade.Why the migration coverage matters
This is the groundwork for an opt-in auto-migration parameter we plan to expose to users, so they can migrate their existing DuckLake connections -- and the catalogs behind DuckLake materializations -- forward to the current format on attach. Before exposing that, we need confidence that forward migration is safe and lossless across every historical format. The migration matrix establishes and continuously verifies exactly that, and the tests document the migration footguns the future parameter must handle:
OVERRIDE_DATA_PATHis set. Publisher computes absolute data paths while catalogs can record relative ones, so this is a live hazard for the future auto-migration path.OVERRIDE_DATA_PATHonly suppresses the mismatch error; it does not persist the passed path into the catalog.Scope / not in this PR
The user-facing auto-migration parameter itself is not implemented here.
SUPPORTED_CATALOG_VERSIONScontinues to gate plain attach-as-is (currently1.0only); 0.3/0.4 catalogs are rejected by the preflight and would be handled by the future migration flow this PR lays the groundwork for.