Skip to content

Align DuckLake catalog preflight with the 1.5.x engine ("1.0" format) and add migration-coverage groundwork#843

Open
sagarswamirao wants to merge 6 commits into
mainfrom
sagark/duckdb-backward-compat
Open

Align DuckLake catalog preflight with the 1.5.x engine ("1.0" format) and add migration-coverage groundwork#843
sagarswamirao wants to merge 6 commits into
mainfrom
sagark/duckdb-backward-compat

Conversation

@sagarswamirao

@sagarswamirao sagarswamirao commented Jun 22, 2026

Copy link
Copy Markdown
Collaborator

What this addresses

The bundled DuckDB engine moved to the 1.5.x line, which produces and reads the DuckLake "1.0" catalog format. The catalog preflight in ducklake_version.ts still only accepted the older 0.3 line, so it rejected the very catalogs the engine generates -- returning a non-retryable 422 ("catalog version 1.0 is newer than this Publisher's extension supports") and blocking DuckLake connection attach and the materialization store. Issue #832.

This PR realigns the preflight with what the engine actually does and builds the test infrastructure to keep them aligned as the engine version moves.

Changes

  • SUPPORTED_CATALOG_VERSIONS is set to the format the baked engine attaches as-is: ["1.0"]. The comment now references the actual @duckdb/node-api pin instead of the stale 1.4.4 note. Older formats (0.3, 0.4) are intentionally not attach-as-is; they require a migration step (see below).
  • Unit spec updated to match (ducklake_version.spec.ts).
  • New real-engine contract test (ducklake_catalog_version.test.ts): drives an actual @malloydata/db-duckdb connection against local-file catalogs and pins SUPPORTED_CATALOG_VERSIONS to observable engine behavior, so the constant cannot silently drift from what the engine does on ATTACH again.
  • New migration matrix test (ducklake_migration.test.ts) plus committed fixtures, one per historical catalog format:
    • 0.3 (written by DuckDB 1.4.x), 0.4 (1.5.0-1.5.1), 1.0 (1.5.2+).
    • For each format it copies the fixture, attaches with AUTOMATIC_MIGRATION, and asserts the data survives the forward migration to the current format. It also asserts a fixture exists for the format the current engine writes, so a future engine bump that introduces a new format fails the suite until a fixture is added.
  • Fixture generator (generate-ducklake-fixtures.sh): regenerates the per-format fixtures with the matching old DuckDB CLI; documents the version-to-format mapping.
  • upgrade-malloy.sh: detects when a Malloy bump moves the DuckDB engine version and prints actionable guidance to regenerate and commit the fixtures, so DuckLake migration coverage stays current on every engine upgrade.

Why the migration coverage matters

This is the groundwork for an opt-in auto-migration parameter we plan to expose to users, so they can migrate their existing DuckLake connections -- and the catalogs behind DuckLake materializations -- forward to the current format on attach. Before exposing that, we need confidence that forward migration is safe and lossless across every historical format. The migration matrix establishes and continuously verifies exactly that, and the tests document the migration footguns the future parameter must handle:

  • Migration is one-way and in-place; it rewrites the user's catalog file, after which an older DuckDB engine can no longer open it.
  • The attach data path must match the path recorded in the catalog, or the attach is refused unless OVERRIDE_DATA_PATH is set. Publisher computes absolute data paths while catalogs can record relative ones, so this is a live hazard for the future auto-migration path.
  • OVERRIDE_DATA_PATH only suppresses the mismatch error; it does not persist the passed path into the catalog.

Scope / not in this PR

The user-facing auto-migration parameter itself is not implemented here. SUPPORTED_CATALOG_VERSIONS continues to gate plain attach-as-is (currently 1.0 only); 0.3/0.4 catalogs are rejected by the preflight and would be handled by the future migration flow this PR lays the groundwork for.

…ests. The internal list now reflects the catalog format changes, and tests have been modified to accept the "1.0" format while ensuring unsupported versions are correctly rejected.

Signed-off-by: Sagar Swami Rao Kulkarni <[email protected]>
…hance related tests. The internal list has been updated to reflect the current catalog format, ensuring that unsupported versions are correctly rejected. New integration tests have been added to validate the behavior of the DuckLake extension with the updated versioning logic.

Signed-off-by: Sagar Swami Rao Kulkarni <[email protected]>
…port. Introduced a new test file to characterize the behavior of AUTOMATIC_MIGRATION, ensuring proper handling of data paths and version upgrades. Added necessary catalog and data files to facilitate testing. This enhances the integration test coverage for DuckLake's migration functionality.

Signed-off-by: Sagar Swami Rao Kulkarni <[email protected]>
…og fixtures for migration tests. The new script regenerates DuckLake catalog fixtures for versions 0.3, 0.4, and 1.0, ensuring compatibility with the migration tests. This update enhances the integration test coverage for DuckLake's migration functionality by providing necessary data files for each catalog format. Additionally, the migration test suite has been updated to reflect these changes, ensuring accurate validation of data integrity during migrations.

Signed-off-by: Sagar Swami Rao Kulkarni <[email protected]>
…talog operations. Updated test logic to prevent sharing violations on Windows by reusing the same connection for attaching and detaching catalogs. Enhanced comments for clarity on connection handling and catalog version retrieval. This improves test reliability and aligns with DuckDB's connection management requirements.

Signed-off-by: Sagar Swami Rao Kulkarni <[email protected]>
@sagarswamirao sagarswamirao changed the title Update DuckLake version support to include "1.0" and adjust related t… Align DuckLake catalog preflight with the 1.5.x engine ("1.0" format) and add migration-coverage groundwork Jun 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant