Skip to content

Test/spark geospatial - #35

Merged
Neuw84 merged 3 commits into
mainfrom
test/spark-geospatial
Aug 3, 2026
Merged

Test/spark geospatial#35
Neuw84 merged 3 commits into
mainfrom
test/spark-geospatial

Conversation

@Neuw84

@Neuw84 Neuw84 commented Aug 3, 2026

Copy link
Copy Markdown
Owner

No description provided.

Angel Conde added 3 commits July 31, 2026 09:21
Angel Conde
feat(flink): Restore report parity on the merged Flink suite
b2a0664 
PR #32 replaced the Flink suite with a stronger, Docker-driven one, but its
report dropped the fields the EMR, Glue, Dataproc and Spark reports carry, so a
Flink run could no longer be read the same way as the others. In particular it
had no notion of matrix coverage, which is what stops a feature added to
features.json from sitting untested indefinitely.

Add platform, platform_label, catalog_mode, versions_tested and coverage to the
JSON report plus summary.uncovered_features, render the matching header lines,
the uncovered-features row and the coverage section in the markdown, and let
uncovered features fail the run as they already do for Spark.

Also make the comparison target configurable through MATRIX_PLATFORM_ID,
MATRIX_DATA_PATH, PLATFORM_LABEL and MATRIX_CATALOG_MODE, so the suite is no
longer pinned to the OSS Flink cells.

Nothing about how tests execute changed: the docker/local modes, the 35 tests,
their per-version runs and every result they produce are untouched. The two
things the merged suite added that the older report lacked, the unverified
count and the execution mode, are kept and now sit alongside the coverage row.

Verified by building one synthetic result per test and version, then asserting
the report fields and every markdown fragment, including re-rendering with a
feature dropped to prove the uncovered section and the failing exit code work.
Coverage reads 35/35 with no ids outside the matrix.
Angel Conde
feat(aws): Run the feature suite on Redshift Serverless, both modes
8348a28 
Adds Redshift to the managed-engine drivers. A low-RPU Serverless workgroup
joins the existing stack, and the suite is driven through the Redshift Data
API rather than shipped to a cluster, so it runs in the runner and needs no
bundle or entry point.

Two things about Redshift shaped the design.

Writes need an external schema that names the IAM role. Creating an Iceberg
table through the auto-mounted awsdatacatalog fails with "No session
credential found": that path authorises data access with the caller's IAM
session, and a Data API connection authenticated as a database user has none.

S3 Tables needs a Glue resource link. A table bucket is a federated Glue
catalog and Redshift cannot name one directly; putting the federated path in
CATALOG_ID silently resolves against the default catalog instead, so the
schema is created and every SELECT then raises EntityNotFoundException. With
a resource link in the default catalog, and CATALOG_ID set to the plain
account id, Redshift creates, reads, updates and deletes Iceberg tables in
S3 Tables.

Some features cannot be tested against a table Redshift is able to create:
it refuses format-version 3 outright. Spark on EMR builds those fixtures
instead and the suite reads and writes them, which separates "cannot write"
from "cannot read". Redshift turns out to read v3 tables and apply deletion
vectors correctly while refusing every v3 write, so those cells are partial
rather than absent. A new "partial" result records that, and compute_match
requires partial to meet partial exactly, so the level is falsifiable
instead of matching anything.

Also fixes two leaks found on the way. DROP TABLE removes only the Glue
entry in the s3buckets mode, leaving the data behind, so the suite deletes
its own S3 prefix; and the shared teardown never swept the fixture
warehouse, which holds real Parquet.

Verified against Redshift 1.0.365190 on 8 RPU, both modes, with the
fixtures in place and no leftover billable resources afterwards.
Angel Conde
fix(data): Correct the Redshift cells from the Serverless measurements
352c02e 
Fourteen cells disagreed with what the engine actually does. Both modes
claimed copy-on-write and time travel; neither exists. Redshift is
merge-on-read only and refuses the write.delete.mode property outright, and
no time-travel syntax is accepted at all, with no snapshot metadata table to
fall back on.

The S3 Tables file was the staler of the two and had the catalog story
backwards. It said S3 Tables uses its own built-in catalog rather than Glue,
and that Redshift consumes the REST API. It is the other way round: Glue is
the only way Redshift reaches S3 Tables, through a resource link, and it has
no Iceberg REST client whatsoever.

Three cells become partial because the feature is genuinely half-present.
Redshift reads format-version 3 tables and applies deletion vectors
correctly, but refuses every v3 write; and on S3 Tables, PARTITIONED BY at
CREATE is accepted and then silently discarded while ALTER ADD PARTITION
FIELD does apply, so transform partitioning takes two statements instead of
one.

Each entry now carries the engine's own error text in its caveats, so a
future reader can tell why a cell says what it says without rerunning
anything.

equality-deletes is deliberately left alone. Its notes claim Redshift reads
equality deletes, and all that could be established is that Redshift never
writes them, which does not contradict a read claim. No fixture carrying
equality deletes could be produced, because Spark writes position deletes,
so the cell stays unverified rather than being changed on a hunch.

Both storage modes now report zero discrepancies.
@Neuw84
Neuw84 merged commit 8c815ec into main Aug 3, 2026
6 checks passed
@github-actions

github-actions Bot commented Aug 3, 2026

Copy link
Copy Markdown

PyIceberg Feature Test Report

  • Timestamp: 2026-08-03T10:11:36.525417+00:00
  • PyIceberg Version: 0.11.1

Summary

Metric Count
Total 33
✅ Passed 18
❌ Failed 7
⏭️ Skipped 8
⚠️ Errors 0
🔍 Discrepancies 0

Test Results

Feature Version Result JSON Level Match Details
Table Creation v2 ✅ pass full ✅ Created and loaded table via SqlCatalog
Read Support v2 ✅ pass full ✅ Read 3 rows via scan().to_arrow()
Write (INSERT) v2 ✅ pass full ✅ Appended data twice, read 4 rows
Write (MERGE/UPDATE/DELETE) v2 ✅ pass partial ✅ Delete filter worked, 2 rows remaining
Position Deletes v2 ✅ pass partial ✅ Position delete via merge-on-read mode worked
Equality Deletes v2 ⏭️ skip partial ✅ PyIceberg reads equality deletes but cannot write them; producing an equality-de
Merge-on-Read v2 ✅ pass partial ✅ Merge-on-read delete mode works
Copy-on-Write v2 ✅ pass full ✅ Copy-on-write delete mode works (default)
Schema Evolution v2 ✅ pass full ✅ Add, rename, drop columns all work
Type Promotion v2 ✅ pass full ✅ Float -> Double type promotion works
Column Default Values v3 ✅ pass partial ✅ Parses and round-trips initial-default/write-default on schema fields; applying
Time Travel / Snapshots v2 ✅ pass full ✅ Time travel to snapshot 8809928823698055003 returned 1 row
Table Maintenance v2 ✅ pass partial ✅ Table maintenance operations available; 3 snapshots
Branching & Tagging v2 ✅ pass full ✅ Created branch 'dev_branch' and tag 'v1_tag'
Hidden Partitioning v2 ✅ pass full ✅ Created table with day + bucket hidden partitioning
Partition Evolution v2 ✅ pass full ✅ Evolved partition spec from day to add hour
Multi-Argument Transforms v3 ❌ fail none ✅ Multi-argument transforms are a V3 feature not yet supported in PyIceberg
Statistics v2 ✅ pass full ✅ Table statistics available via manifests
Bloom Filters v2 ❌ fail none ✅ PyIceberg does not support writing or reading bloom filter indexes
Catalog Integration v2 ✅ pass full ✅ SqlCatalog works; 15 tables in default namespace
Hadoop Catalog v2 ❌ fail none ✅ PyIceberg does not support Hadoop catalog
JDBC Catalog v2 ❌ fail none ✅ PyIceberg does not support JDBC catalog (Python, not JVM)
REST Catalog v2 ⏭️ skip full ✅ Requires running REST catalog server
Hive Metastore v2 ⏭️ skip full ✅ Requires running Hive Metastore service; cannot test in CI without Docker
AWS Glue Catalog v2 ⏭️ skip full ✅ Requires AWS credentials and Glue service
Nessie v2 ⏭️ skip partial ✅ Requires running Nessie server
Polaris v2 ⏭️ skip full ✅ Requires running Polaris server
Unity Catalog v2 ⏭️ skip partial ✅ Requires running Unity Catalog server
Variant Type v3 ❌ fail none ✅ PyIceberg does not yet support the Variant type
Shredded Variant v3 ❌ fail none ✅ PyIceberg does not yet support shredded variant
Geometry / Geo Types v3 ❌ fail none ✅ PyIceberg does not yet support geometry types
Nanosecond Timestamps v3 ⏭️ skip full ✅ PyIceberg cannot write V3 tables yet (Writing V3 is not yet supported, see: http
Lineage Tracking v3 ✅ pass partial ✅ Reads V3 row lineage metadata (next-row-id, snapshot first-row-id); assigning ro

@github-actions

github-actions Bot commented Aug 3, 2026

Copy link
Copy Markdown

DuckDB Iceberg Feature Test Report

Summary

Metric Count
Total 33
✅ Passed 18
❌ Failed 8
⏭️ Skipped 7
⚠️ Errors 0
🔍 Discrepancies 0

Test Results

Feature Version Result JSON Level Match Details
Table Creation v2 ✅ pass full ✅ CREATE TABLE, CREATE TABLE AS SELECT and DROP TABLE via REST catalog
Read Support v2 ✅ pass full ✅ Round-trip read of an Iceberg table via the REST catalog (3 rows)
Write (INSERT) v2 ✅ pass full ✅ INSERT INTO ... VALUES and INSERT INTO ... SELECT committed 3 rows
Write (MERGE/UPDATE/DELETE) v2 ✅ pass full ✅ UPDATE, DELETE and MERGE INTO (upsert) all committed correctly
Position Deletes v2 ✅ pass full ✅ DELETE on a V2 table wrote a positional-delete Parquet file (merge-on-read)
Equality Deletes v2 ⏭️ skip full ✅ DuckDB reads equality deletes but cannot write them; producing an equality-delet
Merge-on-Read v2 ✅ pass full ✅ UPDATE/DELETE use merge-on-read: delete files written, live rows reconciled on r
Copy-on-Write v2 ✅ pass partial ✅ INSERT uses copy-on-write semantics (append-only, no delete files); UPDATE/DELET
Schema Evolution v2 ✅ pass full ✅ ALTER TABLE ADD / RENAME / DROP COLUMN supported via REST catalog
Type Promotion / Widening v2 ⏭️ skip none ✅ ALTER COLUMN type promotion is not a documented DuckDB-Iceberg operation; not ex
Column Default Values v3 ✅ pass full ✅ V3 schema-level column DEFAULT values applied on CREATE and ALTER ADD COLUMN
Time Travel / Snapshots v2 ✅ pass full ✅ Time travel via AT (VERSION => snapshot_id) returns the historical row count
Table Maintenance v2 ❌ fail none ✅ DuckDB Iceberg does not provide maintenance ops (compaction, expire snapshots)
Branching & Tagging v2 ❌ fail none ✅ DuckDB Iceberg does not support branching or tagging
Hidden Partitioning v2 ✅ pass full ✅ Created and inserted into a table partitioned by bucket()/truncate() transforms
Partition Evolution v2 ✅ pass full ✅ Evolved the partition spec with ALTER TABLE ... SET PARTITIONED BY and kept read
Multi-Argument Transforms v3 ⏭️ skip unknown ✅ V3 multi-argument transforms are undocumented for DuckDB; not exercised
Statistics (Column Metrics) v2 ✅ pass full ✅ iceberg_metadata exposes per-file record_count statistics written by DuckDB
Bloom Filters v2 ❌ fail none ✅ DuckDB Iceberg does not read or write Iceberg bloom filters
Catalog Integration v2 ✅ pass full ✅ Attached an Iceberg REST catalog and performed namespace/table operations
Hadoop Catalog v2 ❌ fail none ✅ DuckDB Iceberg supports only REST-based catalogs (Hadoop catalog unsupported)
JDBC Catalog v2 ❌ fail none ✅ DuckDB Iceberg supports only REST-based catalogs (JDBC catalog unsupported)
REST Catalog v2 ✅ pass full ✅ Full read/write round-trip against an Iceberg REST catalog (OAuth2/none auth)
Hive Metastore v2 ❌ fail none ✅ DuckDB Iceberg supports only REST-based catalogs (Hive Metastore unsupported)
AWS Glue Catalog v2 ⏭️ skip full ✅ AWS Glue (SageMaker Lakehouse) catalog requires AWS credentials; not exercised l
Nessie v2 ❌ fail none ✅ DuckDB Iceberg does not natively support the Nessie catalog
Polaris v2 ⏭️ skip full ✅ Polaris uses the Iceberg REST protocol; a Polaris-specific server is not run her
Unity Catalog v2 ⏭️ skip partial ✅ Unity Catalog REST connectivity is undocumented for DuckDB; requires a Unity ser
Variant Type v3 ✅ pass full ✅ Created a V3 table with a VARIANT column and round-tripped a value
Shredded Variant v3 ❌ fail none ✅ DuckDB does not support shredded variant encoding (V3-only feature)
Geometry / Geo Types v3 ⏭️ skip partial ✅ REST catalog rejected the table schema, so DuckDB's support could not be measure
Nanosecond Timestamps v3 ✅ pass full ✅ Created a V3 table with a TIMESTAMP_NS column and inserted a nanosecond value
Lineage Tracking v3 ✅ pass full ✅ V3 write path with row lineage; row-level UPDATE encoded as a binary deletion ve

@github-actions

github-actions Bot commented Aug 3, 2026

Copy link
Copy Markdown

ClickHouse Iceberg Feature Test Report

  • Timestamp: 2026-08-03T10:12:45.990585+00:00
  • ClickHouse Version: 26.7.1.1315

Summary

Metric Count
Total 33
✅ Passed 1
❌ Failed 21
⏭️ Skipped 0
⚠️ Errors 10
🔍 Discrepancies 7

Test Results

Feature Version Result JSON Level Match Details
Table Creation v2 ❌ fail none ✅ ClickHouse cannot create Iceberg tables; icebergLocal() is read-only
Read Support v2 ⚠️ error full ✅ Traceback (most recent call last): File "/home/runner/work/iceberg-matrix/iceb
Write (INSERT) v2 ⚠️ error none ✅ Traceback (most recent call last): File "/home/runner/work/iceberg-matrix/iceb
Write (MERGE/UPDATE/DELETE) v2 ❌ fail none ✅ ClickHouse Iceberg support is read-only; no UPDATE, DELETE, or MERGE support
Position Deletes v2 ⚠️ error full ✅ Traceback (most recent call last): File "/home/runner/work/iceberg-matrix/iceb
Equality Deletes v2 ⚠️ error partial ✅ Traceback (most recent call last): File "/home/runner/work/iceberg-matrix/iceb
Merge-on-Read v2 ⚠️ error full ✅ Traceback (most recent call last): File "/home/runner/work/iceberg-matrix/iceb
Copy-on-Write v2 ❌ fail none ✅ ClickHouse is read-only for Iceberg; Copy-on-Write write mode not applicable
Schema Evolution v2 ⚠️ error partial ✅ Traceback (most recent call last): File "/home/runner/work/iceberg-matrix/iceb
Type Promotion / Widening v2 ⚠️ error partial ✅ Traceback (most recent call last): File "/home/runner/work/iceberg-matrix/iceb
Column Default Values v3 ❌ fail none ✅ Column default values not supported in ClickHouse Iceberg
Time Travel / Snapshots v2 ⚠️ error full ✅ Traceback (most recent call last): File "/home/runner/work/iceberg-matrix/iceb
Table Maintenance v2 ❌ fail none ✅ ClickHouse does not support Iceberg table maintenance (compaction, snapshot expi
Branching & Tagging v2 ❌ fail none ✅ ClickHouse does not support Iceberg branching or tagging
Hidden Partitioning v2 ⚠️ error full ✅ Traceback (most recent call last): File "/home/runner/work/iceberg-matrix/iceb
Partition Evolution v2 ⚠️ error unknown ✅ Traceback (most recent call last): File "/home/runner/work/iceberg-matrix/iceb
Multi-Argument Transforms v3 ❌ fail none ✅ Multi-argument transforms not supported in ClickHouse Iceberg
Statistics (Column Metrics) v2 ✅ pass partial ✅ ClickHouse uses Iceberg column-level statistics from manifest files for scan pru
Bloom Filters v2 ❌ fail none ✅ ClickHouse does not support Iceberg bloom filter indexes
Catalog Integration v2 ❓ partial full ✅ ClickHouse supports icebergLocal() and IcebergS3/IcebergAzureBlobStorage table f
Hadoop Catalog v2 ❌ fail none ✅ ClickHouse does not support Hadoop catalog; uses direct path-based access
JDBC Catalog v2 ❌ fail none ✅ ClickHouse does not support JDBC catalog
REST Catalog v2 ❌ fail full ❌ DISCREPANCY ClickHouse does not support Iceberg REST catalog protocol
Hive Metastore v2 ❌ fail none ✅ ClickHouse does not support Hive Metastore catalog for Iceberg
AWS Glue Catalog v2 ❌ fail full ❌ DISCREPANCY ClickHouse does not support AWS Glue catalog for Iceberg
Nessie v2 ❌ fail none ✅ ClickHouse does not support Nessie catalog
Polaris v2 ❌ fail partial ❌ DISCREPANCY ClickHouse does not support Polaris catalog
Unity Catalog v2 ❌ fail partial ❌ DISCREPANCY ClickHouse does not support Unity Catalog
Variant Type v3 ❌ fail unknown ❌ DISCREPANCY Variant type not supported in ClickHouse Iceberg
Shredded Variant v3 ❌ fail unknown ❌ DISCREPANCY Shredded variant not supported in ClickHouse Iceberg
Geometry / Geo Types v3 ❌ fail unknown ❌ DISCREPANCY Geometry type not supported in ClickHouse Iceberg
Nanosecond Timestamps v3 ❌ fail none ✅ Nanosecond timestamps not supported in ClickHouse Iceberg
Lineage Tracking v2 ❌ fail none ✅ Lineage tracking not supported in ClickHouse Iceberg

⚠️ Discrepancies

  • REST Catalog (v2): test=fail, json=full — ClickHouse does not support Iceberg REST catalog protocol
  • AWS Glue Catalog (v2): test=fail, json=full — ClickHouse does not support AWS Glue catalog for Iceberg
  • Polaris (v2): test=fail, json=partial — ClickHouse does not support Polaris catalog
  • Unity Catalog (v2): test=fail, json=partial — ClickHouse does not support Unity Catalog
  • Variant Type (v3): test=fail, json=unknown — Variant type not supported in ClickHouse Iceberg
  • Shredded Variant (v3): test=fail, json=unknown — Shredded variant not supported in ClickHouse Iceberg
  • Geometry / Geo Types (v3): test=fail, json=unknown — Geometry type not supported in ClickHouse Iceberg

@github-actions

github-actions Bot commented Aug 3, 2026

Copy link
Copy Markdown

Iceberg Feature Test Report (Spark, V2 + V3)

  • Timestamp: 2026-08-03T10:13:18.110251Z
  • Spark Version: 4.1.2
  • Iceberg Version: 1.11.0
  • Catalog: REST (http://127.0.0.1:8181/catalog)
  • Format Versions Tested: v2, v3

Summary

Metric Count
Total 70
✅ Passed 44
❌ Failed 4
⏭️ Skipped 22
⚠️ Errors 0
🔍 Discrepancies 0
🧭 Uncovered matrix features 0

Matrix coverage: 35/35 features in features.json have a test.

Test Results

Feature Version Result JSON Level Match Details
Table Creation v2 ✅ pass full ✅ Created Iceberg V2 table with multiple column types
Table Creation v3 ✅ pass full ✅ Created Iceberg V3 table with multiple column types
Read Support v2 ✅ pass full ✅ SELECT with predicate pushdown and column projection works
Read Support v3 ✅ pass full ✅ SELECT with predicate pushdown and column projection works
Write (INSERT) v2 ✅ pass full ✅ INSERT INTO works correctly with multiple batches
Write (INSERT) v3 ✅ pass full ✅ INSERT INTO works correctly with multiple batches
Write (MERGE/UPDATE/DELETE) v2 ✅ pass full ✅ UPDATE, DELETE, and MERGE INTO all work correctly
Write (MERGE/UPDATE/DELETE) v3 ✅ pass full ✅ UPDATE, DELETE, and MERGE INTO all work correctly
Position Deletes v2 ✅ pass full ✅ Position delete files / DVs created and applied correctly in MoR mode
Position Deletes v3 ✅ pass full ✅ Position delete files / DVs created and applied correctly in MoR mode
Equality Deletes v2 ✅ pass full ✅ Equality deletes readable; Spark SQL DELETE in MoR works correctly
Equality Deletes v3 ✅ pass full ✅ Equality deletes readable; Spark SQL DELETE in MoR works correctly
Merge-on-Read v2 ✅ pass full ✅ Merge-on-read produces delete files/DVs on UPDATE; reads merge correctly
Merge-on-Read v3 ✅ pass full ✅ Merge-on-read produces delete files/DVs on UPDATE; reads merge correctly
Copy-on-Write v2 ✅ pass full ✅ Copy-on-write rewrites data files (no delete files) on UPDATE
Copy-on-Write v3 ✅ pass full ✅ Copy-on-write rewrites data files (no delete files) on UPDATE
Schema Evolution v2 ✅ pass full ✅ ADD COLUMNS, RENAME COLUMN, DROP COLUMN all work correctly
Schema Evolution v3 ✅ pass full ✅ ADD COLUMNS, RENAME COLUMN, DROP COLUMN all work correctly
Type Promotion / Widening v2 ✅ pass full ✅ INT→BIGINT and FLOAT→DOUBLE promotions work correctly
Type Promotion / Widening v3 ✅ pass full ✅ INT→BIGINT and FLOAT→DOUBLE promotions work correctly
Column Default Values v2 ⏭️ skip none ✅ V3-only feature; not applicable to format-version 2 tables
Column Default Values v3 ❌ fail none ✅ Column default values not supported: [UNSUPPORTED_FEATURE.TABLE_OPERATION] The f
Time Travel / Snapshots v2 ✅ pass full ✅ VERSION AS OF time travel works correctly with snapshot IDs
Time Travel / Snapshots v3 ✅ pass full ✅ VERSION AS OF time travel works correctly with snapshot IDs
Table Maintenance v2 ✅ pass full ✅ rewrite_data_files and expire_snapshots procedures work correctly
Table Maintenance v3 ✅ pass full ✅ rewrite_data_files and expire_snapshots procedures work correctly
Branching & Tagging v2 ✅ pass full ✅ CREATE BRANCH, CREATE TAG, write to branch, and read from tag all work
Branching & Tagging v3 ✅ pass full ✅ CREATE BRANCH, CREATE TAG, write to branch, and read from tag all work
Hidden Partitioning v2 ✅ pass full ✅ Hidden partitioning with year(), bucket(), truncate() transforms works
Hidden Partitioning v3 ✅ pass full ✅ Hidden partitioning with year(), bucket(), truncate() transforms works
Partition Evolution v2 ✅ pass full ✅ Partition evolution (ADD PARTITION FIELD) works without rewriting data
Partition Evolution v3 ✅ pass full ✅ Partition evolution (ADD PARTITION FIELD) works without rewriting data
Multi-Argument Transforms v2 ⏭️ skip none ✅ V3-only feature; not applicable to format-version 2 tables
Multi-Argument Transforms v3 ❌ fail none ✅ Multi-source-column transforms not supported: Cannot convert transform with more
Statistics (Column Metrics) v2 ✅ pass full ✅ Column statistics (record_count, value_counts, bounds) present in manifest files
Statistics (Column Metrics) v3 ✅ pass full ✅ Column statistics (record_count, value_counts, bounds) present in manifest files
Bloom Filters v2 ✅ pass full ✅ Bloom filter properties configured and data read correctly
Bloom Filters v3 ✅ pass full ✅ Bloom filter properties configured and data read correctly
Catalog Integration v2 ✅ pass full ✅ Catalog integration works (SHOW NAMESPACES, SHOW TABLES, CREATE/DROP)
Catalog Integration v3 ✅ pass full ✅ Catalog integration works (SHOW NAMESPACES, SHOW TABLES, CREATE/DROP)
Hadoop Catalog v2 ✅ pass full ✅ Hadoop catalog: create, write, read, drop all work on local filesystem
Hadoop Catalog v3 ✅ pass full ✅ Hadoop catalog: create, write, read, drop all work on local filesystem
JDBC Catalog v2 ⏭️ skip full ✅ JDBC catalog test skipped in CI (requires external JDBC database)
JDBC Catalog v3 ⏭️ skip full ✅ JDBC catalog test skipped in CI (requires external JDBC database)
REST Catalog v2 ✅ pass full ✅ REST catalog (http://127.0.0.1:8181/catalog): namespace/table CRUD, write, and r
REST Catalog v3 ✅ pass full ✅ REST catalog (http://127.0.0.1:8181/catalog): namespace/table CRUD, write, and r
Hive Metastore v2 ⏭️ skip full ✅ Hive Metastore test skipped in CI (requires running Hive Metastore service)
Hive Metastore v3 ⏭️ skip full ✅ Hive Metastore test skipped in CI (requires running Hive Metastore service)
AWS Glue Catalog v2 ⏭️ skip full ✅ AWS Glue test skipped in CI (requires AWS credentials and Glue service)
AWS Glue Catalog v3 ⏭️ skip full ✅ AWS Glue test skipped in CI (requires AWS credentials and Glue service)
Nessie v2 ⏭️ skip full ✅ Nessie test skipped in CI (requires running Nessie server)
Nessie v3 ⏭️ skip full ✅ Nessie test skipped in CI (requires running Nessie server)
Polaris v2 ⏭️ skip full ✅ Polaris test skipped in CI (requires running Polaris server)
Polaris v3 ⏭️ skip full ✅ Polaris test skipped in CI (requires running Polaris server)
Unity Catalog v2 ⏭️ skip full ✅ Unity Catalog test skipped in CI (requires Unity Catalog endpoint)
Unity Catalog v3 ⏭️ skip full ✅ Unity Catalog test skipped in CI (requires Unity Catalog endpoint)
Snowflake Horizon Catalog v2 ⏭️ skip full ✅ Snowflake Horizon Catalog test skipped in CI (requires Snowflake REST endpoint)
Snowflake Horizon Catalog v3 ⏭️ skip partial ✅ Snowflake Horizon Catalog test skipped in CI (requires Snowflake REST endpoint)
Variant Type v2 ⏭️ skip none ✅ V3-only feature; not applicable to format-version 2 tables
Variant Type v3 ✅ pass full ✅ VARIANT type column created and written on V3 table
Shredded Variant v2 ⏭️ skip none ✅ V3-only feature; not applicable to format-version 2 tables
Shredded Variant v3 ✅ pass full ✅ VARIANT column with shredding property created and written on V3 table
Geometry / Geo Types v2 ⏭️ skip none ✅ V3-only feature; not applicable to format-version 2 tables
Geometry / Geo Types v3 ❌ fail none ✅ Spark accepts the type but the Iceberg Spark connector rejects it: Not a support
Nanosecond Timestamps v2 ⏭️ skip none ✅ V3-only feature; not applicable to format-version 2 tables
Nanosecond Timestamps v3 ❌ fail none ✅ Nanosecond timestamps not supported: [UNSUPPORTED_DATATYPE] Unsupported data ty
Lineage Tracking v2 ⏭️ skip none ✅ V3-only feature; not applicable to format-version 2 tables
Lineage Tracking v3 ✅ pass full ✅ Row lineage metadata columns (_row_id, _last_updated_sequence_number) readable o
Deletion Vectors v2 ⏭️ skip none ✅ V3-only feature; not applicable to format-version 2 tables
Deletion Vectors v3 ✅ pass full ✅ V3 deletion vectors (Puffin) produced on DELETE

@github-actions

github-actions Bot commented Aug 3, 2026

Copy link
Copy Markdown

Flink Iceberg Feature Test Report

  • Timestamp: 2026-08-03T10:35:54.421439+00:00
  • Flink Version: 2.3.0
  • Iceberg Version: 1.11.0
  • Execution mode: docker
  • Catalog: REST (http://10.1.1.156:8181/catalog, warehouse=demo)
  • Platform: Apache Flink 2.3.0 (OSS) + Iceberg 1.11.0
  • Format Versions Tested: v2, v3

Summary

Metric Count
Total 70
Passed 36
Failed 6
Skipped 24
Errors 4
Discrepancies vs matrix 0
Unverified (skip/error) 28
Uncovered matrix features 0

Failed is a result, not a defect: it records that the engine does not support the feature through Flink SQL. A discrepancy means the observed behaviour disagrees with flink.json.

Matrix coverage: 35/35 features in features.json have a test.

Test Results

Feature Version Result Matrix Match Details
Table Creation v2 PASS full ok CREATE TABLE with 4 column types on a V2 table, then INSERT and read-back of every column
Read Support v2 PASS full ok Batch: read 3 rows with correct predicate filtering and projection. Streaming: a continuous read (streaming=true, monitor-interval) delivered the init
Write (INSERT) v2 PASS full ok Batch: INSERT INTO appended across 2 commits (3 rows); INSERT OVERWRITE replaced them (1 row). Streaming: an unbounded INSERT committed 4 append snaps
Write (MERGE/UPDATE/DELETE) v2 FAIL partial ok Neither DELETE nor UPDATE is supported in Flink SQL; upsert mode is the only row-level write path. DELETE: java.lang.UnsupportedOperationException: Ca
Position Deletes v2 PASS full ok Write path produced position deletes (content=1): ['1:PARQUET'], and the read merged them correctly (rows=['1:updated', '2:second']). Emitted for a ro
Equality Deletes v2 PASS full ok UPSERT replaced the row (rows=['1:updated', '2:second']) and wrote equality delete files (content=2): ['2:PARQUET', '2:PARQUET']
Merge-on-Read v2 PASS full ok Upsert produced delete files that the reader merged at scan time (deletes=['2:PARQUET', '2:PARQUET'], rows=['1:updated', '2:second']). Merge-on-read i
Copy-on-Write v2 PASS partial ok INSERT OVERWRITE rewrote the data files with no delete files, which is the copy-on-write path Flink SQL can reach. Note the scope: copy-on-write prope
Deletion Vectors v2 SKIP none ok V3-only feature; not applicable to format-version 2 tables
Schema Evolution v2 PASS full ok ADD, RENAME and DROP COLUMN all succeeded via Flink DDL; existing rows readable
Type Promotion / Widening v2 PASS full ok INT→BIGINT and FLOAT→DOUBLE widening applied; out-of-INT-range value stored and read back
Column Default Values v2 SKIP none ok V3-only feature; not applicable to format-version 2 tables
Time Travel / Snapshots v2 PASS full ok Current table has 2 rows; reading snapshot 703562442825395592 via the snapshot-id hint returned the 1 row present at that snapshot
Table Maintenance v2 FAIL partial ok Streaming job committed 55 snapshots but no rewrite commit appeared within 180s (operations=['append'])
Branching & Tagging v2 PASS partial ok Branch reads via the branch hint, tag reads via the tag hint, and tag-to-tag incremental scans (start-tag/end-tag) all work against refs created throu
Hidden Partitioning v2 PASS partial ok Flink cannot declare transform partitioning (PARTITIONED BY (days(ts)) is a parser error), but on a day(ts)-partitioned table created through the cata
Partition Evolution v2 PASS partial ok Flink cannot initiate partition evolution (no SQL syntax; ADD PARTITION FIELD is a parser error) but honours it fully once the catalog evolves the spe
Multi-Argument Transforms v2 SKIP none ok V3-only feature; not applicable to format-version 2 tables
Statistics (Column Metrics) v2 PASS full ok Data file manifest carries record_count=3 and per-column value_counts for 2 columns, plus null_value_counts ['2']
Bloom Filters & Puffin v2 SKIP unknown ok Not verifiable from SQL: the Parquet bloom-filter write property is accepted and point lookups return correct results, but no Flink SQL surface or Ice
Catalog Integration v2 PASS full ok Full create/write/read/drop round-trip through an Iceberg catalog
REST Catalog v2 PASS full ok catalog-type='rest' against a live Lakekeeper REST catalog: table created, written, read back and dropped
Hadoop Catalog v2 ERR full ok org.apache.flink.table.catalog.exceptions.CatalogException: A database with name [test_db] does not exist in the catalog: [test_catalog].�[0m Shutting
JDBC Catalog v2 ERR full ok java.io.IOException: Mkdirs failed to create file:/work/hadoop-warehouse/test_db/cat_5c385e5a/metadata (exists=false, cwd=file:/opt/flink)�[0m Shuttin
Hive Metastore v2 SKIP full ok Not exercised: requires a running Hive Metastore (thrift) service. Flink ships the catalog implementation, but this harness has no such endpoint to pr
AWS Glue Catalog v2 SKIP full ok Not exercised: requires AWS credentials and a Glue Data Catalog. Flink ships the catalog implementation, but this harness has no such endpoint to prov
Nessie v2 SKIP full ok Not exercised: requires a running Nessie server. Flink ships the catalog implementation, but this harness has no such endpoint to prove it against
Polaris v2 SKIP full ok Not exercised: requires a running Apache Polaris server (reachable via catalog-type='rest'). Flink ships the catalog implementation, but this harness
Unity Catalog v2 SKIP partial ok Not exercised: requires a Databricks Unity Catalog endpoint. Flink ships the catalog implementation, but this harness has no such endpoint to prove it
Snowflake Horizon Catalog v2 SKIP full ok Not exercised: requires a Snowflake account with Horizon Catalog enabled. Flink ships the catalog implementation, but this harness has no such endpoin
Variant Type v2 SKIP none ok V3-only feature; not applicable to format-version 2 tables
Shredded Variant v2 SKIP none ok V3-only feature; not applicable to format-version 2 tables
Geometry / Geo Types v2 SKIP none ok V3-only feature; not applicable to format-version 2 tables
Nanosecond Timestamps v2 SKIP none ok V3-only feature; not applicable to format-version 2 tables
Lineage Tracking v2 SKIP none ok V3-only feature; not applicable to format-version 2 tables
Table Creation v3 PASS full ok CREATE TABLE with 4 column types on a V3 table, then INSERT and read-back of every column
Read Support v3 PASS full ok Batch: read 3 rows with correct predicate filtering and projection. Streaming: a continuous read (streaming=true, monitor-interval) delivered the init
Write (INSERT) v3 PASS full ok Batch: INSERT INTO appended across 2 commits (3 rows); INSERT OVERWRITE replaced them (1 row). Streaming: an unbounded INSERT committed 4 append snaps
Write (MERGE/UPDATE/DELETE) v3 FAIL partial ok Neither DELETE nor UPDATE is supported in Flink SQL; upsert mode is the only row-level write path. DELETE: java.lang.UnsupportedOperationException: Ca
Position Deletes v3 PASS full ok Write path produced position deletes (content=1): ['1:PUFFIN'], and the read merged them correctly (rows=['1:updated', '2:second']). Emitted for a row
Equality Deletes v3 PASS full ok UPSERT replaced the row (rows=['1:updated', '2:second']) and wrote equality delete files (content=2): ['2:PARQUET', '2:PARQUET']
Merge-on-Read v3 PASS full ok Upsert produced delete files that the reader merged at scan time (deletes=['2:PARQUET', '2:PARQUET'], rows=['1:updated', '2:second']). Merge-on-read i
Copy-on-Write v3 PASS partial ok INSERT OVERWRITE rewrote the data files with no delete files, which is the copy-on-write path Flink SQL can reach. Note the scope: copy-on-write prope
Deletion Vectors v3 PASS full ok V3 deletion vectors written from plain Flink SQL: ['1:PUFFIN'] (content=1 position deletes in puffin), alongside equality deletes for the key-based pa
Schema Evolution v3 PASS full ok ADD, RENAME and DROP COLUMN all succeeded via Flink DDL; existing rows readable
Type Promotion / Widening v3 PASS full ok INT→BIGINT and FLOAT→DOUBLE widening applied; out-of-INT-range value stored and read back
Column Default Values v3 FAIL none ok Flink SQL cannot declare column defaults: org.apache.flink.sql.parser.impl.ParseException: Encountered "DEFAULT" at line 1, column 53. Was expecting o
Time Travel / Snapshots v3 PASS full ok Current table has 2 rows; reading snapshot 4474578814413764679 via the snapshot-id hint returned the 1 row present at that snapshot
Table Maintenance v3 FAIL partial ok Streaming job committed 56 snapshots but no rewrite commit appeared within 180s (operations=['append'])
Branching & Tagging v3 PASS partial ok Branch reads via the branch hint, tag reads via the tag hint, and tag-to-tag incremental scans (start-tag/end-tag) all work against refs created throu
Hidden Partitioning v3 PASS partial ok Flink cannot declare transform partitioning (PARTITIONED BY (days(ts)) is a parser error), but on a day(ts)-partitioned table created through the cata
Partition Evolution v3 PASS partial ok Flink cannot initiate partition evolution (no SQL syntax; ADD PARTITION FIELD is a parser error) but honours it fully once the catalog evolves the spe
Multi-Argument Transforms v3 SKIP unknown ok Not exercised: Flink DDL cannot express any transform partitioning at all (PARTITIONED BY only takes plain column names), so a multi-argument transfor
Statistics (Column Metrics) v3 PASS full ok Data file manifest carries record_count=3 and per-column value_counts for 2 columns, plus null_value_counts ['2']
Bloom Filters & Puffin v3 SKIP unknown ok Not verifiable from SQL: the Parquet bloom-filter write property is accepted and point lookups return correct results, but no Flink SQL surface or Ice
Catalog Integration v3 PASS full ok Full create/write/read/drop round-trip through an Iceberg catalog
REST Catalog v3 PASS full ok catalog-type='rest' against a live Lakekeeper REST catalog: table created, written, read back and dropped
Hadoop Catalog v3 ERR full ok org.apache.flink.table.catalog.exceptions.CatalogException: A database with name [test_db] does not exist in the catalog: [test_catalog].�[0m Shutting
JDBC Catalog v3 ERR full ok java.io.IOException: Mkdirs failed to create file:/work/hadoop-warehouse/test_db/cat_cb62e174/metadata (exists=false, cwd=file:/opt/flink)�[0m Shuttin
Hive Metastore v3 SKIP full ok Not exercised: requires a running Hive Metastore (thrift) service. Flink ships the catalog implementation, but this harness has no such endpoint to pr
AWS Glue Catalog v3 SKIP full ok Not exercised: requires AWS credentials and a Glue Data Catalog. Flink ships the catalog implementation, but this harness has no such endpoint to prov
Nessie v3 SKIP full ok Not exercised: requires a running Nessie server. Flink ships the catalog implementation, but this harness has no such endpoint to prove it against
Polaris v3 SKIP full ok Not exercised: requires a running Apache Polaris server (reachable via catalog-type='rest'). Flink ships the catalog implementation, but this harness
Unity Catalog v3 SKIP partial ok Not exercised: requires a Databricks Unity Catalog endpoint. Flink ships the catalog implementation, but this harness has no such endpoint to prove it
Snowflake Horizon Catalog v3 SKIP partial ok Not exercised: requires a Snowflake account with Horizon Catalog enabled. Flink ships the catalog implementation, but this harness has no such endpoin
Variant Type v3 PASS partial ok VARIANT column on a V3 table: value written with PARSE_JSON and read back non-null. Field extraction is not possible from Flink SQL -- Flink 2.3 expos
Shredded Variant v3 SKIP unknown ok Not verifiable from SQL: the shredding table property is accepted and data round-trips, but whether the writer actually shredded the variant is not ob
Geometry / Geo Types v3 FAIL none ok Blocked by Flink, not Iceberg: the Calcite-based planner keeps GEOMETRY behind its spatial extensions (enabled via the Calcite fun=spatial connect str
Nanosecond Timestamps v3 PASS full ok TIMESTAMP(9) maps to the Iceberg V3 timestamp_ns type (confirmed in the table schema) and a nanosecond-precision value round-trips exactly
Lineage Tracking v3 PASS partial ok Flink maintains V3 row lineage on write -- after inserting 3 rows the table metadata carries next-row-id=3 and the snapshot reports first-row-id=0, ad

Unverified

These could not be exercised here, so they neither confirm nor contradict the matrix:

  • Deletion Vectors (v2): matrix none — V3-only feature; not applicable to format-version 2 tables
  • Column Default Values (v2): matrix none — V3-only feature; not applicable to format-version 2 tables
  • Multi-Argument Transforms (v2): matrix none — V3-only feature; not applicable to format-version 2 tables
  • Bloom Filters & Puffin (v2): matrix unknown — Not verifiable from SQL: the Parquet bloom-filter write property is accepted and point lookups return correct results, but no Flink SQL surface or Iceberg metadata table reports whether a bloom filter
  • Hadoop Catalog (v2): matrix full — org.apache.flink.table.catalog.exceptions.CatalogException: A database with name [test_db] does not exist in the catalog: [test_catalog].�[0m Shutting down the session... done. WARNING: Unknown module
  • JDBC Catalog (v2): matrix full — java.io.IOException: Mkdirs failed to create file:/work/hadoop-warehouse/test_db/cat_5c385e5a/metadata (exists=false, cwd=file:/opt/flink)�[0m Shutting down the session... done. WARNING: Unknown modul
  • Hive Metastore (v2): matrix full — Not exercised: requires a running Hive Metastore (thrift) service. Flink ships the catalog implementation, but this harness has no such endpoint to prove it against
  • AWS Glue Catalog (v2): matrix full — Not exercised: requires AWS credentials and a Glue Data Catalog. Flink ships the catalog implementation, but this harness has no such endpoint to prove it against
  • Nessie (v2): matrix full — Not exercised: requires a running Nessie server. Flink ships the catalog implementation, but this harness has no such endpoint to prove it against
  • Polaris (v2): matrix full — Not exercised: requires a running Apache Polaris server (reachable via catalog-type='rest'). Flink ships the catalog implementation, but this harness has no such endpoint to prove it against
  • Unity Catalog (v2): matrix partial — Not exercised: requires a Databricks Unity Catalog endpoint. Flink ships the catalog implementation, but this harness has no such endpoint to prove it against
  • Snowflake Horizon Catalog (v2): matrix full — Not exercised: requires a Snowflake account with Horizon Catalog enabled. Flink ships the catalog implementation, but this harness has no such endpoint to prove it against
  • Variant Type (v2): matrix none — V3-only feature; not applicable to format-version 2 tables
  • Shredded Variant (v2): matrix none — V3-only feature; not applicable to format-version 2 tables
  • Geometry / Geo Types (v2): matrix none — V3-only feature; not applicable to format-version 2 tables
  • Nanosecond Timestamps (v2): matrix none — V3-only feature; not applicable to format-version 2 tables
  • Lineage Tracking (v2): matrix none — V3-only feature; not applicable to format-version 2 tables
  • Multi-Argument Transforms (v3): matrix unknown — Not exercised: Flink DDL cannot express any transform partitioning at all (PARTITIONED BY only takes plain column names), so a multi-argument transform cannot be declared from SQL
  • Bloom Filters & Puffin (v3): matrix unknown — Not verifiable from SQL: the Parquet bloom-filter write property is accepted and point lookups return correct results, but no Flink SQL surface or Iceberg metadata table reports whether a bloom filter
  • Hadoop Catalog (v3): matrix full — org.apache.flink.table.catalog.exceptions.CatalogException: A database with name [test_db] does not exist in the catalog: [test_catalog].�[0m Shutting down the session... done. WARNING: Unknown module
  • JDBC Catalog (v3): matrix full — java.io.IOException: Mkdirs failed to create file:/work/hadoop-warehouse/test_db/cat_cb62e174/metadata (exists=false, cwd=file:/opt/flink)�[0m Shutting down the session... done. WARNING: Unknown modul
  • Hive Metastore (v3): matrix full — Not exercised: requires a running Hive Metastore (thrift) service. Flink ships the catalog implementation, but this harness has no such endpoint to prove it against
  • AWS Glue Catalog (v3): matrix full — Not exercised: requires AWS credentials and a Glue Data Catalog. Flink ships the catalog implementation, but this harness has no such endpoint to prove it against
  • Nessie (v3): matrix full — Not exercised: requires a running Nessie server. Flink ships the catalog implementation, but this harness has no such endpoint to prove it against
  • Polaris (v3): matrix full — Not exercised: requires a running Apache Polaris server (reachable via catalog-type='rest'). Flink ships the catalog implementation, but this harness has no such endpoint to prove it against
  • Unity Catalog (v3): matrix partial — Not exercised: requires a Databricks Unity Catalog endpoint. Flink ships the catalog implementation, but this harness has no such endpoint to prove it against
  • Snowflake Horizon Catalog (v3): matrix partial — Not exercised: requires a Snowflake account with Horizon Catalog enabled. Flink ships the catalog implementation, but this harness has no such endpoint to prove it against
  • Shredded Variant (v3): matrix unknown — Not verifiable from SQL: the shredding table property is accepted and data round-trips, but whether the writer actually shredded the variant is not observable through any Flink SQL surface or metadata

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@Neuw84