Skip to content

Test/spark geospatial - #33

Merged
Neuw84 merged 3 commits into
mainfrom
test/spark-geospatial
Jul 30, 2026
Merged

Test/spark geospatial#33
Neuw84 merged 3 commits into
mainfrom
test/spark-geospatial

Conversation

@Neuw84

@Neuw84 Neuw84 commented Jul 30, 2026

Copy link
Copy Markdown
Owner

No description provided.

Angel Conde added 3 commits July 30, 2026 12:54
Angel Conde
refactor(aws): Share the driver plumbing between platform engines
3633333 
EMR Serverless and Glue differ only in how a job is created and run. Everything
around that -- the repo bundle, the S3 layout, the report download, the job
summary -- is identical, so it now lives in tests/aws/platform_common.py instead
of being copied. The report format is the part that matters most: the whole point
is that every engine publishes the same shape, and two copies would drift.

run_emr_serverless.py drops from 482 to about 300 lines with no behaviour change,
verified by re-running both storage modes afterwards: 44/4/22 and 42/4/24, zero
errors and zero discrepancies, the same as before.

build_bundle is now engine-agnostic, carrying every engine's matrix JSON rather
than just EMR's, so one bundle serves any driver and adding an engine does not
mean remembering to extend it.

emr_entrypoint.py becomes platform_entrypoint.py and takes --engine and
--platform-id, since the only engine-specific things in it were the platform id
and the matrix path. It now parses with parse_known_args because Glue injects
job arguments of its own (--JOB_NAME, --TempDir), and detect_iceberg_version
searches several roots so it finds the jar on Glue as well as EMR.
Angel Conde
feat(aws): Run the feature suite on AWS Glue, both storage modes
c9e53a7 
Adds tests/aws/run_glue.py and a glue-version input, so Glue is now a supported
engine on the dispatch workflow rather than one that fails fast. Both storage
modes pass with zero errors and zero discrepancies on Glue 5.1.

The catalog wiring is identical to EMR -- GlueCatalog for s3buckets, and the
federated s3tablescatalog through glue.id for s3tables -- which is the point:
what differs between the two engines is the runtime, not the configuration.

glue-version is a choice input rather than free text because it pins Iceberg
rather than just bumping Spark. 5.1 ships Iceberg 1.10.0 and supports format
version 3; 5.0 ships 1.7.1 and 4.0 ships 1.0.0, both of which predate most of V3.
Running V3 below 5.1 mostly measures the absence of V3, so the input says so.

Three Glue-specific behaviours cost a run each and are commented where they bite:

  An argument declared with an empty value reaches the script as a bare flag with
  no value, and argparse rejects it. Empty values are dropped instead, on both the
  job definition and the run.

  Glue treats any SystemExit as a failure, including SystemExit(0). A clean run
  was being reported FAILED with "SystemExit: 0", which hid the report behind it.
  The entrypoint now only raises SystemExit when there is something to report.

  MaxConcurrentRuns of 1 is too tight even though the modes run sequentially:
  Glue still counts a run as active for a moment after it reports SUCCEEDED, so
  starting the next mode failed with ConcurrentRunsExceededException.

Teardown grows Glue job handling. A job definition is free to keep but an active
run bills per DPU-hour, so runs are stopped before the definition is deleted, and
it is prefix-scoped so it also sweeps up anything a crashed run left behind.

The job role now trusts glue.amazonaws.com in addition to EMR Serverless, and can
write CloudWatch Logs, which Glue uses instead of S3 and without which a job does
not start. The Glue trust statement deliberately has no aws:SourceAccount
condition, because Glue does not reliably populate it for job execution roles and
a condition that never matches fails the job with a misleading AccessDenied; the
control is the CI role, which may only pass this role to those two services.
Angel Conde
fix(data): Correct the Glue cells from the Glue 5.1 measurements
b5d7fff 
Measured on Glue 5.1 (Spark 3.5.6-amzn-1, Iceberg 1.10.0-amzn-0). Both storage
modes now run with zero discrepancies.

VARIANT and shredded variant move to none. Glue 5.1 runs Spark 3.5.6, which has
no VARIANT SQL type -- it arrived in Spark 4 -- so declaring the column is
rejected with UNSUPPORTED_DATATYPE. Iceberg 1.10.0 supports variant at the format
level, so this is an engine limitation rather than an Iceberg one, and the
distinction is real rather than theoretical: the same test passes on EMR
Serverless, which runs Spark 4.0.2 with the same Iceberg version. The s3buckets
cell previously said full on the strength of a note that described EMR, not Glue.

On S3 Tables, AWS Glue Catalog moves from none to full for the same reason it did
on EMR: the old note, "S3 Tables uses its own built-in catalog, not AWS Glue", is
wrong about how Glue actually reaches it -- the table bucket is mounted as the
federated s3tablescatalog and addressed through GlueCatalog.

Row lineage on S3 Tables moves from unknown to full, and multi-arg transforms,
variant and shredded variant from unknown to none. These were unknown because AWS
does not document them for S3 Tables; they are measured now.

Also classifies a missing VARIANT type as a failure rather than an error in the
shredded-variant test, matching what the variant-type test already did. An engine
that cannot express the type is a measured "not supported", and reporting it as an
error made a known gap look like a broken harness.

Worth noting for later: Glue 5.1's release notes list column default values as
supported, but the Spark SQL DDL path still rejects them
(UNSUPPORTED_FEATURE.TABLE_OPERATION), exactly as on EMR. The cell stays none,
which is what both engines measure.
@github-actions

github-actions Bot commented Jul 30, 2026

Copy link
Copy Markdown

PyIceberg Feature Test Report

  • Timestamp: 2026-07-30T10:56:52.558746+00:00
  • PyIceberg Version: 0.11.1

Summary

Metric Count
Total 33
✅ Passed 18
❌ Failed 7
⏭️ Skipped 8
⚠️ Errors 0
🔍 Discrepancies 0

Test Results

Feature Version Result JSON Level Match Details
Table Creation v2 ✅ pass full ✅ Created and loaded table via SqlCatalog
Read Support v2 ✅ pass full ✅ Read 3 rows via scan().to_arrow()
Write (INSERT) v2 ✅ pass full ✅ Appended data twice, read 4 rows
Write (MERGE/UPDATE/DELETE) v2 ✅ pass partial ✅ Delete filter worked, 2 rows remaining
Position Deletes v2 ✅ pass partial ✅ Position delete via merge-on-read mode worked
Equality Deletes v2 ⏭️ skip partial ✅ PyIceberg reads equality deletes but cannot write them; producing an equality-de
Merge-on-Read v2 ✅ pass partial ✅ Merge-on-read delete mode works
Copy-on-Write v2 ✅ pass full ✅ Copy-on-write delete mode works (default)
Schema Evolution v2 ✅ pass full ✅ Add, rename, drop columns all work
Type Promotion v2 ✅ pass full ✅ Float -> Double type promotion works
Column Default Values v3 ✅ pass partial ✅ Parses and round-trips initial-default/write-default on schema fields; applying
Time Travel / Snapshots v2 ✅ pass full ✅ Time travel to snapshot 8489160643093991844 returned 1 row
Table Maintenance v2 ✅ pass partial ✅ Table maintenance operations available; 3 snapshots
Branching & Tagging v2 ✅ pass full ✅ Created branch 'dev_branch' and tag 'v1_tag'
Hidden Partitioning v2 ✅ pass full ✅ Created table with day + bucket hidden partitioning
Partition Evolution v2 ✅ pass full ✅ Evolved partition spec from day to add hour
Multi-Argument Transforms v3 ❌ fail none ✅ Multi-argument transforms are a V3 feature not yet supported in PyIceberg
Statistics v2 ✅ pass full ✅ Table statistics available via manifests
Bloom Filters v2 ❌ fail none ✅ PyIceberg does not support writing or reading bloom filter indexes
Catalog Integration v2 ✅ pass full ✅ SqlCatalog works; 15 tables in default namespace
Hadoop Catalog v2 ❌ fail none ✅ PyIceberg does not support Hadoop catalog
JDBC Catalog v2 ❌ fail none ✅ PyIceberg does not support JDBC catalog (Python, not JVM)
REST Catalog v2 ⏭️ skip full ✅ Requires running REST catalog server
Hive Metastore v2 ⏭️ skip full ✅ Requires running Hive Metastore service; cannot test in CI without Docker
AWS Glue Catalog v2 ⏭️ skip full ✅ Requires AWS credentials and Glue service
Nessie v2 ⏭️ skip partial ✅ Requires running Nessie server
Polaris v2 ⏭️ skip full ✅ Requires running Polaris server
Unity Catalog v2 ⏭️ skip partial ✅ Requires running Unity Catalog server
Variant Type v3 ❌ fail none ✅ PyIceberg does not yet support the Variant type
Shredded Variant v3 ❌ fail none ✅ PyIceberg does not yet support shredded variant
Geometry / Geo Types v3 ❌ fail none ✅ PyIceberg does not yet support geometry types
Nanosecond Timestamps v3 ⏭️ skip full ✅ PyIceberg cannot write V3 tables yet (Writing V3 is not yet supported, see: http
Lineage Tracking v3 ✅ pass partial ✅ Reads V3 row lineage metadata (next-row-id, snapshot first-row-id); assigning ro

@github-actions

github-actions Bot commented Jul 30, 2026

Copy link
Copy Markdown

DuckDB Iceberg Feature Test Report

Summary

Metric Count
Total 33
✅ Passed 18
❌ Failed 8
⏭️ Skipped 7
⚠️ Errors 0
🔍 Discrepancies 0

Test Results

Feature Version Result JSON Level Match Details
Table Creation v2 ✅ pass full ✅ CREATE TABLE, CREATE TABLE AS SELECT and DROP TABLE via REST catalog
Read Support v2 ✅ pass full ✅ Round-trip read of an Iceberg table via the REST catalog (3 rows)
Write (INSERT) v2 ✅ pass full ✅ INSERT INTO ... VALUES and INSERT INTO ... SELECT committed 3 rows
Write (MERGE/UPDATE/DELETE) v2 ✅ pass full ✅ UPDATE, DELETE and MERGE INTO (upsert) all committed correctly
Position Deletes v2 ✅ pass full ✅ DELETE on a V2 table wrote a positional-delete Parquet file (merge-on-read)
Equality Deletes v2 ⏭️ skip full ✅ DuckDB reads equality deletes but cannot write them; producing an equality-delet
Merge-on-Read v2 ✅ pass full ✅ UPDATE/DELETE use merge-on-read: delete files written, live rows reconciled on r
Copy-on-Write v2 ✅ pass partial ✅ INSERT uses copy-on-write semantics (append-only, no delete files); UPDATE/DELET
Schema Evolution v2 ✅ pass full ✅ ALTER TABLE ADD / RENAME / DROP COLUMN supported via REST catalog
Type Promotion / Widening v2 ⏭️ skip none ✅ ALTER COLUMN type promotion is not a documented DuckDB-Iceberg operation; not ex
Column Default Values v3 ✅ pass full ✅ V3 schema-level column DEFAULT values applied on CREATE and ALTER ADD COLUMN
Time Travel / Snapshots v2 ✅ pass full ✅ Time travel via AT (VERSION => snapshot_id) returns the historical row count
Table Maintenance v2 ❌ fail none ✅ DuckDB Iceberg does not provide maintenance ops (compaction, expire snapshots)
Branching & Tagging v2 ❌ fail none ✅ DuckDB Iceberg does not support branching or tagging
Hidden Partitioning v2 ✅ pass full ✅ Created and inserted into a table partitioned by bucket()/truncate() transforms
Partition Evolution v2 ✅ pass full ✅ Evolved the partition spec with ALTER TABLE ... SET PARTITIONED BY and kept read
Multi-Argument Transforms v3 ⏭️ skip unknown ✅ V3 multi-argument transforms are undocumented for DuckDB; not exercised
Statistics (Column Metrics) v2 ✅ pass full ✅ iceberg_metadata exposes per-file record_count statistics written by DuckDB
Bloom Filters v2 ❌ fail none ✅ DuckDB Iceberg does not read or write Iceberg bloom filters
Catalog Integration v2 ✅ pass full ✅ Attached an Iceberg REST catalog and performed namespace/table operations
Hadoop Catalog v2 ❌ fail none ✅ DuckDB Iceberg supports only REST-based catalogs (Hadoop catalog unsupported)
JDBC Catalog v2 ❌ fail none ✅ DuckDB Iceberg supports only REST-based catalogs (JDBC catalog unsupported)
REST Catalog v2 ✅ pass full ✅ Full read/write round-trip against an Iceberg REST catalog (OAuth2/none auth)
Hive Metastore v2 ❌ fail none ✅ DuckDB Iceberg supports only REST-based catalogs (Hive Metastore unsupported)
AWS Glue Catalog v2 ⏭️ skip full ✅ AWS Glue (SageMaker Lakehouse) catalog requires AWS credentials; not exercised l
Nessie v2 ❌ fail none ✅ DuckDB Iceberg does not natively support the Nessie catalog
Polaris v2 ⏭️ skip full ✅ Polaris uses the Iceberg REST protocol; a Polaris-specific server is not run her
Unity Catalog v2 ⏭️ skip partial ✅ Unity Catalog REST connectivity is undocumented for DuckDB; requires a Unity ser
Variant Type v3 ✅ pass full ✅ Created a V3 table with a VARIANT column and round-tripped a value
Shredded Variant v3 ❌ fail none ✅ DuckDB does not support shredded variant encoding (V3-only feature)
Geometry / Geo Types v3 ⏭️ skip partial ✅ REST catalog rejected the table schema, so DuckDB's support could not be measure
Nanosecond Timestamps v3 ✅ pass full ✅ Created a V3 table with a TIMESTAMP_NS column and inserted a nanosecond value
Lineage Tracking v3 ✅ pass full ✅ V3 write path with row lineage; row-level UPDATE encoded as a binary deletion ve

@github-actions

github-actions Bot commented Jul 30, 2026

Copy link
Copy Markdown

ClickHouse Iceberg Feature Test Report

  • Timestamp: 2026-07-30T10:57:53.185797+00:00
  • ClickHouse Version: 26.7.1.1315

Summary

Metric Count
Total 33
✅ Passed 1
❌ Failed 21
⏭️ Skipped 0
⚠️ Errors 10
🔍 Discrepancies 7

Test Results

Feature Version Result JSON Level Match Details
Table Creation v2 ❌ fail none ✅ ClickHouse cannot create Iceberg tables; icebergLocal() is read-only
Read Support v2 ⚠️ error full ✅ Traceback (most recent call last): File "/home/runner/work/iceberg-matrix/iceb
Write (INSERT) v2 ⚠️ error none ✅ Traceback (most recent call last): File "/home/runner/work/iceberg-matrix/iceb
Write (MERGE/UPDATE/DELETE) v2 ❌ fail none ✅ ClickHouse Iceberg support is read-only; no UPDATE, DELETE, or MERGE support
Position Deletes v2 ⚠️ error full ✅ Traceback (most recent call last): File "/home/runner/work/iceberg-matrix/iceb
Equality Deletes v2 ⚠️ error partial ✅ Traceback (most recent call last): File "/home/runner/work/iceberg-matrix/iceb
Merge-on-Read v2 ⚠️ error full ✅ Traceback (most recent call last): File "/home/runner/work/iceberg-matrix/iceb
Copy-on-Write v2 ❌ fail none ✅ ClickHouse is read-only for Iceberg; Copy-on-Write write mode not applicable
Schema Evolution v2 ⚠️ error partial ✅ Traceback (most recent call last): File "/home/runner/work/iceberg-matrix/iceb
Type Promotion / Widening v2 ⚠️ error partial ✅ Traceback (most recent call last): File "/home/runner/work/iceberg-matrix/iceb
Column Default Values v3 ❌ fail none ✅ Column default values not supported in ClickHouse Iceberg
Time Travel / Snapshots v2 ⚠️ error full ✅ Traceback (most recent call last): File "/home/runner/work/iceberg-matrix/iceb
Table Maintenance v2 ❌ fail none ✅ ClickHouse does not support Iceberg table maintenance (compaction, snapshot expi
Branching & Tagging v2 ❌ fail none ✅ ClickHouse does not support Iceberg branching or tagging
Hidden Partitioning v2 ⚠️ error full ✅ Traceback (most recent call last): File "/home/runner/work/iceberg-matrix/iceb
Partition Evolution v2 ⚠️ error unknown ✅ Traceback (most recent call last): File "/home/runner/work/iceberg-matrix/iceb
Multi-Argument Transforms v3 ❌ fail none ✅ Multi-argument transforms not supported in ClickHouse Iceberg
Statistics (Column Metrics) v2 ✅ pass partial ✅ ClickHouse uses Iceberg column-level statistics from manifest files for scan pru
Bloom Filters v2 ❌ fail none ✅ ClickHouse does not support Iceberg bloom filter indexes
Catalog Integration v2 ❓ partial full ✅ ClickHouse supports icebergLocal() and IcebergS3/IcebergAzureBlobStorage table f
Hadoop Catalog v2 ❌ fail none ✅ ClickHouse does not support Hadoop catalog; uses direct path-based access
JDBC Catalog v2 ❌ fail none ✅ ClickHouse does not support JDBC catalog
REST Catalog v2 ❌ fail full ❌ DISCREPANCY ClickHouse does not support Iceberg REST catalog protocol
Hive Metastore v2 ❌ fail none ✅ ClickHouse does not support Hive Metastore catalog for Iceberg
AWS Glue Catalog v2 ❌ fail full ❌ DISCREPANCY ClickHouse does not support AWS Glue catalog for Iceberg
Nessie v2 ❌ fail none ✅ ClickHouse does not support Nessie catalog
Polaris v2 ❌ fail partial ❌ DISCREPANCY ClickHouse does not support Polaris catalog
Unity Catalog v2 ❌ fail partial ❌ DISCREPANCY ClickHouse does not support Unity Catalog
Variant Type v3 ❌ fail unknown ❌ DISCREPANCY Variant type not supported in ClickHouse Iceberg
Shredded Variant v3 ❌ fail unknown ❌ DISCREPANCY Shredded variant not supported in ClickHouse Iceberg
Geometry / Geo Types v3 ❌ fail unknown ❌ DISCREPANCY Geometry type not supported in ClickHouse Iceberg
Nanosecond Timestamps v3 ❌ fail none ✅ Nanosecond timestamps not supported in ClickHouse Iceberg
Lineage Tracking v2 ❌ fail none ✅ Lineage tracking not supported in ClickHouse Iceberg

⚠️ Discrepancies

  • REST Catalog (v2): test=fail, json=full — ClickHouse does not support Iceberg REST catalog protocol
  • AWS Glue Catalog (v2): test=fail, json=full — ClickHouse does not support AWS Glue catalog for Iceberg
  • Polaris (v2): test=fail, json=partial — ClickHouse does not support Polaris catalog
  • Unity Catalog (v2): test=fail, json=partial — ClickHouse does not support Unity Catalog
  • Variant Type (v3): test=fail, json=unknown — Variant type not supported in ClickHouse Iceberg
  • Shredded Variant (v3): test=fail, json=unknown — Shredded variant not supported in ClickHouse Iceberg
  • Geometry / Geo Types (v3): test=fail, json=unknown — Geometry type not supported in ClickHouse Iceberg

@github-actions

github-actions Bot commented Jul 30, 2026

Copy link
Copy Markdown

Iceberg Feature Test Report (Spark, V2 + V3)

  • Timestamp: 2026-07-30T10:58:31.619348Z
  • Spark Version: 4.1.2
  • Iceberg Version: 1.11.0
  • Catalog: REST (http://127.0.0.1:8181/catalog)
  • Format Versions Tested: v2, v3

Summary

Metric Count
Total 70
✅ Passed 44
❌ Failed 4
⏭️ Skipped 22
⚠️ Errors 0
🔍 Discrepancies 0
🧭 Uncovered matrix features 0

Matrix coverage: 35/35 features in features.json have a test.

Test Results

Feature Version Result JSON Level Match Details
Table Creation v2 ✅ pass full ✅ Created Iceberg V2 table with multiple column types
Table Creation v3 ✅ pass full ✅ Created Iceberg V3 table with multiple column types
Read Support v2 ✅ pass full ✅ SELECT with predicate pushdown and column projection works
Read Support v3 ✅ pass full ✅ SELECT with predicate pushdown and column projection works
Write (INSERT) v2 ✅ pass full ✅ INSERT INTO works correctly with multiple batches
Write (INSERT) v3 ✅ pass full ✅ INSERT INTO works correctly with multiple batches
Write (MERGE/UPDATE/DELETE) v2 ✅ pass full ✅ UPDATE, DELETE, and MERGE INTO all work correctly
Write (MERGE/UPDATE/DELETE) v3 ✅ pass full ✅ UPDATE, DELETE, and MERGE INTO all work correctly
Position Deletes v2 ✅ pass full ✅ Position delete files / DVs created and applied correctly in MoR mode
Position Deletes v3 ✅ pass full ✅ Position delete files / DVs created and applied correctly in MoR mode
Equality Deletes v2 ✅ pass full ✅ Equality deletes readable; Spark SQL DELETE in MoR works correctly
Equality Deletes v3 ✅ pass full ✅ Equality deletes readable; Spark SQL DELETE in MoR works correctly
Merge-on-Read v2 ✅ pass full ✅ Merge-on-read produces delete files/DVs on UPDATE; reads merge correctly
Merge-on-Read v3 ✅ pass full ✅ Merge-on-read produces delete files/DVs on UPDATE; reads merge correctly
Copy-on-Write v2 ✅ pass full ✅ Copy-on-write rewrites data files (no delete files) on UPDATE
Copy-on-Write v3 ✅ pass full ✅ Copy-on-write rewrites data files (no delete files) on UPDATE
Schema Evolution v2 ✅ pass full ✅ ADD COLUMNS, RENAME COLUMN, DROP COLUMN all work correctly
Schema Evolution v3 ✅ pass full ✅ ADD COLUMNS, RENAME COLUMN, DROP COLUMN all work correctly
Type Promotion / Widening v2 ✅ pass full ✅ INT→BIGINT and FLOAT→DOUBLE promotions work correctly
Type Promotion / Widening v3 ✅ pass full ✅ INT→BIGINT and FLOAT→DOUBLE promotions work correctly
Column Default Values v2 ⏭️ skip none ✅ V3-only feature; not applicable to format-version 2 tables
Column Default Values v3 ❌ fail none ✅ Column default values not supported: [UNSUPPORTED_FEATURE.TABLE_OPERATION] The f
Time Travel / Snapshots v2 ✅ pass full ✅ VERSION AS OF time travel works correctly with snapshot IDs
Time Travel / Snapshots v3 ✅ pass full ✅ VERSION AS OF time travel works correctly with snapshot IDs
Table Maintenance v2 ✅ pass full ✅ rewrite_data_files and expire_snapshots procedures work correctly
Table Maintenance v3 ✅ pass full ✅ rewrite_data_files and expire_snapshots procedures work correctly
Branching & Tagging v2 ✅ pass full ✅ CREATE BRANCH, CREATE TAG, write to branch, and read from tag all work
Branching & Tagging v3 ✅ pass full ✅ CREATE BRANCH, CREATE TAG, write to branch, and read from tag all work
Hidden Partitioning v2 ✅ pass full ✅ Hidden partitioning with year(), bucket(), truncate() transforms works
Hidden Partitioning v3 ✅ pass full ✅ Hidden partitioning with year(), bucket(), truncate() transforms works
Partition Evolution v2 ✅ pass full ✅ Partition evolution (ADD PARTITION FIELD) works without rewriting data
Partition Evolution v3 ✅ pass full ✅ Partition evolution (ADD PARTITION FIELD) works without rewriting data
Multi-Argument Transforms v2 ⏭️ skip none ✅ V3-only feature; not applicable to format-version 2 tables
Multi-Argument Transforms v3 ❌ fail none ✅ Multi-source-column transforms not supported: Cannot convert transform with more
Statistics (Column Metrics) v2 ✅ pass full ✅ Column statistics (record_count, value_counts, bounds) present in manifest files
Statistics (Column Metrics) v3 ✅ pass full ✅ Column statistics (record_count, value_counts, bounds) present in manifest files
Bloom Filters v2 ✅ pass full ✅ Bloom filter properties configured and data read correctly
Bloom Filters v3 ✅ pass full ✅ Bloom filter properties configured and data read correctly
Catalog Integration v2 ✅ pass full ✅ Catalog integration works (SHOW NAMESPACES, SHOW TABLES, CREATE/DROP)
Catalog Integration v3 ✅ pass full ✅ Catalog integration works (SHOW NAMESPACES, SHOW TABLES, CREATE/DROP)
Hadoop Catalog v2 ✅ pass full ✅ Hadoop catalog: create, write, read, drop all work on local filesystem
Hadoop Catalog v3 ✅ pass full ✅ Hadoop catalog: create, write, read, drop all work on local filesystem
JDBC Catalog v2 ⏭️ skip full ✅ JDBC catalog test skipped in CI (requires external JDBC database)
JDBC Catalog v3 ⏭️ skip full ✅ JDBC catalog test skipped in CI (requires external JDBC database)
REST Catalog v2 ✅ pass full ✅ REST catalog (http://127.0.0.1:8181/catalog): namespace/table CRUD, write, and r
REST Catalog v3 ✅ pass full ✅ REST catalog (http://127.0.0.1:8181/catalog): namespace/table CRUD, write, and r
Hive Metastore v2 ⏭️ skip full ✅ Hive Metastore test skipped in CI (requires running Hive Metastore service)
Hive Metastore v3 ⏭️ skip full ✅ Hive Metastore test skipped in CI (requires running Hive Metastore service)
AWS Glue Catalog v2 ⏭️ skip full ✅ AWS Glue test skipped in CI (requires AWS credentials and Glue service)
AWS Glue Catalog v3 ⏭️ skip full ✅ AWS Glue test skipped in CI (requires AWS credentials and Glue service)
Nessie v2 ⏭️ skip full ✅ Nessie test skipped in CI (requires running Nessie server)
Nessie v3 ⏭️ skip full ✅ Nessie test skipped in CI (requires running Nessie server)
Polaris v2 ⏭️ skip full ✅ Polaris test skipped in CI (requires running Polaris server)
Polaris v3 ⏭️ skip full ✅ Polaris test skipped in CI (requires running Polaris server)
Unity Catalog v2 ⏭️ skip full ✅ Unity Catalog test skipped in CI (requires Unity Catalog endpoint)
Unity Catalog v3 ⏭️ skip full ✅ Unity Catalog test skipped in CI (requires Unity Catalog endpoint)
Snowflake Horizon Catalog v2 ⏭️ skip full ✅ Snowflake Horizon Catalog test skipped in CI (requires Snowflake REST endpoint)
Snowflake Horizon Catalog v3 ⏭️ skip partial ✅ Snowflake Horizon Catalog test skipped in CI (requires Snowflake REST endpoint)
Variant Type v2 ⏭️ skip none ✅ V3-only feature; not applicable to format-version 2 tables
Variant Type v3 ✅ pass full ✅ VARIANT type column created and written on V3 table
Shredded Variant v2 ⏭️ skip none ✅ V3-only feature; not applicable to format-version 2 tables
Shredded Variant v3 ✅ pass full ✅ VARIANT column with shredding property created and written on V3 table
Geometry / Geo Types v2 ⏭️ skip none ✅ V3-only feature; not applicable to format-version 2 tables
Geometry / Geo Types v3 ❌ fail none ✅ Spark accepts the type but the Iceberg Spark connector rejects it: Not a support
Nanosecond Timestamps v2 ⏭️ skip none ✅ V3-only feature; not applicable to format-version 2 tables
Nanosecond Timestamps v3 ❌ fail none ✅ Nanosecond timestamps not supported: [UNSUPPORTED_DATATYPE] Unsupported data ty
Lineage Tracking v2 ⏭️ skip none ✅ V3-only feature; not applicable to format-version 2 tables
Lineage Tracking v3 ✅ pass full ✅ Row lineage metadata columns (_row_id, _last_updated_sequence_number) readable o
Deletion Vectors v2 ⏭️ skip none ✅ V3-only feature; not applicable to format-version 2 tables
Deletion Vectors v3 ✅ pass full ✅ V3 deletion vectors (Puffin) produced on DELETE

@Neuw84
Neuw84 merged commit ec98559 into main Jul 30, 2026
6 checks passed
@github-actions

github-actions Bot commented Jul 30, 2026

Copy link
Copy Markdown

Flink Iceberg Feature Test Report

  • Timestamp: 2026-07-30T11:21:10.230355+00:00
  • Flink Version: 2.3.0
  • Iceberg Version: 1.11.0
  • Execution mode: docker

Summary

Metric Count
Total 70
Passed 36
Failed 6
Skipped 24
Errors 4
Discrepancies vs matrix 0
Unverified (skip/error) 28

Failed is a result, not a defect: it records that the engine does not support the feature through Flink SQL. A discrepancy means the observed behaviour disagrees with flink.json.

Test Results

Feature Version Result Matrix Match Details
Table Creation v2 PASS full ok CREATE TABLE with 4 column types on a V2 table, then INSERT and read-back of every column
Read Support v2 PASS full ok Batch: read 3 rows with correct predicate filtering and projection. Streaming: a continuous read (streaming=true, monitor-interval) delivered the init
Write (INSERT) v2 PASS full ok Batch: INSERT INTO appended across 2 commits (3 rows); INSERT OVERWRITE replaced them (1 row). Streaming: an unbounded INSERT committed 4 append snaps
Write (MERGE/UPDATE/DELETE) v2 FAIL partial ok Neither DELETE nor UPDATE is supported in Flink SQL; upsert mode is the only row-level write path. DELETE: java.lang.UnsupportedOperationException: Ca
Position Deletes v2 PASS full ok Write path produced position deletes (content=1): ['1:PARQUET'], and the read merged them correctly (rows=['1:updated', '2:second']). Emitted for a ro
Equality Deletes v2 PASS full ok UPSERT replaced the row (rows=['1:updated', '2:second']) and wrote equality delete files (content=2): ['2:PARQUET', '2:PARQUET']
Merge-on-Read v2 PASS full ok Upsert produced delete files that the reader merged at scan time (deletes=['2:PARQUET', '2:PARQUET'], rows=['1:updated', '2:second']). Merge-on-read i
Copy-on-Write v2 PASS partial ok INSERT OVERWRITE rewrote the data files with no delete files, which is the copy-on-write path Flink SQL can reach. Note the scope: copy-on-write prope
Deletion Vectors v2 SKIP none ok V3-only feature; not applicable to format-version 2 tables
Schema Evolution v2 PASS full ok ADD, RENAME and DROP COLUMN all succeeded via Flink DDL; existing rows readable
Type Promotion / Widening v2 PASS full ok INT→BIGINT and FLOAT→DOUBLE widening applied; out-of-INT-range value stored and read back
Column Default Values v2 SKIP none ok V3-only feature; not applicable to format-version 2 tables
Time Travel / Snapshots v2 PASS full ok Current table has 2 rows; reading snapshot 2952679393888323171 via the snapshot-id hint returned the 1 row present at that snapshot
Table Maintenance v2 FAIL partial ok Streaming job committed 55 snapshots but no rewrite commit appeared within 180s (operations=['append'])
Branching & Tagging v2 PASS partial ok Branch reads via the branch hint, tag reads via the tag hint, and tag-to-tag incremental scans (start-tag/end-tag) all work against refs created throu
Hidden Partitioning v2 PASS partial ok Flink cannot declare transform partitioning (PARTITIONED BY (days(ts)) is a parser error), but on a day(ts)-partitioned table created through the cata
Partition Evolution v2 PASS partial ok Flink cannot initiate partition evolution (no SQL syntax; ADD PARTITION FIELD is a parser error) but honours it fully once the catalog evolves the spe
Multi-Argument Transforms v2 SKIP none ok V3-only feature; not applicable to format-version 2 tables
Statistics (Column Metrics) v2 PASS full ok Data file manifest carries record_count=3 and per-column value_counts for 2 columns, plus null_value_counts ['2']
Bloom Filters & Puffin v2 SKIP unknown ok Not verifiable from SQL: the Parquet bloom-filter write property is accepted and point lookups return correct results, but no Flink SQL surface or Ice
Catalog Integration v2 PASS full ok Full create/write/read/drop round-trip through an Iceberg catalog
REST Catalog v2 PASS full ok catalog-type='rest' against a live Lakekeeper REST catalog: table created, written, read back and dropped
Hadoop Catalog v2 ERR full ok org.apache.flink.table.catalog.exceptions.CatalogException: A database with name [test_db] does not exist in the catalog: [test_catalog].�[0m Shutting
JDBC Catalog v2 ERR full ok java.io.IOException: Mkdirs failed to create file:/work/hadoop-warehouse/test_db/cat_99929532/metadata (exists=false, cwd=file:/opt/flink)�[0m Shuttin
Hive Metastore v2 SKIP full ok Not exercised: requires a running Hive Metastore (thrift) service. Flink ships the catalog implementation, but this harness has no such endpoint to pr
AWS Glue Catalog v2 SKIP full ok Not exercised: requires AWS credentials and a Glue Data Catalog. Flink ships the catalog implementation, but this harness has no such endpoint to prov
Nessie v2 SKIP full ok Not exercised: requires a running Nessie server. Flink ships the catalog implementation, but this harness has no such endpoint to prove it against
Polaris v2 SKIP full ok Not exercised: requires a running Apache Polaris server (reachable via catalog-type='rest'). Flink ships the catalog implementation, but this harness
Unity Catalog v2 SKIP partial ok Not exercised: requires a Databricks Unity Catalog endpoint. Flink ships the catalog implementation, but this harness has no such endpoint to prove it
Snowflake Horizon Catalog v2 SKIP full ok Not exercised: requires a Snowflake account with Horizon Catalog enabled. Flink ships the catalog implementation, but this harness has no such endpoin
Variant Type v2 SKIP none ok V3-only feature; not applicable to format-version 2 tables
Shredded Variant v2 SKIP none ok V3-only feature; not applicable to format-version 2 tables
Geometry / Geo Types v2 SKIP none ok V3-only feature; not applicable to format-version 2 tables
Nanosecond Timestamps v2 SKIP none ok V3-only feature; not applicable to format-version 2 tables
Lineage Tracking v2 SKIP none ok V3-only feature; not applicable to format-version 2 tables
Table Creation v3 PASS full ok CREATE TABLE with 4 column types on a V3 table, then INSERT and read-back of every column
Read Support v3 PASS full ok Batch: read 3 rows with correct predicate filtering and projection. Streaming: a continuous read (streaming=true, monitor-interval) delivered the init
Write (INSERT) v3 PASS full ok Batch: INSERT INTO appended across 2 commits (3 rows); INSERT OVERWRITE replaced them (1 row). Streaming: an unbounded INSERT committed 4 append snaps
Write (MERGE/UPDATE/DELETE) v3 FAIL partial ok Neither DELETE nor UPDATE is supported in Flink SQL; upsert mode is the only row-level write path. DELETE: java.lang.UnsupportedOperationException: Ca
Position Deletes v3 PASS full ok Write path produced position deletes (content=1): ['1:PUFFIN'], and the read merged them correctly (rows=['1:updated', '2:second']). Emitted for a row
Equality Deletes v3 PASS full ok UPSERT replaced the row (rows=['1:updated', '2:second']) and wrote equality delete files (content=2): ['2:PARQUET', '2:PARQUET']
Merge-on-Read v3 PASS full ok Upsert produced delete files that the reader merged at scan time (deletes=['2:PARQUET', '2:PARQUET'], rows=['1:updated', '2:second']). Merge-on-read i
Copy-on-Write v3 PASS partial ok INSERT OVERWRITE rewrote the data files with no delete files, which is the copy-on-write path Flink SQL can reach. Note the scope: copy-on-write prope
Deletion Vectors v3 PASS full ok V3 deletion vectors written from plain Flink SQL: ['1:PUFFIN'] (content=1 position deletes in puffin), alongside equality deletes for the key-based pa
Schema Evolution v3 PASS full ok ADD, RENAME and DROP COLUMN all succeeded via Flink DDL; existing rows readable
Type Promotion / Widening v3 PASS full ok INT→BIGINT and FLOAT→DOUBLE widening applied; out-of-INT-range value stored and read back
Column Default Values v3 FAIL none ok Flink SQL cannot declare column defaults: org.apache.flink.sql.parser.impl.ParseException: Encountered "DEFAULT" at line 1, column 53. Was expecting o
Time Travel / Snapshots v3 PASS full ok Current table has 2 rows; reading snapshot 8840950003780235066 via the snapshot-id hint returned the 1 row present at that snapshot
Table Maintenance v3 FAIL partial ok Streaming job committed 56 snapshots but no rewrite commit appeared within 180s (operations=['append'])
Branching & Tagging v3 PASS partial ok Branch reads via the branch hint, tag reads via the tag hint, and tag-to-tag incremental scans (start-tag/end-tag) all work against refs created throu
Hidden Partitioning v3 PASS partial ok Flink cannot declare transform partitioning (PARTITIONED BY (days(ts)) is a parser error), but on a day(ts)-partitioned table created through the cata
Partition Evolution v3 PASS partial ok Flink cannot initiate partition evolution (no SQL syntax; ADD PARTITION FIELD is a parser error) but honours it fully once the catalog evolves the spe
Multi-Argument Transforms v3 SKIP unknown ok Not exercised: Flink DDL cannot express any transform partitioning at all (PARTITIONED BY only takes plain column names), so a multi-argument transfor
Statistics (Column Metrics) v3 PASS full ok Data file manifest carries record_count=3 and per-column value_counts for 2 columns, plus null_value_counts ['2']
Bloom Filters & Puffin v3 SKIP unknown ok Not verifiable from SQL: the Parquet bloom-filter write property is accepted and point lookups return correct results, but no Flink SQL surface or Ice
Catalog Integration v3 PASS full ok Full create/write/read/drop round-trip through an Iceberg catalog
REST Catalog v3 PASS full ok catalog-type='rest' against a live Lakekeeper REST catalog: table created, written, read back and dropped
Hadoop Catalog v3 ERR full ok org.apache.flink.table.catalog.exceptions.CatalogException: A database with name [test_db] does not exist in the catalog: [test_catalog].�[0m Shutting
JDBC Catalog v3 ERR full ok java.io.IOException: Mkdirs failed to create file:/work/hadoop-warehouse/test_db/cat_974a5eea/metadata (exists=false, cwd=file:/opt/flink)�[0m Shuttin
Hive Metastore v3 SKIP full ok Not exercised: requires a running Hive Metastore (thrift) service. Flink ships the catalog implementation, but this harness has no such endpoint to pr
AWS Glue Catalog v3 SKIP full ok Not exercised: requires AWS credentials and a Glue Data Catalog. Flink ships the catalog implementation, but this harness has no such endpoint to prov
Nessie v3 SKIP full ok Not exercised: requires a running Nessie server. Flink ships the catalog implementation, but this harness has no such endpoint to prove it against
Polaris v3 SKIP full ok Not exercised: requires a running Apache Polaris server (reachable via catalog-type='rest'). Flink ships the catalog implementation, but this harness
Unity Catalog v3 SKIP partial ok Not exercised: requires a Databricks Unity Catalog endpoint. Flink ships the catalog implementation, but this harness has no such endpoint to prove it
Snowflake Horizon Catalog v3 SKIP partial ok Not exercised: requires a Snowflake account with Horizon Catalog enabled. Flink ships the catalog implementation, but this harness has no such endpoin
Variant Type v3 PASS partial ok VARIANT column on a V3 table: value written with PARSE_JSON and read back non-null. Field extraction is not possible from Flink SQL -- Flink 2.3 expos
Shredded Variant v3 SKIP unknown ok Not verifiable from SQL: the shredding table property is accepted and data round-trips, but whether the writer actually shredded the variant is not ob
Geometry / Geo Types v3 FAIL none ok Blocked by Flink, not Iceberg: the Calcite-based planner keeps GEOMETRY behind its spatial extensions (enabled via the Calcite fun=spatial connect str
Nanosecond Timestamps v3 PASS full ok TIMESTAMP(9) maps to the Iceberg V3 timestamp_ns type (confirmed in the table schema) and a nanosecond-precision value round-trips exactly
Lineage Tracking v3 PASS partial ok Flink maintains V3 row lineage on write -- after inserting 3 rows the table metadata carries next-row-id=3 and the snapshot reports first-row-id=0, ad

Unverified

These could not be exercised here, so they neither confirm nor contradict the matrix:

  • Deletion Vectors (v2): matrix none — V3-only feature; not applicable to format-version 2 tables
  • Column Default Values (v2): matrix none — V3-only feature; not applicable to format-version 2 tables
  • Multi-Argument Transforms (v2): matrix none — V3-only feature; not applicable to format-version 2 tables
  • Bloom Filters & Puffin (v2): matrix unknown — Not verifiable from SQL: the Parquet bloom-filter write property is accepted and point lookups return correct results, but no Flink SQL surface or Iceberg metadata table reports whether a bloom filter
  • Hadoop Catalog (v2): matrix full — org.apache.flink.table.catalog.exceptions.CatalogException: A database with name [test_db] does not exist in the catalog: [test_catalog].�[0m Shutting down the session... done. WARNING: Unknown module
  • JDBC Catalog (v2): matrix full — java.io.IOException: Mkdirs failed to create file:/work/hadoop-warehouse/test_db/cat_99929532/metadata (exists=false, cwd=file:/opt/flink)�[0m Shutting down the session... done. WARNING: Unknown modul
  • Hive Metastore (v2): matrix full — Not exercised: requires a running Hive Metastore (thrift) service. Flink ships the catalog implementation, but this harness has no such endpoint to prove it against
  • AWS Glue Catalog (v2): matrix full — Not exercised: requires AWS credentials and a Glue Data Catalog. Flink ships the catalog implementation, but this harness has no such endpoint to prove it against
  • Nessie (v2): matrix full — Not exercised: requires a running Nessie server. Flink ships the catalog implementation, but this harness has no such endpoint to prove it against
  • Polaris (v2): matrix full — Not exercised: requires a running Apache Polaris server (reachable via catalog-type='rest'). Flink ships the catalog implementation, but this harness has no such endpoint to prove it against
  • Unity Catalog (v2): matrix partial — Not exercised: requires a Databricks Unity Catalog endpoint. Flink ships the catalog implementation, but this harness has no such endpoint to prove it against
  • Snowflake Horizon Catalog (v2): matrix full — Not exercised: requires a Snowflake account with Horizon Catalog enabled. Flink ships the catalog implementation, but this harness has no such endpoint to prove it against
  • Variant Type (v2): matrix none — V3-only feature; not applicable to format-version 2 tables
  • Shredded Variant (v2): matrix none — V3-only feature; not applicable to format-version 2 tables
  • Geometry / Geo Types (v2): matrix none — V3-only feature; not applicable to format-version 2 tables
  • Nanosecond Timestamps (v2): matrix none — V3-only feature; not applicable to format-version 2 tables
  • Lineage Tracking (v2): matrix none — V3-only feature; not applicable to format-version 2 tables
  • Multi-Argument Transforms (v3): matrix unknown — Not exercised: Flink DDL cannot express any transform partitioning at all (PARTITIONED BY only takes plain column names), so a multi-argument transform cannot be declared from SQL
  • Bloom Filters & Puffin (v3): matrix unknown — Not verifiable from SQL: the Parquet bloom-filter write property is accepted and point lookups return correct results, but no Flink SQL surface or Iceberg metadata table reports whether a bloom filter
  • Hadoop Catalog (v3): matrix full — org.apache.flink.table.catalog.exceptions.CatalogException: A database with name [test_db] does not exist in the catalog: [test_catalog].�[0m Shutting down the session... done. WARNING: Unknown module
  • JDBC Catalog (v3): matrix full — java.io.IOException: Mkdirs failed to create file:/work/hadoop-warehouse/test_db/cat_974a5eea/metadata (exists=false, cwd=file:/opt/flink)�[0m Shutting down the session... done. WARNING: Unknown modul
  • Hive Metastore (v3): matrix full — Not exercised: requires a running Hive Metastore (thrift) service. Flink ships the catalog implementation, but this harness has no such endpoint to prove it against
  • AWS Glue Catalog (v3): matrix full — Not exercised: requires AWS credentials and a Glue Data Catalog. Flink ships the catalog implementation, but this harness has no such endpoint to prove it against
  • Nessie (v3): matrix full — Not exercised: requires a running Nessie server. Flink ships the catalog implementation, but this harness has no such endpoint to prove it against
  • Polaris (v3): matrix full — Not exercised: requires a running Apache Polaris server (reachable via catalog-type='rest'). Flink ships the catalog implementation, but this harness has no such endpoint to prove it against
  • Unity Catalog (v3): matrix partial — Not exercised: requires a Databricks Unity Catalog endpoint. Flink ships the catalog implementation, but this harness has no such endpoint to prove it against
  • Snowflake Horizon Catalog (v3): matrix partial — Not exercised: requires a Snowflake account with Horizon Catalog enabled. Flink ships the catalog implementation, but this harness has no such endpoint to prove it against
  • Shredded Variant (v3): matrix unknown — Not verifiable from SQL: the shredding table property is accepted and data round-trips, but whether the writer actually shredded the variant is not observable through any Flink SQL surface or metadata

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@Neuw84