From 863042d9a2466aa6385c870278dac8703e524242 Mon Sep 17 00:00:00 2001 From: Kat Batuigas Date: Wed, 22 Jul 2026 18:16:03 -0700 Subject: [PATCH 01/10] Draft Iceberg monitoring guide --- modules/ROOT/nav.adoc | 1 + .../pages/iceberg/monitor-iceberg-health.adoc | 128 ++++++++++++++++++ 2 files changed, 129 insertions(+) create mode 100644 modules/manage/pages/iceberg/monitor-iceberg-health.adoc diff --git a/modules/ROOT/nav.adoc b/modules/ROOT/nav.adoc index 97e56aa5fd..7a47c49106 100644 --- a/modules/ROOT/nav.adoc +++ b/modules/ROOT/nav.adoc @@ -211,6 +211,7 @@ *** xref:manage:iceberg/query-iceberg-topics.adoc[Query Iceberg Topics] *** xref:manage:iceberg/migrate-iceberg-catalog.adoc[Migrate Iceberg Catalogs] *** xref:manage:iceberg/iceberg-performance-tuning.adoc[Tune Iceberg Performance] +*** xref:manage:iceberg/monitor-iceberg-health.adoc[] *** xref:manage:iceberg/iceberg-troubleshooting.adoc[Troubleshoot Iceberg Topics] ** xref:manage:schema-reg/index.adoc[Schema Registry] *** xref:manage:schema-reg/schema-reg-overview.adoc[Overview] diff --git a/modules/manage/pages/iceberg/monitor-iceberg-health.adoc b/modules/manage/pages/iceberg/monitor-iceberg-health.adoc new file mode 100644 index 0000000000..7b7fae2b59 --- /dev/null +++ b/modules/manage/pages/iceberg/monitor-iceberg-health.adoc @@ -0,0 +1,128 @@ += Monitor Iceberg Health +:description: Monitor the health of Iceberg topics using the Admin API to check external catalog reachability and per-partition commit lag. +:page-categories: Iceberg, Monitoring + +// tag::single-source[] +:page-topic-type: how-to +:personas: ops_admin, streaming_developer +:learning-objective-1: Check whether the external Iceberg catalog is reachable +:learning-objective-2: Identify Iceberg topics with the highest commit lag + +ifndef::env-cloud[] +[NOTE] +==== +include::shared:partial$enterprise-license.adoc[] +==== +endif::[] + +{description} + +After reading this page, you will be able to: + +* [ ] {learning-objective-1} +* [ ] {learning-objective-2} + +Use the `GetIcebergStatus` Admin API endpoint to check whether your external Iceberg catalog is reachable and to see how far behind catalog commit is for each Iceberg topic and partition. This helps you detect a misconfigured or unreachable catalog and identify the partitions with the highest commit lag on a busy cluster. + +NOTE: This endpoint is available in Redpanda version 26.2 and later. + +[IMPORTANT] +==== +This endpoint requires superuser privileges, even though it is read-only. See xref:manage:use-admin-api.adoc#authentication[Admin API authentication]. +==== + +== Check Iceberg health + +Send a POST request to the link:/api/doc/admin/v2/operation/operation-redpanda-core-admin-v2-icebergservice-geticebergstatus[`redpanda.core.admin.v2.IcebergService/GetIcebergStatus`] endpoint. With an empty request body, the response covers all Iceberg topics that Redpanda is currently tracking for translation: + +[,bash] +---- +curl \ + -u : \ + --request POST 'http://:/redpanda.core.admin.v2.IcebergService/GetIcebergStatus' \ + --header "Content-Type: application/json" \ + --data '{}' +---- + +To report on specific topics only, set `topicsFilter` to a list of Iceberg topic names: + +[,bash] +---- +curl \ + -u : \ + --request POST 'http://:/redpanda.core.admin.v2.IcebergService/GetIcebergStatus' \ + --header "Content-Type: application/json" \ + --data '{ "topicsFilter": ["orders", "clicks"] }' +---- + +* Request headers `Connect-Protocol-Version` and `Connect-Timeout-Ms` are optional. +* v2 endpoints also accept binary-encoded Protobuf request bodies. Use the `Content-Type: application/proto` header. + +A response looks like the following: + +[,json] +---- +{ + "catalog": { + "reachable": true, + "errorCode": "", + "errorMessage": "" + }, + "topics": [ + { + "topic": "orders", + "lifecycleState": "ICEBERG_LIFECYCLE_STATE_LIVE", + "partitions": [ + { + "partition": 0, + "lastTranslatedOffset": "10432", + "lastCommittedOffset": "10400", + "commitLag": "32" + } + ] + } + ] +} +---- + +== Interpret catalog reachability + +The `catalog` object reports whether Redpanda can reach the external Iceberg catalog: + +* *`reachable`*: `true` if the catalog responded, `false` otherwise. +* *`errorCode`*: When the catalog is unreachable, a short error identifier from the catalog error set, such as `io_error` or `timedout`. Empty when `reachable` is `true`. +* *`errorMessage`*: A human-readable diagnostic for the failure. Empty when `reachable` is `true`. + +If `reachable` is `false`, no commits can succeed until the catalog is available again. Check the catalog endpoint configuration, network connectivity, and credentials. + +== Interpret commit lag + +For each Iceberg topic, `topics[]` reports a `lifecycleState` and a list of `partitions`. The `lifecycleState` is one of `ICEBERG_LIFECYCLE_STATE_LIVE`, `ICEBERG_LIFECYCLE_STATE_CLOSED`, `ICEBERG_LIFECYCLE_STATE_PURGED`, or `ICEBERG_LIFECYCLE_STATE_UNSPECIFIED`. + +Each entry in `partitions[]` reports the following offsets: + +* *`partition`*: The partition ID. +* *`lastTranslatedOffset`*: The highest Kafka offset written to Parquet. Unset if nothing has been translated yet. +* *`lastCommittedOffset`*: The last Kafka offset committed to the Iceberg catalog. Unset if nothing has been committed yet. +* *`commitLag`*: The number of records written to Parquet but not yet committed to the catalog. + +A steadily rising `commitLag` across many partitions can indicate that the cluster is underpowered for the current write rate, or that catalog commits are slow or failing. To triage, sort partitions by `commitLag` and investigate the topics with the highest lag first. + +== Commit lag compared to translation lag + +[IMPORTANT] +==== +This endpoint returns *commit lag* only: the number of records written to Parquet but not yet committed to the catalog. It does not return *translation lag* (the gap between the topic's high watermark and the last translated offset), because translation lag is a client-side calculation against the Kafka API. +==== + +To monitor both dimensions over time, use the corresponding metrics: + +* xref:reference:public-metrics-reference.adoc#redpanda_iceberg_pending_commit_lag[`redpanda_iceberg_pending_commit_lag`]: The metric equivalent of `commitLag` from this endpoint. +* xref:reference:public-metrics-reference.adoc#redpanda_iceberg_pending_translation_lag[`redpanda_iceberg_pending_translation_lag`]: Records that are pending translation to Iceberg. This is the translation lag that the endpoint does not return. + +== Next steps + +* xref:manage:iceberg/iceberg-performance-tuning.adoc[Tune Iceberg performance] if commit lag is consistently high. +* xref:manage:iceberg/iceberg-troubleshooting.adoc[Troubleshoot Iceberg topics] to diagnose translation errors and inspect the dead-letter queue. + +// end::single-source[] From dbfb942e96a5d32d27744e4c5ecfb5d907732880 Mon Sep 17 00:00:00 2001 From: Kat Batuigas Date: Wed, 22 Jul 2026 18:16:35 -0700 Subject: [PATCH 02/10] Add to What's new --- modules/get-started/pages/release-notes/redpanda.adoc | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/modules/get-started/pages/release-notes/redpanda.adoc b/modules/get-started/pages/release-notes/redpanda.adoc index 32a1fdecc0..53fda47708 100644 --- a/modules/get-started/pages/release-notes/redpanda.adoc +++ b/modules/get-started/pages/release-notes/redpanda.adoc @@ -6,7 +6,6 @@ This topic includes new content added in version {page-component-version}. For a * xref:cloud-data-platform:get-started:whats-new-cloud.adoc[] * xref:cloud-data-platform:get-started:cloud-overview.adoc#redpanda-cloud-vs-self-managed-feature-compatibility[Redpanda Cloud vs Self-Managed feature compatibility] - == Iceberg: Decoded message keys and headers You can now independently control how Redpanda translates a record's key, value, and headers into the Iceberg table using the `redpanda.iceberg.mode` topic property. A new section-based syntax lets you decode both keys and values using a schema registered in the Schema Registry, store keys or values as UTF-8 strings, and decode header values from raw bytes to strings. @@ -33,6 +32,12 @@ Redpanda version 26.2 validates a broader set of non-Java Apache Kafka® clients See xref:develop:kafka-clients.adoc[Kafka Compatibility] for the full list of validated clients and known exceptions. +== Iceberg: Health monitoring + +Redpanda now provides an Admin API endpoint to monitor the health of Iceberg topics, including external catalog reachability and per-partition commit lag. Use it to detect an unreachable catalog and identify the partitions with the highest commit lag. + +See xref:manage:iceberg/monitor-iceberg-health.adoc[] for details. + == New configuration properties **Iceberg:** From e85e0444fa5e089df138b3dd0200e6ceb62a6295 Mon Sep 17 00:00:00 2001 From: Kat Batuigas Date: Wed, 22 Jul 2026 18:16:44 -0700 Subject: [PATCH 03/10] Update links in Use Admin API --- modules/manage/pages/use-admin-api.adoc | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/modules/manage/pages/use-admin-api.adoc b/modules/manage/pages/use-admin-api.adoc index a3b6717faf..ce844732ef 100644 --- a/modules/manage/pages/use-admin-api.adoc +++ b/modules/manage/pages/use-admin-api.adoc @@ -81,11 +81,11 @@ The new endpoints differ from the legacy endpoints in the following ways: * URL paths use the fully-qualified names of the ConnectRPC services. * ConnectRPC endpoints accept only POST requests. -Use ConnectRPC endpoints with features introduced in v25.3 such as: +Use ConnectRPC endpoints with features such as: -// TODO: Add links to docs when they are merged -* Shadowing -* Connected client monitoring +* xref:manage:disaster-recovery/shadowing/index.adoc[Shadowing] +* xref:manage:cluster-maintenance/manage-throughput.adoc#view-connected-client-details[Connected client monitoring] +* xref:manage:iceberg/monitor-iceberg-health.adoc[Iceberg health monitoring] For a full list of available endpoints, see the link:/api/doc/admin/v2/[Admin API Reference]. Select "v2" in the version selector to view the ConnectRPC endpoints. From 5a8be8c1481b82ca4f488a172855be0708ef41c2 Mon Sep 17 00:00:00 2001 From: Kat Batuigas Date: Mon, 27 Jul 2026 11:55:38 -0700 Subject: [PATCH 04/10] Add only brief mention of API endpoint per PM feedback --- modules/ROOT/nav.adoc | 1 - .../pages/iceberg/about-iceberg-topics.adoc | 6 + .../pages/iceberg/monitor-iceberg-health.adoc | 128 ------------------ modules/manage/pages/use-admin-api.adoc | 2 +- 4 files changed, 7 insertions(+), 130 deletions(-) delete mode 100644 modules/manage/pages/iceberg/monitor-iceberg-health.adoc diff --git a/modules/ROOT/nav.adoc b/modules/ROOT/nav.adoc index 7a47c49106..97e56aa5fd 100644 --- a/modules/ROOT/nav.adoc +++ b/modules/ROOT/nav.adoc @@ -211,7 +211,6 @@ *** xref:manage:iceberg/query-iceberg-topics.adoc[Query Iceberg Topics] *** xref:manage:iceberg/migrate-iceberg-catalog.adoc[Migrate Iceberg Catalogs] *** xref:manage:iceberg/iceberg-performance-tuning.adoc[Tune Iceberg Performance] -*** xref:manage:iceberg/monitor-iceberg-health.adoc[] *** xref:manage:iceberg/iceberg-troubleshooting.adoc[Troubleshoot Iceberg Topics] ** xref:manage:schema-reg/index.adoc[Schema Registry] *** xref:manage:schema-reg/schema-reg-overview.adoc[Overview] diff --git a/modules/manage/pages/iceberg/about-iceberg-topics.adoc b/modules/manage/pages/iceberg/about-iceberg-topics.adoc index 22c1c5afab..814a4dad4d 100644 --- a/modules/manage/pages/iceberg/about-iceberg-topics.adoc +++ b/modules/manage/pages/iceberg/about-iceberg-topics.adoc @@ -316,6 +316,12 @@ Querying the Iceberg table for `demo-topic` includes the new column `ts`: +---------+--------------+--------------------------+ ---- +ifndef::env-cloud[] +== Monitor Iceberg health + +To check whether your external Iceberg catalog is reachable and how far behind catalog commit is for each Iceberg topic and partition, use the read-only `GetIcebergStatus` Admin API endpoint (Redpanda version 26.2 and later). Send a POST request to link:/api/doc/admin/v2/operation/operation-redpanda-core-admin-v2-icebergservice-geticebergstatus[`redpanda.core.admin.v2.IcebergService/GetIcebergStatus`]; see the link:/api/doc/admin/v2/[Admin API reference] for the full request and response schema. +endif::[] + == Next steps * xref:manage:iceberg/use-iceberg-catalogs.adoc[] diff --git a/modules/manage/pages/iceberg/monitor-iceberg-health.adoc b/modules/manage/pages/iceberg/monitor-iceberg-health.adoc deleted file mode 100644 index 7b7fae2b59..0000000000 --- a/modules/manage/pages/iceberg/monitor-iceberg-health.adoc +++ /dev/null @@ -1,128 +0,0 @@ -= Monitor Iceberg Health -:description: Monitor the health of Iceberg topics using the Admin API to check external catalog reachability and per-partition commit lag. -:page-categories: Iceberg, Monitoring - -// tag::single-source[] -:page-topic-type: how-to -:personas: ops_admin, streaming_developer -:learning-objective-1: Check whether the external Iceberg catalog is reachable -:learning-objective-2: Identify Iceberg topics with the highest commit lag - -ifndef::env-cloud[] -[NOTE] -==== -include::shared:partial$enterprise-license.adoc[] -==== -endif::[] - -{description} - -After reading this page, you will be able to: - -* [ ] {learning-objective-1} -* [ ] {learning-objective-2} - -Use the `GetIcebergStatus` Admin API endpoint to check whether your external Iceberg catalog is reachable and to see how far behind catalog commit is for each Iceberg topic and partition. This helps you detect a misconfigured or unreachable catalog and identify the partitions with the highest commit lag on a busy cluster. - -NOTE: This endpoint is available in Redpanda version 26.2 and later. - -[IMPORTANT] -==== -This endpoint requires superuser privileges, even though it is read-only. See xref:manage:use-admin-api.adoc#authentication[Admin API authentication]. -==== - -== Check Iceberg health - -Send a POST request to the link:/api/doc/admin/v2/operation/operation-redpanda-core-admin-v2-icebergservice-geticebergstatus[`redpanda.core.admin.v2.IcebergService/GetIcebergStatus`] endpoint. With an empty request body, the response covers all Iceberg topics that Redpanda is currently tracking for translation: - -[,bash] ----- -curl \ - -u : \ - --request POST 'http://:/redpanda.core.admin.v2.IcebergService/GetIcebergStatus' \ - --header "Content-Type: application/json" \ - --data '{}' ----- - -To report on specific topics only, set `topicsFilter` to a list of Iceberg topic names: - -[,bash] ----- -curl \ - -u : \ - --request POST 'http://:/redpanda.core.admin.v2.IcebergService/GetIcebergStatus' \ - --header "Content-Type: application/json" \ - --data '{ "topicsFilter": ["orders", "clicks"] }' ----- - -* Request headers `Connect-Protocol-Version` and `Connect-Timeout-Ms` are optional. -* v2 endpoints also accept binary-encoded Protobuf request bodies. Use the `Content-Type: application/proto` header. - -A response looks like the following: - -[,json] ----- -{ - "catalog": { - "reachable": true, - "errorCode": "", - "errorMessage": "" - }, - "topics": [ - { - "topic": "orders", - "lifecycleState": "ICEBERG_LIFECYCLE_STATE_LIVE", - "partitions": [ - { - "partition": 0, - "lastTranslatedOffset": "10432", - "lastCommittedOffset": "10400", - "commitLag": "32" - } - ] - } - ] -} ----- - -== Interpret catalog reachability - -The `catalog` object reports whether Redpanda can reach the external Iceberg catalog: - -* *`reachable`*: `true` if the catalog responded, `false` otherwise. -* *`errorCode`*: When the catalog is unreachable, a short error identifier from the catalog error set, such as `io_error` or `timedout`. Empty when `reachable` is `true`. -* *`errorMessage`*: A human-readable diagnostic for the failure. Empty when `reachable` is `true`. - -If `reachable` is `false`, no commits can succeed until the catalog is available again. Check the catalog endpoint configuration, network connectivity, and credentials. - -== Interpret commit lag - -For each Iceberg topic, `topics[]` reports a `lifecycleState` and a list of `partitions`. The `lifecycleState` is one of `ICEBERG_LIFECYCLE_STATE_LIVE`, `ICEBERG_LIFECYCLE_STATE_CLOSED`, `ICEBERG_LIFECYCLE_STATE_PURGED`, or `ICEBERG_LIFECYCLE_STATE_UNSPECIFIED`. - -Each entry in `partitions[]` reports the following offsets: - -* *`partition`*: The partition ID. -* *`lastTranslatedOffset`*: The highest Kafka offset written to Parquet. Unset if nothing has been translated yet. -* *`lastCommittedOffset`*: The last Kafka offset committed to the Iceberg catalog. Unset if nothing has been committed yet. -* *`commitLag`*: The number of records written to Parquet but not yet committed to the catalog. - -A steadily rising `commitLag` across many partitions can indicate that the cluster is underpowered for the current write rate, or that catalog commits are slow or failing. To triage, sort partitions by `commitLag` and investigate the topics with the highest lag first. - -== Commit lag compared to translation lag - -[IMPORTANT] -==== -This endpoint returns *commit lag* only: the number of records written to Parquet but not yet committed to the catalog. It does not return *translation lag* (the gap between the topic's high watermark and the last translated offset), because translation lag is a client-side calculation against the Kafka API. -==== - -To monitor both dimensions over time, use the corresponding metrics: - -* xref:reference:public-metrics-reference.adoc#redpanda_iceberg_pending_commit_lag[`redpanda_iceberg_pending_commit_lag`]: The metric equivalent of `commitLag` from this endpoint. -* xref:reference:public-metrics-reference.adoc#redpanda_iceberg_pending_translation_lag[`redpanda_iceberg_pending_translation_lag`]: Records that are pending translation to Iceberg. This is the translation lag that the endpoint does not return. - -== Next steps - -* xref:manage:iceberg/iceberg-performance-tuning.adoc[Tune Iceberg performance] if commit lag is consistently high. -* xref:manage:iceberg/iceberg-troubleshooting.adoc[Troubleshoot Iceberg topics] to diagnose translation errors and inspect the dead-letter queue. - -// end::single-source[] diff --git a/modules/manage/pages/use-admin-api.adoc b/modules/manage/pages/use-admin-api.adoc index ce844732ef..15eb523481 100644 --- a/modules/manage/pages/use-admin-api.adoc +++ b/modules/manage/pages/use-admin-api.adoc @@ -85,7 +85,7 @@ Use ConnectRPC endpoints with features such as: * xref:manage:disaster-recovery/shadowing/index.adoc[Shadowing] * xref:manage:cluster-maintenance/manage-throughput.adoc#view-connected-client-details[Connected client monitoring] -* xref:manage:iceberg/monitor-iceberg-health.adoc[Iceberg health monitoring] +* xref:manage:iceberg/about-iceberg-topics.adoc#monitor-iceberg-health[Iceberg health monitoring] For a full list of available endpoints, see the link:/api/doc/admin/v2/[Admin API Reference]. Select "v2" in the version selector to view the ConnectRPC endpoints. From a3e6bf2b6a70d0053eef85ff44a1e8fc5b595b83 Mon Sep 17 00:00:00 2001 From: Kat Batuigas Date: Mon, 27 Jul 2026 13:08:34 -0700 Subject: [PATCH 05/10] Mention in troubleshooting doc --- modules/manage/pages/iceberg/iceberg-troubleshooting.adoc | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/modules/manage/pages/iceberg/iceberg-troubleshooting.adoc b/modules/manage/pages/iceberg/iceberg-troubleshooting.adoc index 31234de327..c9f9631365 100644 --- a/modules/manage/pages/iceberg/iceberg-troubleshooting.adoc +++ b/modules/manage/pages/iceberg/iceberg-troubleshooting.adoc @@ -118,4 +118,8 @@ The following xref:reference:public-metrics-reference.adoc#iceberg-metrics[Icebe * xref:reference:public-metrics-reference.adoc#redpanda_iceberg_translation_invalid_records[`redpanda_iceberg_translation_invalid_records`]: Number of invalid records encountered during translation, labeled by cause. See <> to configure how Redpanda handles these records. * xref:reference:public-metrics-reference.adoc#redpanda_iceberg_rest_client_num_commit_table_update_requests_failed[`redpanda_iceberg_rest_client_num_commit_table_update_requests_failed`]: Failed table commit requests to the REST catalog. Applies only when using a REST catalog (`iceberg_catalog_type: rest`). Persistent failures indicate catalog connectivity or permission issues. +ifndef::env-cloud[] +For an on-demand check of catalog reachability and per-partition commit lag, use the `GetIcebergStatus` Admin API endpoint. See xref:manage:iceberg/about-iceberg-topics.adoc#monitor-iceberg-health[Monitor Iceberg health]. +endif::[] + // end::single-source[] From 2dc7ccb41cc54d5fdd3ec0eaa9aa563e8f497a17 Mon Sep 17 00:00:00 2001 From: Kat Batuigas Date: Mon, 27 Jul 2026 18:22:00 -0700 Subject: [PATCH 06/10] Reflect GetIcebergStatus catalog connectivity per SME review MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Per PM review on PR #1822: GetIcebergStatus does more than a reachability ping — it opens the external REST catalog connection (logs in with credentials). Reword the three references from "reachability" to "connectivity"/"connection" to reflect what the endpoint actually tests. Co-Authored-By: Claude Opus 4.8 (1M context) --- modules/get-started/pages/release-notes/redpanda.adoc | 2 +- modules/manage/pages/iceberg/about-iceberg-topics.adoc | 2 +- modules/manage/pages/iceberg/iceberg-troubleshooting.adoc | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/modules/get-started/pages/release-notes/redpanda.adoc b/modules/get-started/pages/release-notes/redpanda.adoc index 53fda47708..42c434cf6d 100644 --- a/modules/get-started/pages/release-notes/redpanda.adoc +++ b/modules/get-started/pages/release-notes/redpanda.adoc @@ -34,7 +34,7 @@ See xref:develop:kafka-clients.adoc[Kafka Compatibility] for the full list of va == Iceberg: Health monitoring -Redpanda now provides an Admin API endpoint to monitor the health of Iceberg topics, including external catalog reachability and per-partition commit lag. Use it to detect an unreachable catalog and identify the partitions with the highest commit lag. +Redpanda now provides an Admin API endpoint to monitor the health of Iceberg topics, including connectivity to the external REST catalog and per-partition commit lag. Use it to test the catalog connection and identify the partitions with the highest commit lag. See xref:manage:iceberg/monitor-iceberg-health.adoc[] for details. diff --git a/modules/manage/pages/iceberg/about-iceberg-topics.adoc b/modules/manage/pages/iceberg/about-iceberg-topics.adoc index 814a4dad4d..c51f89d10e 100644 --- a/modules/manage/pages/iceberg/about-iceberg-topics.adoc +++ b/modules/manage/pages/iceberg/about-iceberg-topics.adoc @@ -319,7 +319,7 @@ Querying the Iceberg table for `demo-topic` includes the new column `ts`: ifndef::env-cloud[] == Monitor Iceberg health -To check whether your external Iceberg catalog is reachable and how far behind catalog commit is for each Iceberg topic and partition, use the read-only `GetIcebergStatus` Admin API endpoint (Redpanda version 26.2 and later). Send a POST request to link:/api/doc/admin/v2/operation/operation-redpanda-core-admin-v2-icebergservice-geticebergstatus[`redpanda.core.admin.v2.IcebergService/GetIcebergStatus`]; see the link:/api/doc/admin/v2/[Admin API reference] for the full request and response schema. +To test connectivity with your external Iceberg REST catalog and check how far behind catalog commit is for each Iceberg topic and partition, use the read-only `GetIcebergStatus` Admin API endpoint (Redpanda version 26.2 and later). Send a POST request to link:/api/doc/admin/v2/operation/operation-redpanda-core-admin-v2-icebergservice-geticebergstatus[`redpanda.core.admin.v2.IcebergService/GetIcebergStatus`]; see the link:/api/doc/admin/v2/[Admin API reference] for the full request and response schema. endif::[] == Next steps diff --git a/modules/manage/pages/iceberg/iceberg-troubleshooting.adoc b/modules/manage/pages/iceberg/iceberg-troubleshooting.adoc index c9f9631365..90403dd62e 100644 --- a/modules/manage/pages/iceberg/iceberg-troubleshooting.adoc +++ b/modules/manage/pages/iceberg/iceberg-troubleshooting.adoc @@ -119,7 +119,7 @@ The following xref:reference:public-metrics-reference.adoc#iceberg-metrics[Icebe * xref:reference:public-metrics-reference.adoc#redpanda_iceberg_rest_client_num_commit_table_update_requests_failed[`redpanda_iceberg_rest_client_num_commit_table_update_requests_failed`]: Failed table commit requests to the REST catalog. Applies only when using a REST catalog (`iceberg_catalog_type: rest`). Persistent failures indicate catalog connectivity or permission issues. ifndef::env-cloud[] -For an on-demand check of catalog reachability and per-partition commit lag, use the `GetIcebergStatus` Admin API endpoint. See xref:manage:iceberg/about-iceberg-topics.adoc#monitor-iceberg-health[Monitor Iceberg health]. +For an on-demand check of external catalog connectivity and per-partition commit lag, use the `GetIcebergStatus` Admin API endpoint. See xref:manage:iceberg/about-iceberg-topics.adoc#monitor-iceberg-health[Monitor Iceberg health]. endif::[] // end::single-source[] From 617794e5ab887785823ab577c58a275e5a062395 Mon Sep 17 00:00:00 2001 From: Kat Batuigas Date: Mon, 27 Jul 2026 18:30:09 -0700 Subject: [PATCH 07/10] Consolidate Iceberg 26.2 updates in What's New Group the three Iceberg entries (decoded keys/headers, Schema Registry context, health monitoring) under a single "== Iceberg" section as a bullet list instead of scattered top-level sections. Keep configuration properties as its own section. Also fixes two issues in the consolidated region: - Broken xref: health-monitoring pointed to the deleted monitor-iceberg-health.adoc; now points to about-iceberg-topics.adoc#monitor-iceberg-health - Missing blank line before the first section heading Co-Authored-By: Claude Opus 4.8 (1M context) --- .../pages/release-notes/redpanda.adoc | 21 ++++--------------- 1 file changed, 4 insertions(+), 17 deletions(-) diff --git a/modules/get-started/pages/release-notes/redpanda.adoc b/modules/get-started/pages/release-notes/redpanda.adoc index 42c434cf6d..c2efc0f5d6 100644 --- a/modules/get-started/pages/release-notes/redpanda.adoc +++ b/modules/get-started/pages/release-notes/redpanda.adoc @@ -6,19 +6,12 @@ This topic includes new content added in version {page-component-version}. For a * xref:cloud-data-platform:get-started:whats-new-cloud.adoc[] * xref:cloud-data-platform:get-started:cloud-overview.adoc#redpanda-cloud-vs-self-managed-feature-compatibility[Redpanda Cloud vs Self-Managed feature compatibility] -== Iceberg: Decoded message keys and headers -You can now independently control how Redpanda translates a record's key, value, and headers into the Iceberg table using the `redpanda.iceberg.mode` topic property. A new section-based syntax lets you decode both keys and values using a schema registered in the Schema Registry, store keys or values as UTF-8 strings, and decode header values from raw bytes to strings. +== Iceberg -The `key_value`, `value_schema_id_prefix`, and `value_schema_latest` modes continue to work and serve as shorthand for common combinations of these sections. - -See xref:manage:iceberg/specify-iceberg-schema.adoc#configure-key-value-and-header-translation[Specify Iceberg Schema] for the syntax, options, and examples. - -== Iceberg translation with Schema Registry contexts - -Iceberg-enabled topics can now resolve schemas from a specific xref:manage:schema-reg/schema-reg-contexts.adoc[Schema Registry context] instead of always using the default context. Set the new `redpanda.schema.registry.context` topic property to bind a topic's Iceberg translation to a context. Binding to a context is useful for multi-tenant or per-environment schema isolation. - -See xref:manage:iceberg/specify-iceberg-schema.adoc#resolve-schemas-within-a-context[Resolve schemas within a Schema Registry context] for configuration examples and considerations for changing a topic's context. +* *Decoded message keys and headers*: Independently control how Redpanda translates a record's key, value, and headers into the Iceberg table with a new section-based syntax for the `redpanda.iceberg.mode` topic property. You can decode keys and values using a schema registered in the Schema Registry, store keys or values as UTF-8 strings, or decode header values from raw bytes to strings. The `key_value`, `value_schema_id_prefix`, and `value_schema_latest` modes continue to work as shorthand for common combinations. See xref:manage:iceberg/specify-iceberg-schema.adoc#configure-key-value-and-header-translation[Specify Iceberg Schema]. +* *Schema Registry context for translation*: Bind a topic's Iceberg translation to a specific xref:manage:schema-reg/schema-reg-contexts.adoc[Schema Registry context] using the new `redpanda.schema.registry.context` topic property, instead of always using the default context. This is useful for multi-tenant or per-environment schema isolation. See xref:manage:iceberg/specify-iceberg-schema.adoc#resolve-schemas-within-a-context[Resolve schemas within a Schema Registry context]. +* *Health monitoring*: Monitor the health of Iceberg topics with a new Admin API endpoint, including connectivity to the external REST catalog and per-partition commit lag. Use it to test the catalog connection and identify the partitions with the highest commit lag. See xref:manage:iceberg/about-iceberg-topics.adoc#monitor-iceberg-health[Monitor Iceberg health]. == Cluster health metrics @@ -32,12 +25,6 @@ Redpanda version 26.2 validates a broader set of non-Java Apache Kafka® clients See xref:develop:kafka-clients.adoc[Kafka Compatibility] for the full list of validated clients and known exceptions. -== Iceberg: Health monitoring - -Redpanda now provides an Admin API endpoint to monitor the health of Iceberg topics, including connectivity to the external REST catalog and per-partition commit lag. Use it to test the catalog connection and identify the partitions with the highest commit lag. - -See xref:manage:iceberg/monitor-iceberg-health.adoc[] for details. - == New configuration properties **Iceberg:** From e85185c24018029840b7d460b34af789700b8123 Mon Sep 17 00:00:00 2001 From: Kat Batuigas Date: Mon, 27 Jul 2026 18:32:13 -0700 Subject: [PATCH 08/10] Rename What's New Iceberg heading to be more descriptive "Iceberg" -> "Iceberg translation and health monitoring" to name both themes covered by the section and match the feature-phrase style of the other What's New headings. Co-Authored-By: Claude Opus 4.8 (1M context) --- modules/get-started/pages/release-notes/redpanda.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/get-started/pages/release-notes/redpanda.adoc b/modules/get-started/pages/release-notes/redpanda.adoc index c2efc0f5d6..0c2853f0aa 100644 --- a/modules/get-started/pages/release-notes/redpanda.adoc +++ b/modules/get-started/pages/release-notes/redpanda.adoc @@ -7,7 +7,7 @@ This topic includes new content added in version {page-component-version}. For a * xref:cloud-data-platform:get-started:whats-new-cloud.adoc[] * xref:cloud-data-platform:get-started:cloud-overview.adoc#redpanda-cloud-vs-self-managed-feature-compatibility[Redpanda Cloud vs Self-Managed feature compatibility] -== Iceberg +== Iceberg translation and health monitoring * *Decoded message keys and headers*: Independently control how Redpanda translates a record's key, value, and headers into the Iceberg table with a new section-based syntax for the `redpanda.iceberg.mode` topic property. You can decode keys and values using a schema registered in the Schema Registry, store keys or values as UTF-8 strings, or decode header values from raw bytes to strings. The `key_value`, `value_schema_id_prefix`, and `value_schema_latest` modes continue to work as shorthand for common combinations. See xref:manage:iceberg/specify-iceberg-schema.adoc#configure-key-value-and-header-translation[Specify Iceberg Schema]. * *Schema Registry context for translation*: Bind a topic's Iceberg translation to a specific xref:manage:schema-reg/schema-reg-contexts.adoc[Schema Registry context] using the new `redpanda.schema.registry.context` topic property, instead of always using the default context. This is useful for multi-tenant or per-environment schema isolation. See xref:manage:iceberg/specify-iceberg-schema.adoc#resolve-schemas-within-a-context[Resolve schemas within a Schema Registry context]. From cabcf15adf6b4d0fd549659a431a86c260f02dea Mon Sep 17 00:00:00 2001 From: Kat Batuigas Date: Mon, 27 Jul 2026 18:41:57 -0700 Subject: [PATCH 09/10] Minor formatting edit for consolidated Iceberg update list --- modules/get-started/pages/release-notes/redpanda.adoc | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/modules/get-started/pages/release-notes/redpanda.adoc b/modules/get-started/pages/release-notes/redpanda.adoc index 0c2853f0aa..38db55bd02 100644 --- a/modules/get-started/pages/release-notes/redpanda.adoc +++ b/modules/get-started/pages/release-notes/redpanda.adoc @@ -9,9 +9,9 @@ This topic includes new content added in version {page-component-version}. For a == Iceberg translation and health monitoring -* *Decoded message keys and headers*: Independently control how Redpanda translates a record's key, value, and headers into the Iceberg table with a new section-based syntax for the `redpanda.iceberg.mode` topic property. You can decode keys and values using a schema registered in the Schema Registry, store keys or values as UTF-8 strings, or decode header values from raw bytes to strings. The `key_value`, `value_schema_id_prefix`, and `value_schema_latest` modes continue to work as shorthand for common combinations. See xref:manage:iceberg/specify-iceberg-schema.adoc#configure-key-value-and-header-translation[Specify Iceberg Schema]. -* *Schema Registry context for translation*: Bind a topic's Iceberg translation to a specific xref:manage:schema-reg/schema-reg-contexts.adoc[Schema Registry context] using the new `redpanda.schema.registry.context` topic property, instead of always using the default context. This is useful for multi-tenant or per-environment schema isolation. See xref:manage:iceberg/specify-iceberg-schema.adoc#resolve-schemas-within-a-context[Resolve schemas within a Schema Registry context]. -* *Health monitoring*: Monitor the health of Iceberg topics with a new Admin API endpoint, including connectivity to the external REST catalog and per-partition commit lag. Use it to test the catalog connection and identify the partitions with the highest commit lag. See xref:manage:iceberg/about-iceberg-topics.adoc#monitor-iceberg-health[Monitor Iceberg health]. +* Decoded message keys and headers: Independently control how Redpanda translates a record's key, value, and headers into the Iceberg table with a new section-based syntax for the `redpanda.iceberg.mode` topic property. You can decode keys and values using a schema registered in the Schema Registry, store keys or values as UTF-8 strings, or decode header values from raw bytes to strings. The `key_value`, `value_schema_id_prefix`, and `value_schema_latest` modes continue to work as shorthand for common combinations. See xref:manage:iceberg/specify-iceberg-schema.adoc#configure-key-value-and-header-translation[Specify Iceberg Schema]. +* Schema Registry context for translation: Bind a topic's Iceberg translation to a specific xref:manage:schema-reg/schema-reg-contexts.adoc[Schema Registry context] using the new `redpanda.schema.registry.context` topic property, instead of always using the default context. This is useful for multi-tenant or per-environment schema isolation. See xref:manage:iceberg/specify-iceberg-schema.adoc#resolve-schemas-within-a-context[Resolve schemas within a Schema Registry context]. +* Health monitoring: Monitor the health of Iceberg topics with a new Admin API endpoint, including connectivity to the external REST catalog and per-partition commit lag. Use it to test the catalog connection and identify the partitions with the highest commit lag. See xref:manage:iceberg/about-iceberg-topics.adoc#monitor-iceberg-health[Monitor Iceberg health]. == Cluster health metrics From a5ca23481c2ba67b09b097c9f799b69ece17b43d Mon Sep 17 00:00:00 2001 From: Kat Batuigas Date: Mon, 27 Jul 2026 20:23:27 -0700 Subject: [PATCH 10/10] Apply suggestions from doc review --- modules/get-started/pages/release-notes/redpanda.adoc | 6 +++--- modules/manage/pages/iceberg/about-iceberg-topics.adoc | 4 +++- modules/manage/pages/iceberg/iceberg-troubleshooting.adoc | 2 +- 3 files changed, 7 insertions(+), 5 deletions(-) diff --git a/modules/get-started/pages/release-notes/redpanda.adoc b/modules/get-started/pages/release-notes/redpanda.adoc index 38db55bd02..4c27177b43 100644 --- a/modules/get-started/pages/release-notes/redpanda.adoc +++ b/modules/get-started/pages/release-notes/redpanda.adoc @@ -9,9 +9,9 @@ This topic includes new content added in version {page-component-version}. For a == Iceberg translation and health monitoring -* Decoded message keys and headers: Independently control how Redpanda translates a record's key, value, and headers into the Iceberg table with a new section-based syntax for the `redpanda.iceberg.mode` topic property. You can decode keys and values using a schema registered in the Schema Registry, store keys or values as UTF-8 strings, or decode header values from raw bytes to strings. The `key_value`, `value_schema_id_prefix`, and `value_schema_latest` modes continue to work as shorthand for common combinations. See xref:manage:iceberg/specify-iceberg-schema.adoc#configure-key-value-and-header-translation[Specify Iceberg Schema]. -* Schema Registry context for translation: Bind a topic's Iceberg translation to a specific xref:manage:schema-reg/schema-reg-contexts.adoc[Schema Registry context] using the new `redpanda.schema.registry.context` topic property, instead of always using the default context. This is useful for multi-tenant or per-environment schema isolation. See xref:manage:iceberg/specify-iceberg-schema.adoc#resolve-schemas-within-a-context[Resolve schemas within a Schema Registry context]. -* Health monitoring: Monitor the health of Iceberg topics with a new Admin API endpoint, including connectivity to the external REST catalog and per-partition commit lag. Use it to test the catalog connection and identify the partitions with the highest commit lag. See xref:manage:iceberg/about-iceberg-topics.adoc#monitor-iceberg-health[Monitor Iceberg health]. +* *Decoded message keys and headers*: Independently control how Redpanda translates a record's key, value, and headers into the Iceberg table with a new section-based syntax for the `redpanda.iceberg.mode` topic property. You can decode keys and values using a schema registered in the Schema Registry, store keys or values as UTF-8 strings, or decode header values from raw bytes to strings. The `key_value`, `value_schema_id_prefix`, and `value_schema_latest` modes continue to work as shorthand for common combinations. See xref:manage:iceberg/specify-iceberg-schema.adoc#configure-key-value-and-header-translation[Specify Iceberg Schema]. +* *Schema Registry context for translation*: Bind a topic's Iceberg translation to a specific xref:manage:schema-reg/schema-reg-contexts.adoc[Schema Registry context] using the new `redpanda.schema.registry.context` topic property, instead of always using the default context. This is useful for multi-tenant or per-environment schema isolation. See xref:manage:iceberg/specify-iceberg-schema.adoc#resolve-schemas-within-a-context[Resolve schemas within a Schema Registry context]. +* *Health monitoring*: Monitor the health of Iceberg topics with a new Admin API endpoint, including connectivity to the external REST catalog and per-partition commit lag. Use it to test the catalog connection and find the partitions with the highest commit lag. See xref:manage:iceberg/about-iceberg-topics.adoc#monitor-iceberg-health[Monitor Iceberg health]. == Cluster health metrics diff --git a/modules/manage/pages/iceberg/about-iceberg-topics.adoc b/modules/manage/pages/iceberg/about-iceberg-topics.adoc index c51f89d10e..103b45e537 100644 --- a/modules/manage/pages/iceberg/about-iceberg-topics.adoc +++ b/modules/manage/pages/iceberg/about-iceberg-topics.adoc @@ -319,7 +319,9 @@ Querying the Iceberg table for `demo-topic` includes the new column `ts`: ifndef::env-cloud[] == Monitor Iceberg health -To test connectivity with your external Iceberg REST catalog and check how far behind catalog commit is for each Iceberg topic and partition, use the read-only `GetIcebergStatus` Admin API endpoint (Redpanda version 26.2 and later). Send a POST request to link:/api/doc/admin/v2/operation/operation-redpanda-core-admin-v2-icebergservice-geticebergstatus[`redpanda.core.admin.v2.IcebergService/GetIcebergStatus`]; see the link:/api/doc/admin/v2/[Admin API reference] for the full request and response schema. +To test connectivity with your external Iceberg REST catalog and check the per-partition commit lag for each Iceberg topic, use the read-only `GetIcebergStatus` Admin API endpoint (Redpanda version 26.2 and later). Commit lag is the number of records that Redpanda has written to Parquet but not yet committed to the catalog. + +This endpoint requires superuser privileges if authentication is enabled on your cluster. See the link:/api/doc/admin/v2/operation/operation-redpanda-core-admin-v2-icebergservice-geticebergstatus[Admin API reference] for the full request and response schema. endif::[] == Next steps diff --git a/modules/manage/pages/iceberg/iceberg-troubleshooting.adoc b/modules/manage/pages/iceberg/iceberg-troubleshooting.adoc index 90403dd62e..c6681e6259 100644 --- a/modules/manage/pages/iceberg/iceberg-troubleshooting.adoc +++ b/modules/manage/pages/iceberg/iceberg-troubleshooting.adoc @@ -119,7 +119,7 @@ The following xref:reference:public-metrics-reference.adoc#iceberg-metrics[Icebe * xref:reference:public-metrics-reference.adoc#redpanda_iceberg_rest_client_num_commit_table_update_requests_failed[`redpanda_iceberg_rest_client_num_commit_table_update_requests_failed`]: Failed table commit requests to the REST catalog. Applies only when using a REST catalog (`iceberg_catalog_type: rest`). Persistent failures indicate catalog connectivity or permission issues. ifndef::env-cloud[] -For an on-demand check of external catalog connectivity and per-partition commit lag, use the `GetIcebergStatus` Admin API endpoint. See xref:manage:iceberg/about-iceberg-topics.adoc#monitor-iceberg-health[Monitor Iceberg health]. +For an on-demand check of external catalog connectivity and per-partition commit lag, use the `GetIcebergStatus` Admin API endpoint. Commit lag is the number of records written to Parquet but not yet committed to the catalog; the endpoint does not report translation lag (how far Iceberg translation trails the Kafka log). See xref:manage:iceberg/about-iceberg-topics.adoc#monitor-iceberg-health[Monitor Iceberg health]. endif::[] // end::single-source[]