[26.2] Iceberg support for decoded message keys and headers by kbatuigas · Pull Request #1804 · redpanda-data/docs

kbatuigas · 2026-07-16T18:23:51Z

Description

This pull request updates the documentation to introduce and clarify new capabilities for the redpanda.iceberg.mode topic property, allowing users to independently configure how record keys, values, and headers are translated into Iceberg tables. It also improves guidance and references for configuring these options and updates related descriptions and links.

Iceberg integration enhancements:

Added documentation and release notes for the new section-based syntax of redpanda.iceberg.mode, enabling independent configuration of key, value, and header translation, including schema-decoding and UTF-8 string options.
Updated the property description in property-overrides.json to explain the new configuration options and reference the improved documentation.

Documentation improvements:

Added learning objectives and personas to the Iceberg schema specification guide, making it clearer what users will achieve by reading the page. [1] [2]
Updated links and related topics to point to the new, more comprehensive Iceberg schema configuration guide. [1] [2]

Resolves https://redpandadata.atlassian.net/browse/2194
Review deadline: 23 July

Page previews

Specify Iceberg Schema (updated)
What's New in Redpanda (updated)

Checks

New feature
Content gap
Support Follow-up
Small fix (typos, links, copyedits, etc)

coderabbitai · 2026-07-16T18:24:40Z

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 42ba0733-3512-4a1d-8ad3-a1bf2f4709a8

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch DOC-2194-document-feature-iceberg-support-decoded-message

_{Comment @coderabbitai help to get the list of available commands.}

netlify · 2026-07-16T18:27:16Z

✅ Deploy Preview for redpanda-docs-preview ready!

Name	Link
🔨 Latest commit	`fab44ff`
🔍 Latest deploy log	https://app.netlify.com/projects/redpanda-docs-preview/deploys/6a67ff1c2c6f770008e9bd2f
😎 Deploy Preview	https://deploy-preview-1804--redpanda-docs-preview.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

kbatuigas · 2026-07-21T23:34:49Z


+== Configure key, value, and header translation
+
+For Redpanda clusters version 26.2 and later, in addition to the <<supported-iceberg-modes,supported modes>>, `redpanda.iceberg.mode` also accepts a section-based syntax that lets you independently configure how Redpanda translates the record key, value, and headers into the Iceberg table. The `key_value`, `value_schema_id_prefix`, and `value_schema_latest` modes are shorthand for common combinations of these sections.


@wdberkeley could you help confirm that this will also be live for the next Cloud release and that we don't need to conditionalize it for self-managed only? (Not sure which Cloud SME to tag)

kbatuigas · 2026-07-21T23:36:14Z

+
+|===
+
+By default (`layout=flat`), Redpanda places each decoded value field as a top-level column in the generated table, alongside the `redpanda` system struct. If a decoded value field is named `redpanda`, Redpanda moves it into the `redpanda` system struct as a `data` field, to avoid colliding with the record metadata column of the same name.


@wdberkeley I think there is a discrepancy here with the TOI, could you please confirm the behavior for the system struct?

…coded-message

mattschumpert · 2026-07-24T03:31:12Z

 === value_schema_id_prefix

-Creates an Iceberg table whose structure matches the Redpanda schema for the topic, with columns corresponding to each field. You must register a schema in the xref:manage:schema-reg/schema-reg-overview.adoc[Schema Registry] and producers must write to the topic using the Schema Registry wire format.
+Creates an Iceberg table whose structure matches the Redpanda schema for the topic, with columns corresponding to each field. You must register a schema in xref:manage:schema-reg/schema-reg-overview.adoc[Schema Registry] and producers must write to the topic using the Schema Registry wire format.


Does the 'wire format' still denote unambigously the byte prefix now that Confluent introduced the new approach @wdberkeley?

mattschumpert · 2026-07-24T03:46:00Z

+
+[IMPORTANT]
+====
+Configuring anything beyond `key:mode=binary` and `headers:value_type=binary` requires every broker in the cluster to be running Redpanda version 26.2 or later. Until all brokers are upgraded, Redpanda rejects these configurations with an error.


I feel like we could put this further down. gets in the way a bit

mattschumpert · 2026-07-24T03:49:42Z

+| `binary`
+a| `string` stores the field as a UTF-8 string, replacing invalid bytes with the Unicode replacement character (`U+FFFD`).
+
+See <<key-type-by-key-mode,Resulting `redpanda.key` type by `key` mode>> and <<value-type-by-value-mode,Resulting value field type by `value` mode>> for the resulting column types in the generated table.


Huh? This is confusing.

I think we're burying the lead here @kbatuigas .

Let's describe the power of the new section structure first and how it works, including the examples for key,value and header, and the 'layout' optiojs, before mentioning the equivalent resulting values using the 'mode' shorthands. This is really reference material that should be at the bottom , or maybe just above the 'validation rules'.

@mattschumpert I restructured the section so that the key, value, and header options are immediately followed by their examples, the modes shorthand is in its own subsection afterwards, and the all-26.2 broker requirement is under Validation rules. Does this make more sense?

wdberkeley

Looks good. Thanks for doing this... it's a lot of work, there's a lot of new stuff.

Sequential-upgrade enforcement (you can't skip more than one feature release) means the 25.3 migration warning is not relevant on the 26.2 version of the page.

micheleRP · 2026-07-24T23:52:32Z

On single-sourcing to Cloud:

The new "Configure key, value, and header translation" section sits inside the tag=single-source region on specify-iceberg-schema.adoc, so it'll flow into the Cloud page as-is. A couple of bits are self-managed-specific — the "For Redpanda clusters version 26.2 and later" line, and especially the IMPORTANT block's broker-upgrade framing ("requires every broker in the cluster to be running Redpanda version 26.2 or later. Until all brokers are upgraded…"), which isn't actionable for Cloud users since they don't manage broker upgrades. Do you want these wrapped in ifndef::env-cloud[] (or given Cloud-aware conditionals), or is that something you'll handle in the follow-up Cloud PR? This ties into your open question to wdberkeley about whether it's live for the next Cloud release — no blocker either way, just checking so it doesn't render unguarded in Cloud by accident.

micheleRP · 2026-07-24T23:54:55Z

A few more minor items — none blocking.

Optional suggestions

Learning objectives — two of the three start with "Configure" (learning-objective-1 and -3). Consider varying one, for example: "Apply independent translation for record keys, values, and headers." Style only.
Link cleanup (not broken) — a few other Iceberg pages still link to choose-iceberg-mode.adoc, which resolves fine via the page alias: about-iceberg-topics.adoc (lines 159, 226), iceberg-topics-databricks-unity.adoc:213, iceberg-topics-aws-glue.adoc:277, and query-iceberg-topics.adoc:18. Now that the property's related-topics link was repointed to specify-iceberg-schema.adoc, you may want to update these too for consistency. Optional.

Flagging for confirmation before merge (both are open threads above)

The TOI discrepancy you noted on value translation — just confirming that was reconciled with @wdberkeley.
Matt's question on whether "wire format" still unambiguously denotes the byte prefix given Confluent's newer approach — confirming that was resolved.

micheleRP

left little suggestions/reminders but looks good!

Brings the branch current with beta (which now includes #1784 and #1805). Conflict resolution in specify-iceberg-schema.adoc: the section-based "Configure key, value, and header translation" section comes first, then "Resolve schemas within a Schema Registry context" (promoted from === to == and moved after the section-based content), followed by a single "How Iceberg modes translate to table format". release-notes keeps all three 26.2 what's-new entries. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

… 26.2 in Cloud Wrap the new "== Configure key, value, and header translation" section in ifndef::env-cloud so it doesn't surface to Cloud users before 26.2 reaches Cloud (release cut 2026-07-31, available 2026-08-03). Kept as a separate ifndef block from the adjacent "Resolve schemas within a Schema Registry context" section so the two can be un-conditionalized independently once each is confirmed available in Cloud. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

- Reword learning objective 3 from "Configure" to "Apply" independent key/value/header translation - Add editor comment noting the broker-upgrade IMPORTANT block is not applicable to Redpanda Cloud (flags it for the post-26.2 Cloud de-conditionalization pass) - Align a merged-in #1784 reference to the "Schema Registry" (no article) terminology standard: "registered in the Schema Registry" -> "registered in Schema Registry" Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

kbatuigas force-pushed the DOC-2194-document-feature-iceberg-support-decoded-message branch from d77121c to 0127264 Compare July 16, 2026 19:39

kbatuigas added 7 commits July 21, 2026 16:04

Initial draft

b0107fd

Review pass

b3c80b0

DOC-2312 Also applicable for Cloud

3bba196

Formatting fixes

f2f9dfc

Iceberg mode property description override

ca6de8e

Add to What's new

3a5fc39

Update topic property reference

f2690c7

kbatuigas force-pushed the DOC-2194-document-feature-iceberg-support-decoded-message branch from 0127264 to f2690c7 Compare July 21, 2026 23:15

kbatuigas commented Jul 21, 2026

View reviewed changes

kbatuigas marked this pull request as ready for review July 21, 2026 23:36

kbatuigas requested a review from a team as a code owner July 21, 2026 23:36

kbatuigas requested a review from wdberkeley July 21, 2026 23:37

Remove feature flag

cdfd4ca

kbatuigas requested a review from mattschumpert July 23, 2026 18:30

kbatuigas and others added 2 commits July 23, 2026 11:31

Merge branch 'beta' into DOC-2194-document-feature-iceberg-support-de…

ba8b716

…coded-message

Edits for readability

0dcd99b

mattschumpert approved these changes Jul 24, 2026

View reviewed changes

wdberkeley approved these changes Jul 24, 2026

View reviewed changes

kbatuigas added 2 commits July 24, 2026 13:24

Remove 25.3 upgrade note for 26.2

c3fd34f

Sequential-upgrade enforcement (you can't skip more than one feature release) means the 25.3 migration warning is not relevant on the 26.2 version of the page.

Apply suggestions from SME review

e28df2b

micheleRP reviewed Jul 24, 2026

View reviewed changes

Comment thread modules/manage/pages/iceberg/specify-iceberg-schema.adoc

micheleRP approved these changes Jul 24, 2026

View reviewed changes

kbatuigas and others added 2 commits July 27, 2026 17:23

kbatuigas merged commit 6e11433 into beta Jul 28, 2026
9 checks passed

kbatuigas deleted the DOC-2194-document-feature-iceberg-support-decoded-message branch July 28, 2026 01:11


		== Configure key, value, and header translation

		For Redpanda clusters version 26.2 and later, in addition to the <<supported-iceberg-modes,supported modes>>, `redpanda.iceberg.mode` also accepts a section-based syntax that lets you independently configure how Redpanda translates the record key, value, and headers into the Iceberg table. The `key_value`, `value_schema_id_prefix`, and `value_schema_latest` modes are shorthand for common combinations of these sections.


		\|===

		By default (`layout=flat`), Redpanda places each decoded value field as a top-level column in the generated table, alongside the `redpanda` system struct. If a decoded value field is named `redpanda`, Redpanda moves it into the `redpanda` system struct as a `data` field, to avoid colliding with the record metadata column of the same name.

Uh oh!

Conversation

kbatuigas commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Page previews

Checks

Uh oh!

coderabbitai Bot commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Uh oh!

netlify Bot commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for redpanda-docs-preview ready!

Uh oh!

kbatuigas Jul 21, 2026

Choose a reason for hiding this comment

Uh oh!

kbatuigas Jul 21, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mattschumpert Jul 24, 2026

Choose a reason for hiding this comment

Uh oh!

mattschumpert Jul 24, 2026

Choose a reason for hiding this comment

Uh oh!

kbatuigas Jul 24, 2026

Choose a reason for hiding this comment

Uh oh!

mattschumpert Jul 24, 2026

Choose a reason for hiding this comment

Uh oh!

kbatuigas Jul 24, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

wdberkeley left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

micheleRP commented Jul 24, 2026

Uh oh!

micheleRP commented Jul 24, 2026

Uh oh!

micheleRP left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

kbatuigas commented Jul 16, 2026 •

edited

Loading

coderabbitai Bot commented Jul 16, 2026 •

edited

Loading

netlify Bot commented Jul 16, 2026 •

edited

Loading