From 92c6daf9f9f33ad847412a07c765858ea28397d3 Mon Sep 17 00:00:00 2001 From: Joyce Fee Date: Wed, 22 Jul 2026 10:01:03 -0400 Subject: [PATCH 1/9] docs: add deferred upgrade finalization page (DOC-2013) New upgrade-module page covering the 26.2 deferred-finalization feature: concept, Linux/rpk procedure, rollback, gated-features table, Admin API v2 FeaturesService, license behavior, limitations, and troubleshooting. Adds a nav entry and a pointer TIP on the Linux rolling-upgrade page. Sourced from the engineering TOI (cupboard engineering/core/toi/v26.2/2026-07-17-unfinalized-upgrades.md). Coordinated with #1816, which owns the Kubernetes procedure and the shared upgrade-limitations partial. Co-Authored-By: Claude Fable 5 --- modules/ROOT/nav.adoc | 1 + .../upgrade/pages/deferred-finalization.adoc | 241 ++++++++++++++++++ modules/upgrade/pages/rolling-upgrade.adoc | 2 + 3 files changed, 244 insertions(+) create mode 100644 modules/upgrade/pages/deferred-finalization.adoc diff --git a/modules/ROOT/nav.adoc b/modules/ROOT/nav.adoc index a413d36df3..33d2c63bc4 100644 --- a/modules/ROOT/nav.adoc +++ b/modules/ROOT/nav.adoc @@ -253,6 +253,7 @@ * xref:upgrade:index.adoc[Upgrade] ** xref:upgrade:rolling-upgrade.adoc[Upgrade Redpanda in Linux] ** xref:upgrade:k-rolling-upgrade.adoc[Upgrade Redpanda in Kubernetes] +** xref:upgrade:deferred-finalization.adoc[Defer Upgrade Finalization] ** xref:upgrade:k-upgrade-operator.adoc[Upgrade the Redpanda Operator] ** xref:upgrade:k-compatibility.adoc[] ** xref:upgrade:k-migrate-node-pools.adoc[] diff --git a/modules/upgrade/pages/deferred-finalization.adoc b/modules/upgrade/pages/deferred-finalization.adoc new file mode 100644 index 0000000000..86e805ba6e --- /dev/null +++ b/modules/upgrade/pages/deferred-finalization.adoc @@ -0,0 +1,241 @@ += Defer Upgrade Finalization +:description: Keep a feature-release upgrade open so you can soak test the new version with a guaranteed rollback path, then finalize the upgrade when you're ready. +:page-categories: Upgrades + +[NOTE] +==== +include::shared:partial$enterprise-license.adoc[] +==== + +By default, Redpanda finalizes a feature-release upgrade (for example, 26.1 to 26.2) automatically: within seconds of the last broker restarting on the new version, the cluster commits to the new release, version-gated features activate, and rollback is no longer supported. Deferred finalization gives you control over that commit step. When you disable automatic finalization before the upgrade, the cluster holds at the previous logical version after all brokers are rolled to the new binaries. New-version features stay inactive, nothing that would break a downgrade is written to disk, and you can roll the binaries back to the previous release, repeatedly if needed, until you explicitly finalize. + +Use deferred finalization when you need a soak period: run the new release under production load for hours, days, or weeks, and keep a guaranteed path back to the previous release if something looks wrong. This is especially valuable for organizations with regulatory or operational reversibility requirements for production changes. + +Deferred finalization is available in self-hosted deployments (Linux and Kubernetes). It is not available in Redpanda Cloud. + +== How upgrade finalization works + +Redpanda separates the *binary version* (the release each broker runs) from the cluster's *logical version* (the version all brokers most recently agreed on, persisted in the controller log). Features that introduce new on-disk or wire formats are gated behind the logical version: they cannot activate until the cluster's active version reaches the version that introduced them. + +A rolling upgrade moves through three phases: + +. *Mixed versions*: Some brokers are upgraded, some are not. The active version holds at the old version and new features stay gated off. This phase can last hours during a normal rolling upgrade. +. *All brokers upgraded, not yet finalized*: Every broker runs the new binary, but the cluster has not yet committed to the new version. With automatic finalization (the default), this phase lasts only seconds. +. *Finalized*: The cluster commits the new active version and the newly unlocked features activate. The active version is also the *downgrade floor*: a broker running an older binary refuses to start against finalized cluster state. + +Deferred finalization extends the second phase for as long as you choose. Nothing about the first or third phase changes; the feature adds an explicit commit step between them. + +== Prerequisites + +* A valid Enterprise Edition license. Setting `features_auto_finalization` to `false` is rejected unless a valid license (including the built-in trial license) is in effect. See <>. +* The cluster must be running at least version 26.1.9 or 25.3.15 before the upgrade, so that the xref:reference:properties/cluster-properties.adoc#features_auto_finalization[`features_auto_finalization`] property is available to set. +* The upgrade target must be 26.2 or later. The deferral logic ships in 26.2, so the first upgrade that can be held open is 26.1 to 26.2. Earlier upgrades, such as 25.3 to 26.1, finalize automatically regardless of the property setting. +* The `rpk cluster upgrade` commands and the underlying Admin API calls require superuser privileges. + +== Defer finalization for an upgrade + +. Before you upgrade, on the version you are upgrading from, disable automatic finalization: ++ +[,bash] +---- +rpk cluster config set features_auto_finalization false +---- ++ +This is a runtime change to a cluster configuration property; no restart is required. The property must be set to `false` before the last broker rolls to the new binary. Setting it after the upgrade has already finalized has no effect. + +. Perform the xref:upgrade:rolling-upgrade.adoc#perform-a-rolling-upgrade[rolling upgrade] as usual, one broker at a time. Nothing about the roll changes. + +. After every broker runs the new binary, verify that the cluster is holding the upgrade open: ++ +[,bash] +---- +rpk cluster upgrade status +---- ++ +Example output: ++ +[.no-copy] +---- +State ready to finalize +Active version 18 +Version after finalization 19 +Auto-finalization enabled false + +NODE RELEASE-VERSION LOGICAL-VERSION VERSION-KNOWN ALIVE +0 v26.2.1 19 true true +1 v26.2.1 19 true true +2 v26.2.1 19 true true +---- ++ +The `ready to finalize` state means all brokers run the new binary and the cluster is waiting for your explicit commit. The active version is the cluster's committed logical version, which is also the downgrade floor. The command supports `--format json` and `--format yaml` for automation. + +. Soak test for as long as your validation requires. The cluster serves traffic normally; only features gated on the new version are unavailable. See <>. + +When your validation passes, <>. If it doesn't, <>. + +== What is unavailable while unfinalized + +While the cluster is unfinalized, everything that worked on the previous release keeps working, including produce and consume traffic, Tiered Storage, existing shadow links, and Iceberg topics in existing modes. What you give up during the soak is the new release's version-gated features. These gates are the mechanism of downgrade safety: they prevent new-format state from being persisted while the rollback path is open. + +For 26.2, the following gated features have user-visible behavior while the upgrade is unfinalized: + +|=== +| Feature | Behavior while unfinalized + +| Cloud topics storage mode +| Creating a topic with the `tiered_v2` storage mode is rejected with an `INVALID_CONFIG` error. The storage mode becomes available automatically when you finalize. + +| Shadow link role sync +| Configuring role sync on a shadow link fails with a `FAILED_PRECONDITION` error: "Role sync cannot be configured until the upgrade is finalized." + +| Shadow link Schema Registry API sync +| Configuring Schema Registry API-mode sync on a shadow link fails with a `FAILED_PRECONDITION` error: "Schema Registry API sync mode cannot be configured until the upgrade is finalized." The pre-existing topic-mode Schema Registry sync is unaffected. See xref:manage:disaster-recovery/shadowing/migrate-schemas-confluent.adoc[]. + +| Extended Iceberg mode configurations +| The extended Iceberg mode topic configurations introduced in 26.2 are rejected until you finalize. +|=== + +Two additional 26.2 feature gates are internal-only and have no user-visible difference while unfinalized. + +If new-in-26.2 functionality returns an error that mentions finalization, the feature gate is working as designed: the upgrade is not done until you finalize. Run `rpk cluster upgrade status` to confirm the cluster state. + +[[roll-back-to-the-previous-release]] +== Roll back to the previous release + +While `rpk cluster upgrade status` reports `ready to finalize` (or while the roll is still in progress), you can roll back: reinstall the previous release's binaries and perform a rolling restart, following your normal upgrade procedure in reverse. No cluster command is needed. Just don't finalize. + +* The cluster returns to normal operation on the previous release with the same data. Data written while the upgrade was unfinalized is readable by the previous release's binaries. +* Rolling back, re-upgrading, and rolling back again is supported. +* A rollback is a full rolling restart, so the usual rolling-restart care applies: place each broker in maintenance mode and check cluster health between brokers. +* After rolling back, the `rpk cluster upgrade` commands are not available on the previous release's binaries. To check cluster version state, use the v1 Admin API instead: `GET /v1/features` reports `cluster_version` and `node_latest_version`. + +[[finalize-the-upgrade]] +== Finalize the upgrade + +Finalization is the point of no return: after the cluster commits the new version, downgrade is no longer possible. + +. Finalize the upgrade: ++ +[,bash] +---- +rpk cluster upgrade finalize +---- ++ +The command prompts for confirmation (use `--no-confirm` to skip). It validates the cluster state first and fails fast with an actionable message if finalization isn't possible, for example if auto-finalization is enabled, the upgrade is still in progress, or the upgrade is already finalized. + +. Confirm that the version advance completed. Finalization is asynchronous: a successful `finalize` means the request was accepted, not that the version has advanced. ++ +[,bash] +---- +rpk cluster upgrade status +---- ++ +When finalization completes, the state reports `finalized` and the active version shows the new logical version. + +. (Optional) Restore automatic finalization for future upgrades: ++ +[,bash] +---- +rpk cluster config set features_auto_finalization true +---- + +IMPORTANT: Do not try to finalize by setting `features_auto_finalization` back to `true` while the cluster is waiting to finalize. This does not reliably trigger the version advance. Use `rpk cluster upgrade finalize` (or the Admin API) to finalize, then optionally restore the property afterward. + +== Use the Admin API + +The `rpk cluster upgrade` commands wrap two Admin API v2 RPCs on the admin port (default 9644). You can call any broker; the request is forwarded to the controller leader. Both calls require superuser privileges, and both are recorded by audit logging (admin and management events) when auditing is enabled. + +To check upgrade status: + +[,bash] +---- +curl -s -X POST \ + http://:9644/redpanda.core.admin.v2.FeaturesService/GetUpgradeStatus \ + -H 'Content-Type: application/json' -d '{}' +---- + +The response includes: + +|=== +| Field | Description + +| `state` +| `FINALIZATION_STATE_FINALIZED` (nothing to finalize), `FINALIZATION_STATE_READY_TO_FINALIZE` (all brokers uniform at a higher version and alive; a finalize request would act), or `FINALIZATION_STATE_UPGRADE_IN_PROGRESS` (broker versions differ, or a broker's version is unknown or the broker is not alive). + +| `active_version` +| The cluster's committed logical version. This is also the downgrade floor. + +| `version_after_finalization` +| The logical version a finalize request would advance to. Exceeds `active_version` only in the `READY_TO_FINALIZE` state. + +| `auto_finalization_enabled` +| Mirrors the `features_auto_finalization` cluster property. + +| `members[]` +| Per broker: `node_id`, `logical_version`, `version_known`, `alive`, and a best-effort human-readable `release_version` (for example, `v26.2.1`). The `release_version` field may be empty; the numeric fields are authoritative. +|=== + +NOTE: Integer fields serialize as strings in the JSON response. + +To finalize the upgrade: + +[,bash] +---- +curl -s -X POST \ + http://:9644/redpanda.core.admin.v2.FeaturesService/FinalizeUpgrade \ + -H 'Content-Type: application/json' -d '{}' +---- + +The request takes no parameters: at any moment there is exactly one valid target version (or none). `FinalizeUpgrade` returns `FAILED_PRECONDITION` if `features_auto_finalization` is enabled (the RPC is only valid in manual mode), and `UNAVAILABLE` if controller leadership moved mid-request (retry the call). A success response means the request was accepted, not that the version advanced; poll `GetUpgradeStatus` to confirm. + +The v1 Admin API is unchanged and still useful: `GET /v1/features` reports `cluster_version`, `node_latest_version`, and each feature's state (`unavailable` while gated, `active` after finalization). + +== License behavior + +* License enforcement happens when you set the property: without a valid Enterprise license, setting `features_auto_finalization` to `false` is rejected, and the error names the property. +* If the license expires after the property is set, deferred finalization keeps operating. The cluster does not flip back to auto-finalizing mid-soak. +* Finalizing is not license-checked: `rpk cluster upgrade finalize` works even if the license expired during the soak. The enforcement point is setting the property, nothing else. + +== Limitations + +* *Not available in Redpanda Cloud.* Cloud manages its own upgrade lifecycle, and `rpk` refuses to run the upgrade commands against cloud clusters. +* *You cannot skip finalization and keep upgrading.* Redpanda supports upgrades from exactly one feature release back, so an unfinalized 26.2 cluster must finalize before you can begin an upgrade to 26.3. Staying unfinalized indefinitely blocks future upgrades. +* *In multi-hop upgrades, only the newest hop is held.* The property value persists across hops, but intermediate hops finalize automatically. For example, in a 25.3 to 26.1 to 26.2 upgrade path, the 25.3 to 26.1 hop auto-finalizes (26.1 has the property definition but not the deferral logic), and only the 26.1 to 26.2 hop holds. You cannot roll back to 25.3 after reaching 26.1. +* *No dedicated metric or alert.* There is no metric for "finalization pending." Monitor with `rpk cluster upgrade status --format json` or the `GetUpgradeStatus` RPC. +* *Brokers advertise the new release version during the soak.* The Admin API, health reports, `rpk redpanda admin brokers list`, and the Redpanda Console broker view all show the new release string while unfinalized; only the logical version and feature activation are held back. The one wire-visible exception: Kafka API keys gated behind held features are absent from `ApiVersions` responses until finalization. +* *`features_auto_finalization` is independent of `features_auto_enable`.* Auto-finalization controls when the cluster version advances; `features_auto_enable` is an older, unrelated property that controls whether individual features activate on their own after the version has advanced. Leave `features_auto_enable` alone. + +== Deferred finalization in Kubernetes + +Deferred finalization works in Kubernetes deployments, with one important caveat: the Redpanda Operator and Helm chart consider an upgrade complete when the pod rollout finishes and have no awareness of finalization state. Set the property through your usual cluster configuration mechanism, finalize manually with `rpk` or the Admin API after the rollout completes, and roll back by setting the previous image tag while the upgrade is unfinalized. + +For the Kubernetes procedure, see xref:upgrade:k-rolling-upgrade.adoc[]. + +== Troubleshooting + +The design principle throughout is that the cluster never advances without an explicit request: the failure mode is always "you must ask again," never "it finalized when it shouldn't have." + +Finalize appears to do nothing:: +Finalize requests are asynchronous and one-shot. The controller validates preconditions (all broker versions known, uniform at the target version, all brokers alive) when it processes the request. If a precondition fails, for example because a broker died between your status check and the request being processed, the request is consumed and the version advance is skipped. Re-issue the finalize request once the cluster is healthy, and always confirm the result with `rpk cluster upgrade status`. + +Finalize request lost after a controller leadership change:: +A pending finalize request does not survive controller leader changes or restarts. If leadership moves after you finalize but before the advance lands, re-issue the request against any broker; it is forwarded to the new leader. + +Status output looks wrong right after a leadership change:: +Immediately after a controller leadership change, the new leader rebuilds its per-broker version map. Until it has, status output can list brokers with `version_known: false` and can momentarily report a misleading state, and a finalize request issued in that window is consumed as a no-op. Treat status output as unsettled unless every broker reports `version_known: true` and `alive: true`, and re-poll before acting. + +A down broker blocks finalization:: +If a broker is down, status reports `upgrade in progress` with `alive: false` for that broker, and finalization is blocked. This is by design and matches the rule the automatic path has always had. Restore or decommission the broker first. + +`Incompatible downgrade detected` error after finalization:: +If a broker starts with a binary older than the finalized cluster version, it logs `Incompatible downgrade detected!` and aborts with a backtrace and crash report. This looks like a crash, but it is the guard rail that protects finalized cluster state. To recover, reinstall the new release's binary on that broker; the data is untouched. If you believe you need to bypass this check, contact Redpanda support. + +A fresh cluster reports `finalized`:: +New clusters bootstrap already finalized at their binary's version, so `rpk cluster upgrade status` reports `finalized` with the active version equal to the version after finalization. This is normal; the `features_auto_finalization` property only matters across upgrades. + +== Suggested reading + +* xref:upgrade:rolling-upgrade.adoc[] +* xref:upgrade:k-rolling-upgrade.adoc[] +* xref:reference:properties/cluster-properties.adoc#features_auto_finalization[`features_auto_finalization`] +* xref:get-started:licensing/overview.adoc[] diff --git a/modules/upgrade/pages/rolling-upgrade.adoc b/modules/upgrade/pages/rolling-upgrade.adoc index eff68e5ead..73f4b2006d 100644 --- a/modules/upgrade/pages/rolling-upgrade.adoc +++ b/modules/upgrade/pages/rolling-upgrade.adoc @@ -13,6 +13,8 @@ include::partial$versioning.adoc[] include::partial$rolling-upgrades/upgrade-limitations.adoc[] +TIP: By default, a feature-release upgrade finalizes automatically as soon as every broker runs the new version, and rollback is then no longer possible. Starting with upgrades to 26.2, you can defer finalization to soak test the new version while keeping a rollback path open. See xref:upgrade:deferred-finalization.adoc[]. + == Prerequisites * A running Redpanda cluster. From 6e39b351c873f2915cddd963cd54f85d60efaafe Mon Sep 17 00:00:00 2001 From: Joyce Fee Date: Wed, 22 Jul 2026 10:53:09 -0400 Subject: [PATCH 2/9] docs: reword broker failure example in finalize troubleshooting Co-Authored-By: Claude Fable 5 --- modules/upgrade/pages/deferred-finalization.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/upgrade/pages/deferred-finalization.adoc b/modules/upgrade/pages/deferred-finalization.adoc index 86e805ba6e..2fb49775ad 100644 --- a/modules/upgrade/pages/deferred-finalization.adoc +++ b/modules/upgrade/pages/deferred-finalization.adoc @@ -216,7 +216,7 @@ For the Kubernetes procedure, see xref:upgrade:k-rolling-upgrade.adoc[]. The design principle throughout is that the cluster never advances without an explicit request: the failure mode is always "you must ask again," never "it finalized when it shouldn't have." Finalize appears to do nothing:: -Finalize requests are asynchronous and one-shot. The controller validates preconditions (all broker versions known, uniform at the target version, all brokers alive) when it processes the request. If a precondition fails, for example because a broker died between your status check and the request being processed, the request is consumed and the version advance is skipped. Re-issue the finalize request once the cluster is healthy, and always confirm the result with `rpk cluster upgrade status`. +Finalize requests are asynchronous and one-shot. The controller validates preconditions (all broker versions known, uniform at the target version, all brokers alive) when it processes the request. If a precondition fails, for example because a broker went offline between your status check and the request being processed, the request is consumed and the version advance is skipped. Re-issue the finalize request once the cluster is healthy, and always confirm the result with `rpk cluster upgrade status`. Finalize request lost after a controller leadership change:: A pending finalize request does not survive controller leader changes or restarts. If leadership moves after you finalize but before the advance lands, re-issue the request against any broker; it is forwarded to the new leader. From 349b7bbae352fe39f8d15803215fb7e15517c018 Mon Sep 17 00:00:00 2001 From: Joyce Fee <102751339+Feediver1@users.noreply.github.com> Date: Wed, 22 Jul 2026 09:58:02 -0500 Subject: [PATCH 3/9] Clean up before review Co-authored-by: Joyce Fee <102751339+Feediver1@users.noreply.github.com> --- .../upgrade/pages/deferred-finalization.adoc | 26 +++++++++---------- 1 file changed, 13 insertions(+), 13 deletions(-) diff --git a/modules/upgrade/pages/deferred-finalization.adoc b/modules/upgrade/pages/deferred-finalization.adoc index 2fb49775ad..74c1df4cd6 100644 --- a/modules/upgrade/pages/deferred-finalization.adoc +++ b/modules/upgrade/pages/deferred-finalization.adoc @@ -7,7 +7,7 @@ include::shared:partial$enterprise-license.adoc[] ==== -By default, Redpanda finalizes a feature-release upgrade (for example, 26.1 to 26.2) automatically: within seconds of the last broker restarting on the new version, the cluster commits to the new release, version-gated features activate, and rollback is no longer supported. Deferred finalization gives you control over that commit step. When you disable automatic finalization before the upgrade, the cluster holds at the previous logical version after all brokers are rolled to the new binaries. New-version features stay inactive, nothing that would break a downgrade is written to disk, and you can roll the binaries back to the previous release, repeatedly if needed, until you explicitly finalize. +By default, Redpanda finalizes a feature-release upgrade (for example, v26.1 to v26.2) automatically: within seconds of the last broker restarting on the new version, the cluster commits to the new release, version-gated features activate, and rollback is no longer supported. Deferred finalization gives you control over that cluster commit step. When you disable automatic finalization before the upgrade, the cluster holds at the previous logical version after all brokers are rolled to the new binaries. New-version features stay inactive, nothing that would break a downgrade is written to disk, and you can roll the binaries back to the previous release, repeatedly if needed, until you explicitly finalize. Use deferred finalization when you need a soak period: run the new release under production load for hours, days, or weeks, and keep a guaranteed path back to the previous release if something looks wrong. This is especially valuable for organizations with regulatory or operational reversibility requirements for production changes. @@ -21,15 +21,15 @@ A rolling upgrade moves through three phases: . *Mixed versions*: Some brokers are upgraded, some are not. The active version holds at the old version and new features stay gated off. This phase can last hours during a normal rolling upgrade. . *All brokers upgraded, not yet finalized*: Every broker runs the new binary, but the cluster has not yet committed to the new version. With automatic finalization (the default), this phase lasts only seconds. -. *Finalized*: The cluster commits the new active version and the newly unlocked features activate. The active version is also the *downgrade floor*: a broker running an older binary refuses to start against finalized cluster state. +. *Finalized*: The cluster commits the new active version and the newly unlocked features activate. The active version is also the *downgrade floor*: a broker running an older binary that refuses to start against finalized cluster state. Deferred finalization extends the second phase for as long as you choose. Nothing about the first or third phase changes; the feature adds an explicit commit step between them. == Prerequisites * A valid Enterprise Edition license. Setting `features_auto_finalization` to `false` is rejected unless a valid license (including the built-in trial license) is in effect. See <>. -* The cluster must be running at least version 26.1.9 or 25.3.15 before the upgrade, so that the xref:reference:properties/cluster-properties.adoc#features_auto_finalization[`features_auto_finalization`] property is available to set. -* The upgrade target must be 26.2 or later. The deferral logic ships in 26.2, so the first upgrade that can be held open is 26.1 to 26.2. Earlier upgrades, such as 25.3 to 26.1, finalize automatically regardless of the property setting. +* The cluster must be running at least v26.1.9 or v25.3.15 before the upgrade, so that the xref:reference:properties/cluster-properties.adoc#features_auto_finalization[`features_auto_finalization`] property is available to set. +* The upgrade target must be v26.2 or later. The deferral logic ships in v26.2, so the first upgrade that can be held open is v26.1 to v26.2. Earlier upgrades, such as v25.3 to v26.1, finalize automatically regardless of the property setting. * The `rpk cluster upgrade` commands and the underlying Admin API calls require superuser privileges. == Defer finalization for an upgrade @@ -77,12 +77,12 @@ When your validation passes, <>. If i While the cluster is unfinalized, everything that worked on the previous release keeps working, including produce and consume traffic, Tiered Storage, existing shadow links, and Iceberg topics in existing modes. What you give up during the soak is the new release's version-gated features. These gates are the mechanism of downgrade safety: they prevent new-format state from being persisted while the rollback path is open. -For 26.2, the following gated features have user-visible behavior while the upgrade is unfinalized: +For v26.2, the following gated features have user-visible behavior while the upgrade is unfinalized: |=== | Feature | Behavior while unfinalized -| Cloud topics storage mode +| Cloud Topics storage mode | Creating a topic with the `tiered_v2` storage mode is rejected with an `INVALID_CONFIG` error. The storage mode becomes available automatically when you finalize. | Shadow link role sync @@ -92,12 +92,12 @@ For 26.2, the following gated features have user-visible behavior while the upgr | Configuring Schema Registry API-mode sync on a shadow link fails with a `FAILED_PRECONDITION` error: "Schema Registry API sync mode cannot be configured until the upgrade is finalized." The pre-existing topic-mode Schema Registry sync is unaffected. See xref:manage:disaster-recovery/shadowing/migrate-schemas-confluent.adoc[]. | Extended Iceberg mode configurations -| The extended Iceberg mode topic configurations introduced in 26.2 are rejected until you finalize. +| The extended Iceberg mode topic configurations introduced in v26.2 are rejected until you finalize. |=== -Two additional 26.2 feature gates are internal-only and have no user-visible difference while unfinalized. +Two additional v26.2 feature gates are internal-only and have no user-visible difference while unfinalized. -If new-in-26.2 functionality returns an error that mentions finalization, the feature gate is working as designed: the upgrade is not done until you finalize. Run `rpk cluster upgrade status` to confirm the cluster state. +If new-in-v26.2 functionality returns an error that mentions finalization, the feature gate is working as designed: the upgrade is not done until you finalize. Run `rpk cluster upgrade status` to confirm the cluster state. [[roll-back-to-the-previous-release]] == Roll back to the previous release @@ -199,8 +199,8 @@ The v1 Admin API is unchanged and still useful: `GET /v1/features` reports `clus == Limitations * *Not available in Redpanda Cloud.* Cloud manages its own upgrade lifecycle, and `rpk` refuses to run the upgrade commands against cloud clusters. -* *You cannot skip finalization and keep upgrading.* Redpanda supports upgrades from exactly one feature release back, so an unfinalized 26.2 cluster must finalize before you can begin an upgrade to 26.3. Staying unfinalized indefinitely blocks future upgrades. -* *In multi-hop upgrades, only the newest hop is held.* The property value persists across hops, but intermediate hops finalize automatically. For example, in a 25.3 to 26.1 to 26.2 upgrade path, the 25.3 to 26.1 hop auto-finalizes (26.1 has the property definition but not the deferral logic), and only the 26.1 to 26.2 hop holds. You cannot roll back to 25.3 after reaching 26.1. +* *You cannot skip finalization and keep upgrading.* Redpanda supports upgrades from exactly one feature release back, so an unfinalized v26.2 cluster must finalize before you can begin an upgrade to v26.3. Staying unfinalized indefinitely blocks future upgrades. +* *In multi-hop upgrades, only the newest hop is held.* The property value persists across hops, but intermediate hops finalize automatically. For example, in a v25.3 to v26.1 to v26.2 upgrade path, the v25.3 to v26.1 hop auto-finalizes (v26.1 has the property definition but not the deferral logic), and only the v26.1 to v26.2 hop holds. You cannot roll back to v25.3 after reaching v26.1. * *No dedicated metric or alert.* There is no metric for "finalization pending." Monitor with `rpk cluster upgrade status --format json` or the `GetUpgradeStatus` RPC. * *Brokers advertise the new release version during the soak.* The Admin API, health reports, `rpk redpanda admin brokers list`, and the Redpanda Console broker view all show the new release string while unfinalized; only the logical version and feature activation are held back. The one wire-visible exception: Kafka API keys gated behind held features are absent from `ApiVersions` responses until finalization. * *`features_auto_finalization` is independent of `features_auto_enable`.* Auto-finalization controls when the cluster version advances; `features_auto_enable` is an older, unrelated property that controls whether individual features activate on their own after the version has advanced. Leave `features_auto_enable` alone. @@ -216,7 +216,7 @@ For the Kubernetes procedure, see xref:upgrade:k-rolling-upgrade.adoc[]. The design principle throughout is that the cluster never advances without an explicit request: the failure mode is always "you must ask again," never "it finalized when it shouldn't have." Finalize appears to do nothing:: -Finalize requests are asynchronous and one-shot. The controller validates preconditions (all broker versions known, uniform at the target version, all brokers alive) when it processes the request. If a precondition fails, for example because a broker went offline between your status check and the request being processed, the request is consumed and the version advance is skipped. Re-issue the finalize request once the cluster is healthy, and always confirm the result with `rpk cluster upgrade status`. +Finalize requests are asynchronous and one-shot. The controller validates preconditions (all broker versions known, uniform at the target version, all brokers online) when it processes the request. If a precondition fails, for example because a broker went offline between your status check and the request being processed, the request is consumed and the version advance is skipped. Re-issue the finalize request once the cluster is healthy, and always confirm the result with `rpk cluster upgrade status`. Finalize request lost after a controller leadership change:: A pending finalize request does not survive controller leader changes or restarts. If leadership moves after you finalize but before the advance lands, re-issue the request against any broker; it is forwarded to the new leader. @@ -228,7 +228,7 @@ A down broker blocks finalization:: If a broker is down, status reports `upgrade in progress` with `alive: false` for that broker, and finalization is blocked. This is by design and matches the rule the automatic path has always had. Restore or decommission the broker first. `Incompatible downgrade detected` error after finalization:: -If a broker starts with a binary older than the finalized cluster version, it logs `Incompatible downgrade detected!` and aborts with a backtrace and crash report. This looks like a crash, but it is the guard rail that protects finalized cluster state. To recover, reinstall the new release's binary on that broker; the data is untouched. If you believe you need to bypass this check, contact Redpanda support. +If a broker starts with a binary older than the finalized cluster version, it logs `Incompatible downgrade detected!` and aborts with a backtrace and crash report. This looks like a crash, but it is the guard rail that protects finalized cluster state. To recover, reinstall the new release's binary on that broker; the data is untouched. If you must bypass this check, contact Redpanda support. A fresh cluster reports `finalized`:: New clusters bootstrap already finalized at their binary's version, so `rpk cluster upgrade status` reports `finalized` with the active version equal to the version after finalization. This is normal; the `features_auto_finalization` property only matters across upgrades. From 5dd3bdbbf7fe646b7afcd45bf01a7f5bbf761174 Mon Sep 17 00:00:00 2001 From: Joyce Fee Date: Wed, 22 Jul 2026 11:16:52 -0400 Subject: [PATCH 4/9] docs: add License behavior and Limitations intros from review; v-prefix in rolling-upgrade TIP Co-Authored-By: Claude Fable 5 --- modules/upgrade/pages/deferred-finalization.adoc | 4 ++++ modules/upgrade/pages/rolling-upgrade.adoc | 2 +- 2 files changed, 5 insertions(+), 1 deletion(-) diff --git a/modules/upgrade/pages/deferred-finalization.adoc b/modules/upgrade/pages/deferred-finalization.adoc index 74c1df4cd6..7ced5e8532 100644 --- a/modules/upgrade/pages/deferred-finalization.adoc +++ b/modules/upgrade/pages/deferred-finalization.adoc @@ -192,12 +192,16 @@ The v1 Admin API is unchanged and still useful: `GET /v1/features` reports `clus == License behavior +As noted in Prerequisites, a valid Enterprise license is required to specify deferred finalization. The following behaviors occur if you attempt to use deferred finalization without a license. + * License enforcement happens when you set the property: without a valid Enterprise license, setting `features_auto_finalization` to `false` is rejected, and the error names the property. * If the license expires after the property is set, deferred finalization keeps operating. The cluster does not flip back to auto-finalizing mid-soak. * Finalizing is not license-checked: `rpk cluster upgrade finalize` works even if the license expired during the soak. The enforcement point is setting the property, nothing else. == Limitations +Before starting, familiarize yourself with the limitations of the deferred finalization feature: + * *Not available in Redpanda Cloud.* Cloud manages its own upgrade lifecycle, and `rpk` refuses to run the upgrade commands against cloud clusters. * *You cannot skip finalization and keep upgrading.* Redpanda supports upgrades from exactly one feature release back, so an unfinalized v26.2 cluster must finalize before you can begin an upgrade to v26.3. Staying unfinalized indefinitely blocks future upgrades. * *In multi-hop upgrades, only the newest hop is held.* The property value persists across hops, but intermediate hops finalize automatically. For example, in a v25.3 to v26.1 to v26.2 upgrade path, the v25.3 to v26.1 hop auto-finalizes (v26.1 has the property definition but not the deferral logic), and only the v26.1 to v26.2 hop holds. You cannot roll back to v25.3 after reaching v26.1. diff --git a/modules/upgrade/pages/rolling-upgrade.adoc b/modules/upgrade/pages/rolling-upgrade.adoc index 73f4b2006d..b5bd791f6a 100644 --- a/modules/upgrade/pages/rolling-upgrade.adoc +++ b/modules/upgrade/pages/rolling-upgrade.adoc @@ -13,7 +13,7 @@ include::partial$versioning.adoc[] include::partial$rolling-upgrades/upgrade-limitations.adoc[] -TIP: By default, a feature-release upgrade finalizes automatically as soon as every broker runs the new version, and rollback is then no longer possible. Starting with upgrades to 26.2, you can defer finalization to soak test the new version while keeping a rollback path open. See xref:upgrade:deferred-finalization.adoc[]. +TIP: By default, a feature-release upgrade finalizes automatically as soon as every broker runs the new version, and rollback is then no longer possible. Starting with upgrades to v26.2, you can defer finalization to soak test the new version while keeping a rollback path open. See xref:upgrade:deferred-finalization.adoc[]. == Prerequisites From a4ee5cd79a0316384251cdea3c018890fe7ea799 Mon Sep 17 00:00:00 2001 From: Joyce Fee Date: Wed, 22 Jul 2026 11:19:12 -0400 Subject: [PATCH 5/9] docs: restore main verb in downgrade-floor sentence Co-Authored-By: Claude Fable 5 --- modules/upgrade/pages/deferred-finalization.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/upgrade/pages/deferred-finalization.adoc b/modules/upgrade/pages/deferred-finalization.adoc index 7ced5e8532..89374f457c 100644 --- a/modules/upgrade/pages/deferred-finalization.adoc +++ b/modules/upgrade/pages/deferred-finalization.adoc @@ -21,7 +21,7 @@ A rolling upgrade moves through three phases: . *Mixed versions*: Some brokers are upgraded, some are not. The active version holds at the old version and new features stay gated off. This phase can last hours during a normal rolling upgrade. . *All brokers upgraded, not yet finalized*: Every broker runs the new binary, but the cluster has not yet committed to the new version. With automatic finalization (the default), this phase lasts only seconds. -. *Finalized*: The cluster commits the new active version and the newly unlocked features activate. The active version is also the *downgrade floor*: a broker running an older binary that refuses to start against finalized cluster state. +. *Finalized*: The cluster commits the new active version and the newly unlocked features activate. The active version is also the *downgrade floor*: a broker running an older binary refuses to start against finalized cluster state. Deferred finalization extends the second phase for as long as you choose. Nothing about the first or third phase changes; the feature adds an explicit commit step between them. From 9f2bb820a2e1d998f5c0e3bd99bfd0eab765f3f8 Mon Sep 17 00:00:00 2001 From: Joyce Fee Date: Sat, 25 Jul 2026 15:40:25 -0400 Subject: [PATCH 6/9] docs: DOC-2395: Add What's New entry for deferred upgrade finalization Includes the merge of current beta (the What's New page was reset for 26.2 after this branch was cut). The Shadowing-topology xref resolves once DOC-2217 (#1819) merges into this branch. Co-Authored-By: Claude Fable 5 --- modules/get-started/pages/release-notes/redpanda.adoc | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/modules/get-started/pages/release-notes/redpanda.adoc b/modules/get-started/pages/release-notes/redpanda.adoc index a0d27a5b3b..42dabdb8fa 100644 --- a/modules/get-started/pages/release-notes/redpanda.adoc +++ b/modules/get-started/pages/release-notes/redpanda.adoc @@ -13,3 +13,9 @@ This topic includes new content added in version {page-component-version}. For a Redpanda version 26.2 validates a broader set of non-Java Apache Kafka® clients at their current versions that support Kafka 4.x, in addition to the Apache Kafka Java client. Newly validated clients include `confluent-kafka-go` and Sarama for Go, and `kafka-python` and `confluent-kafka-python` for Python. Validation confirms connectivity and correctness across core Kafka APIs, such as produce, consume, and transaction operations. See xref:develop:kafka-clients.adoc[Kafka Compatibility] for the full list of validated clients and known exceptions. + +== Deferred upgrade finalization + +You can now keep a feature-release upgrade open, soak testing the new version with a guaranteed rollback path, and finalize the upgrade when you are ready. For clusters in a Shadowing topology, new guidance covers upgrade order: upgrade the shadow cluster before the source cluster, and use deferred finalization for bidirectional topologies. + +See xref:upgrade:deferred-finalization.adoc[Defer Upgrade Finalization] and xref:manage:disaster-recovery/shadowing/upgrade.adoc[Upgrade Clusters in a Shadowing Topology]. From 4fc22cf168110ef3b65c88da1a479c50383aca6b Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Tue, 28 Jul 2026 01:51:03 +0000 Subject: [PATCH 7/9] Merge remote-tracking branch 'origin/beta' into DOC-2013-deferred-finalization --- modules/get-started/pages/release-notes/redpanda.adoc | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/modules/get-started/pages/release-notes/redpanda.adoc b/modules/get-started/pages/release-notes/redpanda.adoc index c03035e77c..ac8a733397 100644 --- a/modules/get-started/pages/release-notes/redpanda.adoc +++ b/modules/get-started/pages/release-notes/redpanda.adoc @@ -33,13 +33,12 @@ Redpanda version 26.2 validates a broader set of non-Java Apache Kafka® clients See xref:develop:kafka-clients.adoc[Kafka Compatibility] for the full list of validated clients and known exceptions. -<<<<<<< HEAD == Deferred upgrade finalization You can now keep a feature-release upgrade open, soak testing the new version with a guaranteed rollback path, and finalize the upgrade when you are ready. For clusters in a Shadowing topology, new guidance covers upgrade order: upgrade the shadow cluster before the source cluster, and use deferred finalization for bidirectional topologies. See xref:upgrade:deferred-finalization.adoc[Defer Upgrade Finalization] and xref:manage:disaster-recovery/shadowing/upgrade.adoc[Upgrade Clusters in a Shadowing Topology]. -======= + == Tiered Storage v2 Tiered Storage is now available in two versions. Tiered Storage v2 (`tiered_v2`) adds full compaction support for topics stored in object storage: compaction runs on the data in object storage rather than on local data only, eventually retains only the latest value for each key across the entire partition, and removes tombstone records after the retention period configured by `delete.retention.ms`. Topics using Tiered Storage v2 can also be converted to and from Cloud Topics. @@ -53,4 +52,3 @@ See xref:manage:tiered-storage.adoc#tiered-storage-versions[Choose a Tiered Stor **Iceberg:** * xref:reference:properties/topic-properties.adoc#redpanda-schema-registry-context[`redpanda.schema.registry.context`]: Bind a topic's Iceberg translation to a specific Schema Registry context ->>>>>>> origin/beta From 878a7f3773d4d9b8ce1b30d6e6d75d5c062afd9c Mon Sep 17 00:00:00 2001 From: Joyce Fee Date: Mon, 27 Jul 2026 22:04:01 -0400 Subject: [PATCH 8/9] docs: fix review findings on deferred finalization - Replace rpk redpanda admin brokers list (deprecated in 26.2, swept in #1832) with rpk cluster info -b --detailed in the Limitations bullet. - Move the Shadowing-topology sentence and xref out of the What's New entry: the target page ships in #1819, so this PR is now self-contained and can merge independently. #1819 re-adds the sentence alongside its page. Co-Authored-By: Claude Fable 5 --- modules/get-started/pages/release-notes/redpanda.adoc | 4 ++-- modules/upgrade/pages/deferred-finalization.adoc | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/modules/get-started/pages/release-notes/redpanda.adoc b/modules/get-started/pages/release-notes/redpanda.adoc index ac8a733397..87c62bf6b9 100644 --- a/modules/get-started/pages/release-notes/redpanda.adoc +++ b/modules/get-started/pages/release-notes/redpanda.adoc @@ -35,9 +35,9 @@ See xref:develop:kafka-clients.adoc[Kafka Compatibility] for the full list of va == Deferred upgrade finalization -You can now keep a feature-release upgrade open, soak testing the new version with a guaranteed rollback path, and finalize the upgrade when you are ready. For clusters in a Shadowing topology, new guidance covers upgrade order: upgrade the shadow cluster before the source cluster, and use deferred finalization for bidirectional topologies. +You can now keep a feature-release upgrade open, soak testing the new version with a guaranteed rollback path, and finalize the upgrade when you are ready. -See xref:upgrade:deferred-finalization.adoc[Defer Upgrade Finalization] and xref:manage:disaster-recovery/shadowing/upgrade.adoc[Upgrade Clusters in a Shadowing Topology]. +See xref:upgrade:deferred-finalization.adoc[Defer Upgrade Finalization]. == Tiered Storage v2 diff --git a/modules/upgrade/pages/deferred-finalization.adoc b/modules/upgrade/pages/deferred-finalization.adoc index 89374f457c..44dc2b2aad 100644 --- a/modules/upgrade/pages/deferred-finalization.adoc +++ b/modules/upgrade/pages/deferred-finalization.adoc @@ -206,7 +206,7 @@ Before starting, familiarize yourself with the limitations of the deferred final * *You cannot skip finalization and keep upgrading.* Redpanda supports upgrades from exactly one feature release back, so an unfinalized v26.2 cluster must finalize before you can begin an upgrade to v26.3. Staying unfinalized indefinitely blocks future upgrades. * *In multi-hop upgrades, only the newest hop is held.* The property value persists across hops, but intermediate hops finalize automatically. For example, in a v25.3 to v26.1 to v26.2 upgrade path, the v25.3 to v26.1 hop auto-finalizes (v26.1 has the property definition but not the deferral logic), and only the v26.1 to v26.2 hop holds. You cannot roll back to v25.3 after reaching v26.1. * *No dedicated metric or alert.* There is no metric for "finalization pending." Monitor with `rpk cluster upgrade status --format json` or the `GetUpgradeStatus` RPC. -* *Brokers advertise the new release version during the soak.* The Admin API, health reports, `rpk redpanda admin brokers list`, and the Redpanda Console broker view all show the new release string while unfinalized; only the logical version and feature activation are held back. The one wire-visible exception: Kafka API keys gated behind held features are absent from `ApiVersions` responses until finalization. +* *Brokers advertise the new release version during the soak.* The Admin API, health reports, `rpk cluster info -b --detailed`, and the Redpanda Console broker view all show the new release string while unfinalized; only the logical version and feature activation are held back. The one wire-visible exception: Kafka API keys gated behind held features are absent from `ApiVersions` responses until finalization. * *`features_auto_finalization` is independent of `features_auto_enable`.* Auto-finalization controls when the cluster version advances; `features_auto_enable` is an older, unrelated property that controls whether individual features activate on their own after the version has advanced. Leave `features_auto_enable` alone. == Deferred finalization in Kubernetes From b9492f0cb829ed7976c766caafee00003eba81df Mon Sep 17 00:00:00 2001 From: Joyce Fee <102751339+Feediver1@users.noreply.github.com> Date: Mon, 27 Jul 2026 22:41:24 -0400 Subject: [PATCH 9/9] docs: upgrade guidance for shadow-linked clusters (DOC-2217) (#1819) Co-authored-by: Claude Fable 5 --- modules/ROOT/nav.adoc | 1 + .../pages/release-notes/redpanda.adoc | 4 +- .../disaster-recovery/shadowing/setup.adoc | 4 +- .../disaster-recovery/shadowing/upgrade.adoc | 93 +++++++++++++++++++ .../upgrade/pages/deferred-finalization.adoc | 1 + 5 files changed, 100 insertions(+), 3 deletions(-) create mode 100644 modules/manage/pages/disaster-recovery/shadowing/upgrade.adoc diff --git a/modules/ROOT/nav.adoc b/modules/ROOT/nav.adoc index 0fb12605b7..9a3577c1e6 100644 --- a/modules/ROOT/nav.adoc +++ b/modules/ROOT/nav.adoc @@ -230,6 +230,7 @@ **** xref:manage:disaster-recovery/shadowing/monitor.adoc[Monitor Shadowing] **** xref:manage:disaster-recovery/shadowing/failover.adoc[Failover] **** xref:manage:disaster-recovery/shadowing/failover-runbook.adoc[Failover Runbook] +**** xref:manage:disaster-recovery/shadowing/upgrade.adoc[Upgrade Clusters in a Shadowing Topology] *** xref:manage:disaster-recovery/whole-cluster-restore.adoc[Whole Cluster Restore] *** xref:manage:disaster-recovery/topic-recovery.adoc[Topic Recovery] ** xref:manage:remote-read-replicas.adoc[Remote Read Replicas] diff --git a/modules/get-started/pages/release-notes/redpanda.adoc b/modules/get-started/pages/release-notes/redpanda.adoc index 87c62bf6b9..ac8a733397 100644 --- a/modules/get-started/pages/release-notes/redpanda.adoc +++ b/modules/get-started/pages/release-notes/redpanda.adoc @@ -35,9 +35,9 @@ See xref:develop:kafka-clients.adoc[Kafka Compatibility] for the full list of va == Deferred upgrade finalization -You can now keep a feature-release upgrade open, soak testing the new version with a guaranteed rollback path, and finalize the upgrade when you are ready. +You can now keep a feature-release upgrade open, soak testing the new version with a guaranteed rollback path, and finalize the upgrade when you are ready. For clusters in a Shadowing topology, new guidance covers upgrade order: upgrade the shadow cluster before the source cluster, and use deferred finalization for bidirectional topologies. -See xref:upgrade:deferred-finalization.adoc[Defer Upgrade Finalization]. +See xref:upgrade:deferred-finalization.adoc[Defer Upgrade Finalization] and xref:manage:disaster-recovery/shadowing/upgrade.adoc[Upgrade Clusters in a Shadowing Topology]. == Tiered Storage v2 diff --git a/modules/manage/pages/disaster-recovery/shadowing/setup.adoc b/modules/manage/pages/disaster-recovery/shadowing/setup.adoc index 3a90cbf24a..eddbb2a955 100644 --- a/modules/manage/pages/disaster-recovery/shadowing/setup.adoc +++ b/modules/manage/pages/disaster-recovery/shadowing/setup.adoc @@ -41,7 +41,9 @@ ifndef::env-cloud[] - Both clusters must be running Redpanda v25.3 or later. - If you use Redpanda Console, ensure that it is running v3.30 or later. -- You must have xref:get-started:licensing/overview.adoc[Enterprise Edition] licenses on both clusters. +- You must have xref:get-started:licensing/overview.adoc[Enterprise Edition] licenses on both clusters. + +TIP: When you later upgrade linked clusters, the upgrade order matters. See xref:manage:disaster-recovery/shadowing/upgrade.adoc[]. endif::[] === Cluster configuration diff --git a/modules/manage/pages/disaster-recovery/shadowing/upgrade.adoc b/modules/manage/pages/disaster-recovery/shadowing/upgrade.adoc new file mode 100644 index 0000000000..c645315ff1 --- /dev/null +++ b/modules/manage/pages/disaster-recovery/shadowing/upgrade.adoc @@ -0,0 +1,93 @@ += Upgrade Clusters in a Shadowing Topology +:description: Upgrade clusters in a Shadowing topology safely: upgrade the shadow cluster before the source cluster, and use deferred finalization for bidirectional topologies. +:page-categories: Management, Disaster Recovery, Upgrades + +[NOTE] +==== +include::shared:partial$enterprise-license.adoc[] +==== + +When two clusters are connected by a xref:manage:disaster-recovery/shadowing/overview.adoc[shadow link], an upgrade is no longer a single-cluster operation. The source and shadow clusters temporarily run different versions during the upgrade window, and the order in which you upgrade them determines whether topic metadata replication continues safely. This page gives you the upgrade order for shadow-linked clusters, and a coordinated procedure for bidirectional topologies, where each cluster shadows the other and no safe upgrade order exists. The bidirectional procedure uses xref:upgrade:deferred-finalization.adoc[deferred upgrade finalization] to keep new-version features inactive on both clusters until both are fully upgraded, so replication is never exposed to features that only one side supports. + +Use this guidance whenever you upgrade clusters that participate in a shadow link. Upgrading them like independent clusters risks breaking replication mid-upgrade. + +== Upgrade the shadow cluster before the source cluster + +When you upgrade a source cluster and its shadow cluster, upgrade the shadow cluster first. + +Shadowing replicates topic configuration from the source to the shadow. If the source cluster is upgraded first, it can start using topic property values that the not-yet-upgraded shadow cluster does not support, which can break the topic metadata replication flow. Upgrading the shadow cluster first means the shadow always understands everything the source can send. + +For each cluster, follow the standard rolling upgrade procedure for your platform: xref:upgrade:rolling-upgrade.adoc[Linux] or xref:upgrade:k-rolling-upgrade.adoc[Kubernetes]. After each cluster's upgrade completes, verify shadow link health before proceeding: run `rpk shadow status ` on the shadow cluster and confirm that the link and its topics are in expected states. See xref:manage:disaster-recovery/shadowing/monitor.adoc[]. + +== Upgrade a bidirectional topology + +In a bidirectional topology, two clusters each maintain a shadow link to the other, replicating non-overlapping sets of topics in opposite directions. Each cluster is an active source for its own topics and a shadow for the other cluster's topics. The topic sets on the two links cannot overlap. + +Because each cluster is simultaneously a source and a shadow, the shadow-first upgrade order cannot be satisfied: whichever cluster you upgrade first is also a source for the other. The workaround is to defer upgrade finalization on both clusters. Both clusters roll to the new binaries while their version-gated features stay inactive, so neither side can start using functionality the other side doesn't support yet. When both clusters are fully upgraded, you finalize both together. + +=== Prerequisites + +* Both clusters must be running the same Redpanda version before you start the upgrade. +* Both clusters must be running at least v26.1.9 or v25.3.15, and the upgrade target must be v26.2 or later. See xref:upgrade:deferred-finalization.adoc#prerequisites[Defer Upgrade Finalization: Prerequisites]. +* Both clusters need a valid Enterprise Edition license. Setting `features_auto_finalization` to `false` is rejected without one. +* You need superuser access on both clusters. + +=== Procedure + +. On both clusters, before beginning either cluster's upgrade, disable automatic finalization: ++ +[,bash] +---- +rpk cluster config set features_auto_finalization false +---- ++ +This is a runtime change; no restart is required. + +. Perform a rolling upgrade of the first cluster, following the standard procedure for your platform (xref:upgrade:rolling-upgrade.adoc[Linux] or xref:upgrade:k-rolling-upgrade.adoc[Kubernetes]). + +. Verify both replication directions: on each cluster, run `rpk shadow status ` and confirm the link, topic, and task states are healthy. See xref:manage:disaster-recovery/shadowing/monitor.adoc[]. + +. Perform a rolling upgrade of the second cluster, then verify both replication directions again. + +. Confirm that both clusters are holding the upgrade open: ++ +[,bash] +---- +rpk cluster upgrade status +---- ++ +Each cluster should report `ready to finalize`. Soak for as long as your validation requires. While either cluster is unfinalized, it can still xref:upgrade:deferred-finalization.adoc#roll-back-to-the-previous-release[roll back to the previous release]. + +. Finalize both clusters promptly together. Do not leave one cluster finalized and the other unfinalized for longer than necessary. On each cluster: ++ +[,bash] +---- +rpk cluster upgrade finalize +---- ++ +Finalization is asynchronous: confirm on each cluster that `rpk cluster upgrade status` reports `finalized` before moving on. See xref:upgrade:deferred-finalization.adoc#finalize-the-upgrade[Finalize the upgrade]. + +. On both clusters, restore automatic finalization for future upgrades: ++ +[,bash] +---- +rpk cluster config set features_auto_finalization true +---- + +. Start using new-version features only after both clusters are finalized. + +=== What to expect while the upgrade is unfinalized + +* Existing shadow links keep replicating in both directions throughout the upgrade, the soak, and any rollback. +* New shadow-link capabilities introduced by the new version cannot be configured yet. For example, on v26.2, configuring role sync or Schema Registry API-mode sync on a shadow link fails with a `FAILED_PRECONDITION` error stating that the feature cannot be configured until the upgrade is finalized. These errors are the feature gates working as designed. For the full list of gated features, see xref:upgrade:deferred-finalization.adoc#what-is-unavailable-while-unfinalized[What is unavailable while unfinalized]. + +== Limitations + +* *Do not stay unfinalized indefinitely.* An unfinalized cluster blocks the next feature-release upgrade: Redpanda supports upgrades from exactly one feature release back, so both clusters must finalize before either can begin the next upgrade. +* *New features wait for both clusters.* Adopt new-version features only after both clusters are upgraded and finalized. + +== Suggested reading + +* xref:upgrade:deferred-finalization.adoc[] +* xref:manage:disaster-recovery/shadowing/setup.adoc[] +* xref:manage:disaster-recovery/shadowing/monitor.adoc[] diff --git a/modules/upgrade/pages/deferred-finalization.adoc b/modules/upgrade/pages/deferred-finalization.adoc index 44dc2b2aad..59562a57d6 100644 --- a/modules/upgrade/pages/deferred-finalization.adoc +++ b/modules/upgrade/pages/deferred-finalization.adoc @@ -241,5 +241,6 @@ New clusters bootstrap already finalized at their binary's version, so `rpk clus * xref:upgrade:rolling-upgrade.adoc[] * xref:upgrade:k-rolling-upgrade.adoc[] +* xref:manage:disaster-recovery/shadowing/upgrade.adoc[] * xref:reference:properties/cluster-properties.adoc#features_auto_finalization[`features_auto_finalization`] * xref:get-started:licensing/overview.adoc[]