diff --git a/modules/ROOT/nav.adoc b/modules/ROOT/nav.adoc index 0fb12605b7..9a3577c1e6 100644 --- a/modules/ROOT/nav.adoc +++ b/modules/ROOT/nav.adoc @@ -230,6 +230,7 @@ **** xref:manage:disaster-recovery/shadowing/monitor.adoc[Monitor Shadowing] **** xref:manage:disaster-recovery/shadowing/failover.adoc[Failover] **** xref:manage:disaster-recovery/shadowing/failover-runbook.adoc[Failover Runbook] +**** xref:manage:disaster-recovery/shadowing/upgrade.adoc[Upgrade Clusters in a Shadowing Topology] *** xref:manage:disaster-recovery/whole-cluster-restore.adoc[Whole Cluster Restore] *** xref:manage:disaster-recovery/topic-recovery.adoc[Topic Recovery] ** xref:manage:remote-read-replicas.adoc[Remote Read Replicas] diff --git a/modules/get-started/pages/release-notes/redpanda.adoc b/modules/get-started/pages/release-notes/redpanda.adoc index 87c62bf6b9..ac8a733397 100644 --- a/modules/get-started/pages/release-notes/redpanda.adoc +++ b/modules/get-started/pages/release-notes/redpanda.adoc @@ -35,9 +35,9 @@ See xref:develop:kafka-clients.adoc[Kafka Compatibility] for the full list of va == Deferred upgrade finalization -You can now keep a feature-release upgrade open, soak testing the new version with a guaranteed rollback path, and finalize the upgrade when you are ready. +You can now keep a feature-release upgrade open, soak testing the new version with a guaranteed rollback path, and finalize the upgrade when you are ready. For clusters in a Shadowing topology, new guidance covers upgrade order: upgrade the shadow cluster before the source cluster, and use deferred finalization for bidirectional topologies. -See xref:upgrade:deferred-finalization.adoc[Defer Upgrade Finalization]. +See xref:upgrade:deferred-finalization.adoc[Defer Upgrade Finalization] and xref:manage:disaster-recovery/shadowing/upgrade.adoc[Upgrade Clusters in a Shadowing Topology]. == Tiered Storage v2 diff --git a/modules/manage/pages/disaster-recovery/shadowing/setup.adoc b/modules/manage/pages/disaster-recovery/shadowing/setup.adoc index 3a90cbf24a..eddbb2a955 100644 --- a/modules/manage/pages/disaster-recovery/shadowing/setup.adoc +++ b/modules/manage/pages/disaster-recovery/shadowing/setup.adoc @@ -41,7 +41,9 @@ ifndef::env-cloud[] - Both clusters must be running Redpanda v25.3 or later. - If you use Redpanda Console, ensure that it is running v3.30 or later. -- You must have xref:get-started:licensing/overview.adoc[Enterprise Edition] licenses on both clusters. +- You must have xref:get-started:licensing/overview.adoc[Enterprise Edition] licenses on both clusters. + +TIP: When you later upgrade linked clusters, the upgrade order matters. See xref:manage:disaster-recovery/shadowing/upgrade.adoc[]. endif::[] === Cluster configuration diff --git a/modules/manage/pages/disaster-recovery/shadowing/upgrade.adoc b/modules/manage/pages/disaster-recovery/shadowing/upgrade.adoc new file mode 100644 index 0000000000..c645315ff1 --- /dev/null +++ b/modules/manage/pages/disaster-recovery/shadowing/upgrade.adoc @@ -0,0 +1,93 @@ += Upgrade Clusters in a Shadowing Topology +:description: Upgrade clusters in a Shadowing topology safely: upgrade the shadow cluster before the source cluster, and use deferred finalization for bidirectional topologies. +:page-categories: Management, Disaster Recovery, Upgrades + +[NOTE] +==== +include::shared:partial$enterprise-license.adoc[] +==== + +When two clusters are connected by a xref:manage:disaster-recovery/shadowing/overview.adoc[shadow link], an upgrade is no longer a single-cluster operation. The source and shadow clusters temporarily run different versions during the upgrade window, and the order in which you upgrade them determines whether topic metadata replication continues safely. This page gives you the upgrade order for shadow-linked clusters, and a coordinated procedure for bidirectional topologies, where each cluster shadows the other and no safe upgrade order exists. The bidirectional procedure uses xref:upgrade:deferred-finalization.adoc[deferred upgrade finalization] to keep new-version features inactive on both clusters until both are fully upgraded, so replication is never exposed to features that only one side supports. + +Use this guidance whenever you upgrade clusters that participate in a shadow link. Upgrading them like independent clusters risks breaking replication mid-upgrade. + +== Upgrade the shadow cluster before the source cluster + +When you upgrade a source cluster and its shadow cluster, upgrade the shadow cluster first. + +Shadowing replicates topic configuration from the source to the shadow. If the source cluster is upgraded first, it can start using topic property values that the not-yet-upgraded shadow cluster does not support, which can break the topic metadata replication flow. Upgrading the shadow cluster first means the shadow always understands everything the source can send. + +For each cluster, follow the standard rolling upgrade procedure for your platform: xref:upgrade:rolling-upgrade.adoc[Linux] or xref:upgrade:k-rolling-upgrade.adoc[Kubernetes]. After each cluster's upgrade completes, verify shadow link health before proceeding: run `rpk shadow status ` on the shadow cluster and confirm that the link and its topics are in expected states. See xref:manage:disaster-recovery/shadowing/monitor.adoc[]. + +== Upgrade a bidirectional topology + +In a bidirectional topology, two clusters each maintain a shadow link to the other, replicating non-overlapping sets of topics in opposite directions. Each cluster is an active source for its own topics and a shadow for the other cluster's topics. The topic sets on the two links cannot overlap. + +Because each cluster is simultaneously a source and a shadow, the shadow-first upgrade order cannot be satisfied: whichever cluster you upgrade first is also a source for the other. The workaround is to defer upgrade finalization on both clusters. Both clusters roll to the new binaries while their version-gated features stay inactive, so neither side can start using functionality the other side doesn't support yet. When both clusters are fully upgraded, you finalize both together. + +=== Prerequisites + +* Both clusters must be running the same Redpanda version before you start the upgrade. +* Both clusters must be running at least v26.1.9 or v25.3.15, and the upgrade target must be v26.2 or later. See xref:upgrade:deferred-finalization.adoc#prerequisites[Defer Upgrade Finalization: Prerequisites]. +* Both clusters need a valid Enterprise Edition license. Setting `features_auto_finalization` to `false` is rejected without one. +* You need superuser access on both clusters. + +=== Procedure + +. On both clusters, before beginning either cluster's upgrade, disable automatic finalization: ++ +[,bash] +---- +rpk cluster config set features_auto_finalization false +---- ++ +This is a runtime change; no restart is required. + +. Perform a rolling upgrade of the first cluster, following the standard procedure for your platform (xref:upgrade:rolling-upgrade.adoc[Linux] or xref:upgrade:k-rolling-upgrade.adoc[Kubernetes]). + +. Verify both replication directions: on each cluster, run `rpk shadow status ` and confirm the link, topic, and task states are healthy. See xref:manage:disaster-recovery/shadowing/monitor.adoc[]. + +. Perform a rolling upgrade of the second cluster, then verify both replication directions again. + +. Confirm that both clusters are holding the upgrade open: ++ +[,bash] +---- +rpk cluster upgrade status +---- ++ +Each cluster should report `ready to finalize`. Soak for as long as your validation requires. While either cluster is unfinalized, it can still xref:upgrade:deferred-finalization.adoc#roll-back-to-the-previous-release[roll back to the previous release]. + +. Finalize both clusters promptly together. Do not leave one cluster finalized and the other unfinalized for longer than necessary. On each cluster: ++ +[,bash] +---- +rpk cluster upgrade finalize +---- ++ +Finalization is asynchronous: confirm on each cluster that `rpk cluster upgrade status` reports `finalized` before moving on. See xref:upgrade:deferred-finalization.adoc#finalize-the-upgrade[Finalize the upgrade]. + +. On both clusters, restore automatic finalization for future upgrades: ++ +[,bash] +---- +rpk cluster config set features_auto_finalization true +---- + +. Start using new-version features only after both clusters are finalized. + +=== What to expect while the upgrade is unfinalized + +* Existing shadow links keep replicating in both directions throughout the upgrade, the soak, and any rollback. +* New shadow-link capabilities introduced by the new version cannot be configured yet. For example, on v26.2, configuring role sync or Schema Registry API-mode sync on a shadow link fails with a `FAILED_PRECONDITION` error stating that the feature cannot be configured until the upgrade is finalized. These errors are the feature gates working as designed. For the full list of gated features, see xref:upgrade:deferred-finalization.adoc#what-is-unavailable-while-unfinalized[What is unavailable while unfinalized]. + +== Limitations + +* *Do not stay unfinalized indefinitely.* An unfinalized cluster blocks the next feature-release upgrade: Redpanda supports upgrades from exactly one feature release back, so both clusters must finalize before either can begin the next upgrade. +* *New features wait for both clusters.* Adopt new-version features only after both clusters are upgraded and finalized. + +== Suggested reading + +* xref:upgrade:deferred-finalization.adoc[] +* xref:manage:disaster-recovery/shadowing/setup.adoc[] +* xref:manage:disaster-recovery/shadowing/monitor.adoc[] diff --git a/modules/upgrade/pages/deferred-finalization.adoc b/modules/upgrade/pages/deferred-finalization.adoc index 44dc2b2aad..59562a57d6 100644 --- a/modules/upgrade/pages/deferred-finalization.adoc +++ b/modules/upgrade/pages/deferred-finalization.adoc @@ -241,5 +241,6 @@ New clusters bootstrap already finalized at their binary's version, so `rpk clus * xref:upgrade:rolling-upgrade.adoc[] * xref:upgrade:k-rolling-upgrade.adoc[] +* xref:manage:disaster-recovery/shadowing/upgrade.adoc[] * xref:reference:properties/cluster-properties.adoc#features_auto_finalization[`features_auto_finalization`] * xref:get-started:licensing/overview.adoc[]