From eae5c571ce0ed8d6d38f88904c6e46eda7fce85a Mon Sep 17 00:00:00 2001 From: David Yu Date: Fri, 24 Jul 2026 14:10:46 -0700 Subject: [PATCH 1/3] docs: describe the per-broker restart probes gating operator rolling restarts Add 'How the Redpanda Operator gates each restart' to the Kubernetes rolling-restart page (operator PR #1547): the pre-restart probe blocks a broker's restart while it would lose acks=1 data, make partitions unavailable, or stall acks=all produce; the post-restart probe holds the next restart until the recovering broker reports 100% of its in-sync-replica load reclaimed, since pod readiness alone doesn't prove catch-up. One broker per pass, Redpanda 25.1+ for the probe endpoints, cluster-wide health fallback on older versions. Adds the v26.2.x what's-new entry. Co-Authored-By: Claude Fable 5 --- .../pages/release-notes/operator.adoc | 8 ++++++++ .../pages/kubernetes/k-rolling-restart.adoc | 19 +++++++++++++++++++ 2 files changed, 27 insertions(+) diff --git a/modules/get-started/pages/release-notes/operator.adoc b/modules/get-started/pages/release-notes/operator.adoc index 352760eb79..b854526eb4 100644 --- a/modules/get-started/pages/release-notes/operator.adoc +++ b/modules/get-started/pages/release-notes/operator.adoc @@ -22,6 +22,14 @@ The release also adds the xref:reference:rpk/rpk-k8s/rpk-k8s.adoc[`rpk k8s` plug For deployment steps, see xref:deploy:redpanda/kubernetes/k-stretch-clusters.adoc[]. +=== Safer rolling restarts with per-broker health probes + +The Redpanda Operator now gates each step of a rolling restart with Redpanda's per-broker restart probes instead of a single cluster-wide health check. Before restarting a broker, the operator confirms that the restart cannot lose `acks=1`-acknowledged data, make partitions unavailable, or block `acks=all` produce requests. After each restart, the operator waits until the broker reports that it has fully re-synced before restarting the next one, because a Pod can pass its readiness checks while the broker is still replaying partition state. + +The probes require Redpanda 25.1 or later. On earlier versions, the operator falls back to the cluster-wide health check. + +See xref:manage:kubernetes/k-rolling-restart.adoc#operator-restart-gates[How the Redpanda Operator gates each restart]. + === Broker pool resources by deployment type The new RedpandaBrokerPool custom resource manages broker pools for Stretch Clusters only. The RedpandaBrokerPool CRD is installed only in multicluster mode, and its controller does not reconcile pools for standalone Redpanda resources. diff --git a/modules/manage/pages/kubernetes/k-rolling-restart.adoc b/modules/manage/pages/kubernetes/k-rolling-restart.adoc index c0c9668ba0..e66e0fa9f8 100644 --- a/modules/manage/pages/kubernetes/k-rolling-restart.adoc +++ b/modules/manage/pages/kubernetes/k-rolling-restart.adoc @@ -42,6 +42,25 @@ use the `statefulset.podTemplate.spec.terminationGracePeriodSeconds` setting. . The `postStart` hook is executed immediately after a container is created. The `postStart` hook takes the broker out of maintenance mode. This action re-integrates the broker into the cluster, allowing it to start handling requests and participate in the cluster's operations again. +[[operator-restart-gates]] +=== How the Redpanda Operator gates each restart + +If the Redpanda Operator manages your cluster, the operator itself replaces the Pods during restarts that it performs, such as applying a configuration change that requires a restart or rolling out a version upgrade. In addition to the maintenance-mode hooks, the operator gates every restart with Redpanda's per-broker restart probes. It restarts at most one broker per pass and re-evaluates the cluster against a fresh view of the brokers before touching the next Pod: + +. Before restarting a broker, the operator calls the broker's pre-restart probe (admin API `GET /v1/broker/pre_restart_probe`), which answers the question: if this broker restarts right now, which partitions are put at risk? The operator delays the restart, and retries on its next pass, while the probe reports any of the following risks: ++ +-- +* Partitions that would lose data acknowledged with `acks=1`. +* Partitions that would become unavailable. +* Partitions that could no longer complete `acks=all` produce requests. +-- ++ +Partitions on topics with a replication factor of 1 do not block the restart, because they have no redundancy for the restart to preserve. To keep them available, temporarily increase their replication factor before you restart. See <>. + +. After the restarted broker's Pod passes its Kubernetes readiness checks, the operator calls the broker's post-restart probe (admin API `GET /v1/broker/post_restart_probe`) and holds the next restart until the broker reports that it has reclaimed 100% of its in-sync-replica load. A Pod can become Ready while the broker inside it is still replaying partition state from its peers, so this gate prevents the operator from taking a second replica down before the first has caught up. + +The restart probes require Redpanda 25.1 or later. On earlier Redpanda versions, the operator falls back to gating restarts on the cluster-wide health check. The probes run only while a restart is in progress, so they add no load during steady-state operation. + include::upgrade:partial$rolling-upgrades/restart-impact.adoc[leveloffset=+1] == Perform a rolling restart From c614f839513ada3b85b5e5bbab6045c2c27499fa Mon Sep 17 00:00:00 2001 From: David Yu Date: Fri, 24 Jul 2026 14:14:02 -0700 Subject: [PATCH 2/3] docs: state that probe-gated rolling restarts start with operator v26.2.1 Co-Authored-By: Claude Fable 5 --- modules/get-started/pages/release-notes/operator.adoc | 2 +- modules/manage/pages/kubernetes/k-rolling-restart.adoc | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/modules/get-started/pages/release-notes/operator.adoc b/modules/get-started/pages/release-notes/operator.adoc index b854526eb4..cd95d07134 100644 --- a/modules/get-started/pages/release-notes/operator.adoc +++ b/modules/get-started/pages/release-notes/operator.adoc @@ -24,7 +24,7 @@ For deployment steps, see xref:deploy:redpanda/kubernetes/k-stretch-clusters.ado === Safer rolling restarts with per-broker health probes -The Redpanda Operator now gates each step of a rolling restart with Redpanda's per-broker restart probes instead of a single cluster-wide health check. Before restarting a broker, the operator confirms that the restart cannot lose `acks=1`-acknowledged data, make partitions unavailable, or block `acks=all` produce requests. After each restart, the operator waits until the broker reports that it has fully re-synced before restarting the next one, because a Pod can pass its readiness checks while the broker is still replaying partition state. +Starting with Redpanda Operator v26.2.1, rolling restarts are safer by default: the operator gates each step of a rolling restart with Redpanda's per-broker restart probes instead of a single cluster-wide health check. Before restarting a broker, the operator confirms that the restart cannot lose `acks=1`-acknowledged data, make partitions unavailable, or block `acks=all` produce requests. After each restart, the operator waits until the broker reports that it has fully re-synced before restarting the next one, because a Pod can pass its readiness checks while the broker is still replaying partition state. The probes require Redpanda 25.1 or later. On earlier versions, the operator falls back to the cluster-wide health check. diff --git a/modules/manage/pages/kubernetes/k-rolling-restart.adoc b/modules/manage/pages/kubernetes/k-rolling-restart.adoc index e66e0fa9f8..791b9cbb2c 100644 --- a/modules/manage/pages/kubernetes/k-rolling-restart.adoc +++ b/modules/manage/pages/kubernetes/k-rolling-restart.adoc @@ -45,7 +45,7 @@ The `postStart` hook takes the broker out of maintenance mode. This action re-in [[operator-restart-gates]] === How the Redpanda Operator gates each restart -If the Redpanda Operator manages your cluster, the operator itself replaces the Pods during restarts that it performs, such as applying a configuration change that requires a restart or rolling out a version upgrade. In addition to the maintenance-mode hooks, the operator gates every restart with Redpanda's per-broker restart probes. It restarts at most one broker per pass and re-evaluates the cluster against a fresh view of the brokers before touching the next Pod: +If the Redpanda Operator manages your cluster, the operator itself replaces the Pods during restarts that it performs, such as applying a configuration change that requires a restart or rolling out a version upgrade. Starting with Redpanda Operator v26.2.1, the operator gates every restart with Redpanda's per-broker restart probes, in addition to the maintenance-mode hooks. It restarts at most one broker per pass and re-evaluates the cluster against a fresh view of the brokers before touching the next Pod: . Before restarting a broker, the operator calls the broker's pre-restart probe (admin API `GET /v1/broker/pre_restart_probe`), which answers the question: if this broker restarts right now, which partitions are put at risk? The operator delays the restart, and retries on its next pass, while the probe reports any of the following risks: + From db2aeeabb8dcf110cf8e11746ba887ce2509313b Mon Sep 17 00:00:00 2001 From: JakeSCahill Date: Mon, 27 Jul 2026 12:32:03 +0100 Subject: [PATCH 3/3] docs: apply standards review fixes - Capitalize Admin API - Use the 'v26.2.1 or later' version phrasing in both files - Rework the release-note opener and the pre-restart probe sentence into active voice - Use the canonical terms readiness probe, in-sync replica, and resynced, and drop periods from noun-fragment bullets --- .../get-started/pages/release-notes/operator.adoc | 2 +- .../manage/pages/kubernetes/k-rolling-restart.adoc | 12 ++++++------ 2 files changed, 7 insertions(+), 7 deletions(-) diff --git a/modules/get-started/pages/release-notes/operator.adoc b/modules/get-started/pages/release-notes/operator.adoc index cd95d07134..86663b4b77 100644 --- a/modules/get-started/pages/release-notes/operator.adoc +++ b/modules/get-started/pages/release-notes/operator.adoc @@ -24,7 +24,7 @@ For deployment steps, see xref:deploy:redpanda/kubernetes/k-stretch-clusters.ado === Safer rolling restarts with per-broker health probes -Starting with Redpanda Operator v26.2.1, rolling restarts are safer by default: the operator gates each step of a rolling restart with Redpanda's per-broker restart probes instead of a single cluster-wide health check. Before restarting a broker, the operator confirms that the restart cannot lose `acks=1`-acknowledged data, make partitions unavailable, or block `acks=all` produce requests. After each restart, the operator waits until the broker reports that it has fully re-synced before restarting the next one, because a Pod can pass its readiness checks while the broker is still replaying partition state. +In Redpanda Operator v26.2.1 or later, the operator gates each step of a rolling restart with Redpanda's per-broker restart probes instead of a single cluster-wide health check. Before restarting a broker, the operator confirms that the restart cannot lose data acknowledged with `acks=1`, make partitions unavailable, or block `acks=all` produce requests. After each restart, the operator waits until the broker reports that it has fully resynced before restarting the next one, because a Pod can pass its readiness probe while the broker is still replaying partition state. The probes require Redpanda 25.1 or later. On earlier versions, the operator falls back to the cluster-wide health check. diff --git a/modules/manage/pages/kubernetes/k-rolling-restart.adoc b/modules/manage/pages/kubernetes/k-rolling-restart.adoc index 791b9cbb2c..a895701006 100644 --- a/modules/manage/pages/kubernetes/k-rolling-restart.adoc +++ b/modules/manage/pages/kubernetes/k-rolling-restart.adoc @@ -45,19 +45,19 @@ The `postStart` hook takes the broker out of maintenance mode. This action re-in [[operator-restart-gates]] === How the Redpanda Operator gates each restart -If the Redpanda Operator manages your cluster, the operator itself replaces the Pods during restarts that it performs, such as applying a configuration change that requires a restart or rolling out a version upgrade. Starting with Redpanda Operator v26.2.1, the operator gates every restart with Redpanda's per-broker restart probes, in addition to the maintenance-mode hooks. It restarts at most one broker per pass and re-evaluates the cluster against a fresh view of the brokers before touching the next Pod: +If the Redpanda Operator manages your cluster, the operator itself replaces the Pods during restarts that it performs, such as applying a configuration change that requires a restart or rolling out a version upgrade. In Redpanda Operator v26.2.1 or later, the operator gates every restart with Redpanda's per-broker restart probes, in addition to the maintenance-mode hooks. It restarts at most one broker per pass and re-evaluates the cluster against a fresh view of the brokers before touching the next Pod: -. Before restarting a broker, the operator calls the broker's pre-restart probe (admin API `GET /v1/broker/pre_restart_probe`), which answers the question: if this broker restarts right now, which partitions are put at risk? The operator delays the restart, and retries on its next pass, while the probe reports any of the following risks: +. Before restarting a broker, the operator calls the broker's pre-restart probe (Admin API `GET /v1/broker/pre_restart_probe`) to determine which partitions the restart would put at risk. The operator delays the restart, and retries on its next pass, while the probe reports any of the following risks: + -- -* Partitions that would lose data acknowledged with `acks=1`. -* Partitions that would become unavailable. -* Partitions that could no longer complete `acks=all` produce requests. +* Partitions that would lose data acknowledged with `acks=1` +* Partitions that would become unavailable +* Partitions that could no longer complete `acks=all` produce requests -- + Partitions on topics with a replication factor of 1 do not block the restart, because they have no redundancy for the restart to preserve. To keep them available, temporarily increase their replication factor before you restart. See <>. -. After the restarted broker's Pod passes its Kubernetes readiness checks, the operator calls the broker's post-restart probe (admin API `GET /v1/broker/post_restart_probe`) and holds the next restart until the broker reports that it has reclaimed 100% of its in-sync-replica load. A Pod can become Ready while the broker inside it is still replaying partition state from its peers, so this gate prevents the operator from taking a second replica down before the first has caught up. +. After the restarted broker's Pod passes its Kubernetes readiness probe, the operator calls the broker's post-restart probe (Admin API `GET /v1/broker/post_restart_probe`) and holds the next restart until the broker reports that it has reclaimed 100% of its in-sync replica load. A Pod can become Ready while the broker inside it is still replaying partition state from its peers, so this gate prevents the operator from taking a second replica down before the first has caught up. The restart probes require Redpanda 25.1 or later. On earlier Redpanda versions, the operator falls back to gating restarts on the cluster-wide health check. The probes run only while a restart is in progress, so they add no load during steady-state operation.