Implement metric(s) to KafkaRoller v2 for see what is happening during rolling updates

[im-konge]

As described in #12575, it would be a good thing to have metrics that would track what is happening currently in the Kafka cluster during rolling update. This should be one of additions to the KafkaRoller v2.

The metrics discussed in the issue were:

• strimzi_rolling_update_in_progress as gauge 0/1, showing if a rolling update is currently active
• strimzi_rolling_update_pods_to_update as gauge, with the number of remaining pods to be updated in the current cycle

However for the second metric there is issue with how current KafkaRoller rolls the Pods - one by one without information on the number of Pods remaining. Maybe it's another thing that will be possible in KafkaRoller v2.

[dariocazas]

As an infrastructure team, when an incident occurs our priority is understanding cluster state as fast as possible. A common pattern we see: another team observes their service degradation, notices the Kafka cluster is "doing a rollout", and immediately assumes the cluster is the root cause. Better rolling-update observability would help to distinguish between cause and coincidence much faster.

The in_progress and pods_to_update are a great start. From operational experience, we'd also benefit from:

• Rolling update reason/trigger: why was the roll initiated? By config change, certificate renewal, manual annotation, ... We have different points to check in our runbooks to find that information, but a direct metric would reduce triage time
• Current step: knowing if the roller is restarting a pod, waiting for readiness, paused due to a FatalProblem... those are useful insights into where the process is stuck rather than just that it's running

We initially check the Kafka CR conditions to get part of that information as we shared in #12575, but it reflects the outcome of previous reconciliation cycle. During a rolling update it can show a stale value even though the operator has long since recovered and is actively making progress. Using it as an indicator points operators in the wrong direction.

---

After context, thinking in concrete proposals:

• strimzi_rolling_update_in_progress: gauge 0/1
• strimzi_rolling_update_pods_remaining: gauge with number of pods still pending a restart in the current rolling cycle. I like to suggest include a label reason to share the trigger of this restart
• strimzi_rolling_update_pods_done: as gauge, for me made sense aligned with the previous one, and with the same reason label
• strimzi_rolling_update_current_pod_phase: gauge 0/1, with a label with pod name and phase (pending_restart, restarting, waiting_readiness, waiting_isr, done, ...). If a phase waiting_readiness has been 1 for 15m, the operator can immediately focus on that specific pod instead of guessing

All of these metrics could be labeled with namespace and cluster (Kafka name)

We could also consider distinguishing pods that directly affect Kafka clients (broker-role nodes) from supporting components rolled in the same cycle (e.g., Cruise Control). During an incident, operators need to know whether the rolling update is impacting the data plane (a Cruise Control restart is operationally irrelevant to client-facing impact).

cc: @santimillang

[katheris]

+1 from me but this should have a proposal (or be incorporated into a new Kafka Roller v2 proposal)