Leverage mechanism to cordon brokers when auto-rebalancing on scale down

[ppatierno]

With the current auto-rebalancing on scale down feature, it's possible to trigger a rebalancing when scaling down brokers, so that Cruise Control can move partitions off the brokers which are going to be removed.
During the rebalancing operation, there is nothing to prevent users/applications to create new topics partitions which will be also placed on the brokers to be removed. Of course, it's not ideal because it would mean more rebalancing before scaling down and it could be get stuck for long time.

With KIP-1066, the Apache Kafka community is going to add a mechanism to cordon brokers (or even specific log dirs) so that placing partitions on them is not possible anymore. This feature will be available in the future Kafka 4.3.

We should think about leveraging such a feature as a way to cordon the brokers being scale down.
The idea could be that the new cordoned.log.dirs dynamic configuration should be set as "*" to cordon all log dirs on target scale down brokers before running the rebalancing (to move partitions off). This would avoid any partitions assignments to such brokers while rebalancing is running, so that they will be removed at the end.

[ppatierno]

I was also wondering if cordoning specific log dirs on a broker could be used in case of failed disk when running a remove-disks rebalancing operation, to avoid that while moving data off the failed disk, new partitions could be placed to it (maybe also failing because the non functional disk).