Skip to content

Commit 45d69b4

Browse files
author
Simonx Xu
authored
Merge pull request #9343 from MicrosoftDocs/main
Auto push to live 2025-07-16 02:39:08
2 parents 5ce53f1 + 6a0f38a commit 45d69b4

2 files changed

Lines changed: 183 additions & 34 deletions

File tree

‎support/azure/azure-kubernetes/availability-performance/cluster-service-health-probe-mode-issues.md‎

Lines changed: 156 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -4,8 +4,9 @@ description: Diagnoses and fixes common issues with the health probe mode featur
44
ms.date: 06/03/2024
55
ms.reviewer: niqi, cssakscic, v-weizhu
66
ms.service: azure-kubernetes-service
7-
ms.custom: sap:Node/node pool availability and performance, devx-track-azurecli
7+
ms.custom: sap:Node/node pool availability and performance, devx-track-azurecli, innovation-engine
88
---
9+
910
# Troubleshoot issues when enabling the AKS cluster service health probe mode
1011

1112
The health probe mode feature allows you to configure how Azure Load Balancer probes the health of the nodes in your Azure Kubernetes Service (AKS) cluster. You can choose between two modes: Shared and ServiceNodePort. The Shared mode uses a single health probe for all external traffic policy cluster services that use the same load balancer. In contrast, the ServiceNodePort mode uses a separate health probe for each service. The Shared mode can reduce the number of health probes and improve the performance of the load balancer, but it requires some additional components to work properly. To enable this feature, see [How to enable the health probe mode feature using the Azure CLI](#how-to-enable-the-health-probe-mode-feature-using-the-azure-cli).
@@ -36,11 +37,92 @@ The following operations also happen:
3637

3738
To troubleshoot these issues, follow these steps:
3839

39-
1. Check the RP frontend log to see if the health probe mode in the LoadBalancerProfile is properly configured. You can use the `az aks show` command to view the LoadBalancerProfile property of your cluster.
40-
41-
2. Check the *overlaymgr* log to see if the cloud provider secret is updated. The keyword to look for is `cloudConfigSecretResolver`. Or check the contents of the cloud-provider-config secret in the `ccp` namespace. You can use the `kubectl get secret` command to view the secret.
42-
43-
3. Check the chart or overlay daemonset cloud-node-manager to see if the health-probe-proxy sidecar container is enabled. You can use the `kubectl get ds` command to view the daemonset.
40+
1. First, connect to your AKS cluster using the Azure CLI:
41+
42+
```azurecli
43+
export RESOURCE_GROUP="aks-rg"
44+
export AKS_CLUSTER_NAME="aks-cluster"
45+
az aks get-credentials --resource-group $RESOURCE_GROUP --name $AKS_CLUSTER_NAME --overwrite-existing
46+
```
47+
48+
2. Next, check the RP frontend log to see if the health probe mode in the LoadBalancerProfile is properly configured. You can use the `az aks show` command to view the LoadBalancerProfile property of your cluster.
49+
50+
```azurecli
51+
export RESOURCE_GROUP="aks-rg"
52+
export AKS_CLUSTER_NAME="aks-cluster"
53+
az aks show --resource-group $RESOURCE_GROUP --name $AKS_CLUSTER_NAME --query "networkProfile.loadBalancerProfile"
54+
```
55+
Results:
56+
57+
<!-- expected_similarity=0.3 -->
58+
59+
```output
60+
{
61+
"clusterServiceLoadBalancerHealthProbeMode": "Shared",
62+
"managedOutboundIPs": null,
63+
"outboundIPs": null,
64+
"outboundIPPrefixes": null,
65+
"allocatedOutboundPorts": null,
66+
"effectiveOutboundIPs": [
67+
{
68+
"id": "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/MC_aks-rg_aks-cluster_eastus2/providers/Microsoft.Network/publicIPAddresses/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
69+
}
70+
],
71+
"idleTimeoutInMinutes": 30,
72+
"loadBalancerSku": "standard",
73+
"managedOutboundIPv6": null
74+
}
75+
```
76+
77+
3. Check the cloud provider configuration. In modern AKS clusters, the cloud provider configuration is managed internally and the `ccp` namespace doesn't exist. Instead, check for cloud provider related resources and verify the cloud-node-manager pods are running properly:
78+
79+
80+
```bash
81+
# Check for cloud provider related ConfigMaps in kube-system
82+
kubectl get configmap -n kube-system | grep -i azure
83+
84+
# Check if cloud-node-manager pods are running (indicates cloud provider integration is working)
85+
kubectl get pods -n kube-system | grep cloud-node-manager
86+
87+
# Check the azure-ip-masq-agent-config if it exists
88+
kubectl get configmap azure-ip-masq-agent-config-reconciled -n kube-system -o yaml 2>/dev/null || echo "ConfigMap not found"
89+
```
90+
Results:
91+
92+
<!-- expected_similarity=0.3 -->
93+
94+
```output
95+
configmap/azure-ip-masq-agent-config-reconciled 1 11h
96+
97+
cloud-node-manager-rfb2w 2/2 Running 0 16m
98+
```
99+
100+
4. Check the chart or overlay daemonset cloud-node-manager to see if the health-probe-proxy sidecar container is enabled. You can use the `kubectl get ds` command to view the daemonset.
101+
102+
```shell
103+
kubectl get ds -n kube-system cloud-node-manager -o yaml
104+
```
105+
Results:
106+
107+
<!-- expected_similarity=0.3 -->
108+
109+
```output
110+
apiVersion: apps/v1
111+
kind: DaemonSet
112+
metadata:
113+
name: cloud-node-manager
114+
namespace: kube-system
115+
...
116+
spec:
117+
template:
118+
spec:
119+
containers:
120+
- name: cloud-node-manager
121+
image: mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:xxxxxxxx
122+
- name: health-probe-proxy
123+
image: mcr.microsoft.com/oss/kubernetes/azure-health-probe-proxy:xxxxxxxx
124+
...
125+
```
44126
45127
## Cause 1: The health probe mode isn't Shared or ServiceNodePort
46128
@@ -74,6 +156,26 @@ The health probe mode feature requires you to register the feature on your subsc
74156
75157
Make sure you register the feature for your subscription before creating or updating your cluster. You can use the `az feature register` command to register the feature.
76158
159+
```azurecli
160+
export FEATURE_NAME="EnableSLBSharedHealthProbePreview"
161+
export PROVIDER_NAMESPACE="Microsoft.ContainerService"
162+
az feature register --name $FEATURE_NAME --namespace $PROVIDER_NAMESPACE
163+
```
164+
Results:
165+
166+
<!-- expected_similarity=0.3 -->
167+
168+
```output
169+
{
170+
"id": "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/providers/Microsoft.Features/providers/Microsoft.ContainerService/features/EnableAKSClusterServiceLoadBalancerHealthProbeMode",
171+
"name": "Microsoft.ContainerService/EnableAKSClusterServiceLoadBalancerHealthProbeMode",
172+
"properties": {
173+
"state": "Registering"
174+
},
175+
"type": "Microsoft.Features/providers/features"
176+
}
177+
```
178+
77179
## Cause 5: The Kubernetes version is earlier than v1.28.0
78180

79181
The health probe mode feature requires a minimum Kubernetes version of v1.28.0. If you use an older version, the feature won't work.
@@ -90,8 +192,53 @@ For Windows, the kube-proxy component doesn't start until you create the first n
90192

91193
To enable the health probe mode feature, run one of the following commands:
92194

93-
- `az aks create/update --cluster-service-load-balancer-health-probe-mode Shared`
94-
95-
- `az aks create/update --cluster-service-load-balancer-health-probe-mode ServiceNodePort (default)`
195+
Enable `ServiceNodePort` health probe mode (default) for a cluster:
196+
197+
```shell
198+
export RESOURCE_GROUP="aks-rg"
199+
export AKS_CLUSTER_NAME="aks-cluster"
200+
az aks update --resource-group $RESOURCE_GROUP --name $AKS_CLUSTER_NAME --cluster-service-load-balancer-health-probe-mode ServiceNodePort
201+
```
202+
Results:
203+
204+
```output
205+
{
206+
"name": "aks-cluster",
207+
"location": "eastus2",
208+
"resourceGroup": "aks-rg",
209+
"kubernetesVersion": "1.28.x",
210+
"provisioningState": "Succeeded",
211+
"loadBalancerProfile": {
212+
"clusterServiceLoadBalancerHealthProbeMode": "ServiceNodePort",
213+
...
214+
},
215+
...
216+
}
217+
```
218+
219+
Enable `Shared` health probe mode for a cluster:
220+
221+
```shell
222+
export RESOURCE_GROUP="MyAksResourceGroup"
223+
export AKS_CLUSTER_NAME="MyAksCluster"
224+
az aks update --resource-group $RESOURCE_GROUP --name $AKS_CLUSTER_NAME --cluster-service-load-balancer-health-probe-mode Shared
225+
```
226+
227+
Results:
228+
229+
```output
230+
{
231+
"name": "MyAksCluster",
232+
"location": "eastus2",
233+
"resourceGroup": "MyAksResourceGroup",
234+
"kubernetesVersion": "1.28.x",
235+
"provisioningState": "Succeeded",
236+
"loadBalancerProfile": {
237+
"clusterServiceLoadBalancerHealthProbeMode": "Shared",
238+
...
239+
},
240+
...
241+
}
242+
```
96243

97244
[!INCLUDE [Azure Help Support](../../../includes/azure-help-support.md)]

‎support/azure/azure-kubernetes/create-upgrade-delete/cannot-scale-cluster-autoscaler-enabled-node-pool.md‎

Lines changed: 27 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -3,14 +3,15 @@ title: Cluster autoscaler fails to scale with cannot scale cluster autoscaler en
33
description: Learn how to troubleshoot the cannot scale cluster autoscaler enabled node pool error when your autoscaler isn't scaling up or down.
44
author: sgeannina
55
ms.author: ninasegares
6-
ms.date: 04/17/2025
7-
ms.reviewer: aritraghosh, chiragpa.momajed
6+
ms.date: 06/09/2024
7+
ms.reviewer: aritraghosh, chiragpa
88
ms.service: azure-kubernetes-service
9-
ms.custom: sap:Create, Upgrade, Scale and Delete operations (cluster or nodepool)
9+
ms.custom: sap:Create, Upgrade, Scale and Delete operations (cluster or nodepool), innovation-engine
1010
---
11+
1112
# Cluster autoscaler fails to scale with "cannot scale cluster autoscaler enabled node pool" error
1213

13-
This article discusses how to resolve the "cannot scale cluster autoscaler enabled node pool" error that occurs when you scale a cluster that has an autoscaler-enabled node pool.
14+
This article discusses how to resolve the "cannot scale cluster autoscaler enabled node pool" error that appears when scaling a cluster with an autoscaler enabled node pool.
1415

1516
## Symptoms
1617

@@ -22,33 +23,33 @@ You receive an error message that resembles the following message:
2223
2324
## Troubleshooting checklist
2425

25-
Azure Kubernetes Service (AKS) uses Azure Virtual Machine Scale Sets-based agent pools. These pools contain cluster nodes and [cluster autoscaling capabilities](/azure/aks/cluster-autoscaler), if they're enabled.
26+
Azure Kubernetes Service (AKS) uses virtual machine scale sets-based agent pools, which contain cluster nodes and [cluster autoscaling capabilities](/azure/aks/cluster-autoscaler) if enabled.
2627

2728
### Check that the cluster virtual machine scale set exists
2829

29-
1. Sign in to the [Azure portal](https://portal.azure.com).
30-
1. Find the node resource group by searching for the following names:
30+
1. Sign in to [Azure portal](https://portal.azure.com).
31+
1. Find the node resource group by searching the following names:
32+
33+
- The default name `MC_{AksResourceGroupName}_{YourAksClusterName}_{AksResourceLocation}`.
34+
- The custom name (if it was provided at creation).
3135

32-
- The default name `MC_{AksResourceGroupName}_{YourAksClusterName}_{AksResourceLocation}`
33-
- The custom name (if it was provided at creation)
34-
>
3536
> [!NOTE]
36-
> When you create a cluster, AKS automatically creates a second resource group to store the AKS resources. For more information, see [Why are two resource groups created with AKS?](/azure/aks/faq#why-are-two-resource-groups-created-with-aks)
37+
> When you create a new cluster, AKS automatically creates a second resource group to store the AKS resources. For more information, see [Why are two resource groups created with AKS?](/azure/aks/faq#why-are-two-resource-groups-created-with-aks)
3738
38-
1. Check the list of resources to make sure that a virtual machine scale set exists.
39+
1. Check the list of resources and make sure that there's a virtual machine scale set.
3940

4041
## Cause 1: The cluster virtual machine scale set was deleted
4142

42-
If you delete the virtual machine scale set that's attached to the cluster, this action causes the cluster autoscaler to fail. It also causes issues when you provision resources such as nodes and pods.
43+
Deleting the virtual machine scale set attached to the cluster causes the cluster autoscaler to fail. It also causes issues when provisioning resources such as nodes and pods.
4344

4445
> [!NOTE]
45-
> Modifying any resource under the node resource group in the AKS cluster is an unsupported action and causes cluster operation failures. You can prevent changes from being made to the node resource group by [blocking users from modifying resources](/azure/aks/cluster-configuration#fully-managed-resource-group-preview) that are managed by the AKS cluster.
46+
> Modifying any resource under the node resource group in the AKS cluster is an unsupported action and will cause cluster operation failures. You can prevent changes from being made to the node resource group by [blocking users from modifying resources](/azure/aks/cluster-configuration#fully-managed-resource-group-preview) managed by the AKS cluster.
4647
4748
### Reconcile node pool
4849

4950
If the cluster virtual machine scale set is accidentally deleted, you can reconcile the node pool by using `az aks nodepool update`:
5051

51-
```bash
52+
```shell
5253
# Update Node Pool Configuration
5354
az aks nodepool update --resource-group <resource-group-name> --cluster-name <cluster-name> --name <nodepool-name> --tags <tags> --node-taints <taints> --labels <labels>
5455

@@ -59,13 +60,13 @@ Monitor the node pool to make sure that it's functioning as expected and that al
5960

6061
## Cause 2: Tags or any other properties were modified from the node resource group
6162

62-
You may experience scaling errors if you modify or delete Azure-created tags and other resource properties in the node resource group. For more information, see [Can I modify tags and other properties of the AKS resources in the node resource group?](/azure/aks/faq#can-i-modify-tags-and-other-properties-of-the-aks-resources-in-the-node-resource-group)
63+
You may receive scaling errors if you modify or delete Azure-created tags and other resource properties in the node resource group. For more information, see [Can I modify tags and other properties of the AKS resources in the node resource group?](/azure/aks/faq#can-i-modify-tags-and-other-properties-of-the-aks-resources-in-the-node-resource-group)
6364

6465
### Reconcile node resource group tags
6566

6667
Use the Azure CLI to make sure that the node resource group has the correct tags for AKS name and the AKS group name:
6768

68-
```bash
69+
```shell
6970
# Add or update tags for AKS name and AKS group name
7071
az group update --name <node-resource-group-name> --set tags.AKS-Managed-Cluster-Name=<aks-managed-cluster-name> tags.AKS-Managed-Cluster-RG=<aks-managed-cluster-rg>
7172

@@ -76,21 +77,22 @@ Monitor the resource group to make sure that the tags are correctly applied and
7677

7778
## Cause 3: The cluster node resource group was deleted
7879

79-
Deleting the cluster node resource group causes issues when you provision the infrastructure resources that are required by the cluster. This action causes the cluster autoscaler to fail.
80+
Deleting the cluster node resource group causes issues when provisioning the infrastructure resources required by the cluster, which causes the cluster autoscaler to fail.
8081

8182
## Solution: Update the cluster to the goal state without changing the configuration
8283

83-
To resolve this issue, run the following command to recover the deleted virtual machine scale set or any tags (missing or modified).
84+
To resolve this issue, you can run the following command to recover the deleted virtual machine scale set or any tags (missing or modified):
8485

8586
> [!NOTE]
86-
> It might take a few minutes until the operation finishes.
87+
> It might take a few minutes until the operation completes.
88+
89+
Set your environment variables for the AKS cluster resource group and cluster name before running the command. A random suffix is included to prevent name collisions during repeatable executions, but you must ensure the resource group and cluster exist.
8790

8891
```azurecli
89-
az aks update --resource-group <resource-group-name> --name <aks-cluster-name>
92+
export RANDOM_SUFFIX=$(head -c 3 /dev/urandom | xxd -p)
93+
export AKS_RG_NAME="MyAksResourceGroup$RANDOM_SUFFIX"
94+
export AKS_CLUSTER_NAME="MyAksCluster$RANDOM_SUFFIX"
95+
az aks update --resource-group $AKS_RG_NAME --name $AKS_CLUSTER_NAME --no-wait
9096
```
9197

92-
### Additional troubleshooting tips
93-
94-
- Check the Azure Activity Log for any recent changes or deletions.
95-
9698
[!INCLUDE [Azure Help Support](../../../includes/azure-help-support.md)]

0 commit comments

Comments
 (0)