You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: support/azure/azure-kubernetes/connectivity/tunnel-connectivity-issues.md
+21-18Lines changed: 21 additions & 18 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -251,20 +251,22 @@ If everything is OK within the application, you'll have to adjust the allocated
251
251
252
252
You can set up a new cluster to use a Managed Network Address Translation (NAT) Gateway for outbound connections. For more information, see [Create an AKS cluster with a Managed NAT Gateway](/azure/aks/nat-gateway#create-an-aks-cluster-with-a-managed-nat-gateway).
253
253
254
-
## Cause 6: Konnectivity Agents performance challenges with Cluster Growth
254
+
## Cause 6: Konnectivity Agents performance issues with Cluster growth
255
+
256
+
As the cluster grows, the performance of Konnectivity Agents may degrade due to increased network traffic, higher numbers of requests, or resource constraints.
255
257
256
258
> [!NOTE]
257
259
> This cause applies to only the `Konnectivity-agent` pods.
258
260
259
-
### Solution 6: Cluster Proportional Autoscaler (CPA) for Konnectivity Agent
261
+
### Solution 6: Cluster Proportional Autoscaler for Konnectivity Agent
260
262
261
-
To address scalability challenges in large clusters, we have implemented the Cluster Proportional Autoscaler (CPA) for our Konnectivity Agents. This approach aligns with industry standards and best practices, ensuring optimal resource usage and enhanced performance.
263
+
To address scalability challenges in large clusters, we have implemented the Cluster Proportional Autoscaler for our Konnectivity Agents. This approach aligns with industry standards and best practices. It ensures optimal resource usage and enhanced performance.
262
264
263
265
**Why was this change made?**
264
-
Previously, the Konnectivity agent had a fixed replica count, which could create a bottleneck as the cluster grew. With the implementation of the Cluster Proportional Autoscaler (CPA), the replica count now dynamically adjusts based on node-scaling rules, ensuring optimal performance and resource usage.
266
+
Previously, the Konnectivity agent had a fixed replica count, which could create a bottleneck as the cluster grew. With the implementation of the Cluster Proportional Autoscaler, the replica count now dynamically adjusts based on node-scaling rules, ensuring optimal performance and resource usage.
265
267
266
-
**How does the CPA work?**
267
-
The CPA uses a ladder configuration to determine the number of Konnectivity agent replicas based on the cluster size. The ladder configuration is defined in the konnectivity-agent-autoscaler configmap in the kube-system namespace. Here is an example of the ladder configuration:
268
+
**How does the Cluster Proportional Autoscaler work?**
269
+
The Cluster Proportional Autoscaler work uses a ladder configuration to determine the number of Konnectivity agent replicas based on the cluster size. The ladder configuration is defined in the konnectivity-agent-autoscaler configmap in the kube-system namespace. Here is an example of the ladder configuration:
268
270
269
271
```
270
272
nodesToReplicas": [
@@ -279,26 +281,27 @@ nodesToReplicas": [
279
281
280
282
This configuration ensures that the number of replicas scales appropriately with the number of nodes in the cluster, providing optimal resource allocation and improved networking reliability.
281
283
282
-
**How do customers use the Cluster Proportional Autoscaler (CPA)?**
283
-
Customers can override default values by updating the konnectivity-agent-autoscaler configmap in the kube-system namespace. Here is a sample command to update the configmap:
284
+
**How to use the Cluster Proportional Autoscaler?**
285
+
You can override default values by updating the konnectivity-agent-autoscaler configmap in the kube-system namespace. Here is a sample command to update the configmap:
284
286
285
-
```
287
+
```bash
286
288
kubectl edit configmap <pod-name> -n kube-system
287
289
```
288
-
This command opens the configmap in an editor where customers can make the necessary changes.
290
+
This command opens the configmap in an editor where you can make the necessary changes.
291
+
292
+
**What should be checked?**
289
293
290
-
**What should customers check for?**
291
-
Customers need to monitor for Out Of Memory (OOM) kills on their nodes because misconfiguration of the CPA can lead to insufficient memory allocation for the Konnectivity agents. Here are the key reasons:
294
+
You need to monitor for Out Of Memory (OOM) kills on the nodes because misconfiguration of the Cluster Proportional Autoscaler can lead to insufficient memory allocation for the Konnectivity agents. Here are the key reasons:
292
295
293
-
**High Memory Usage:** As the cluster grows, the memory usage of Konnectivity agents can increase significantly, especially during peak loads or when handling large numbers of connections. If the CPA configuration does not scale the replicas appropriately, the agents may run out of memory.
296
+
**High Memory Usage:** As the cluster grows, the memory usage of Konnectivity agents can increase significantly, especially during peak loads or when handling large numbers of connections. If the Cluster Proportional Autoscaler configuration does not scale the replicas appropriately, the agents may run out of memory.
294
297
295
-
**Fixed Resource Limits:** If the resource requests and limits for the Konnectivity agents are set too low, they may not have enough memory to handle the workload, leading to OOM kills. Misconfigured CPA settings can exacerbate this issue by not providing enough replicas to distribute the load.
298
+
**Fixed Resource Limits:** If the resource requests and limits for the Konnectivity agents are set too low, they may not have enough memory to handle the workload, leading to OOM kills. Misconfigured Cluster Proportional Autoscaler settings can exacerbate this issue by not providing enough replicas to distribute the load.
296
299
297
-
**Cluster Size and Workload Variability:** The CPU and memory needed by the Konnectivity agents can vary widely depending on the size of the cluster and the workload. If the CPA ladder configuration is not right-sized and adaptively resized for the cluster's usage patterns, it can lead to memory overcommitment and OOM kills.
300
+
**Cluster Size and Workload Variability:** The CPU and memory needed by the Konnectivity agents can vary widely depending on the size of the cluster and the workload. If the Cluster Proportional Autoscaler ladder configuration is not right-sized and adaptively resized for the cluster's usage patterns, it can lead to memory overcommitment and OOM kills.
298
301
299
-
Here are the steps to identify and troubleshoot OOMKills:
302
+
Here are the steps to identify and troubleshoot OOM Kills:
300
303
301
-
1. Check for OOMKills on Nodes: Use the following command to check for OOMKills on your nodes:
304
+
1. Check for OOM Kills on Nodes: Use the following command to check for OOM Kills on your nodes:
302
305
303
306
```
304
307
kubectl get events --all-namespaces | grep -i 'oomkill'
3. Review Pod Resource Requests and Limits: Ensure that the Konnectivity agent pods have appropriate resource requests and limits set to prevent OOMKills:
316
+
3. Review Pod Resource Requests and Limits: Ensure that the Konnectivity agent pods have appropriate resource requests and limits set to prevent OOM Kills:
314
317
315
318
```
316
319
kubectl get pod <pod-name> -n kube-system -o yaml | grep -A5 "resources:"
0 commit comments