|
| 1 | +--- |
| 2 | +title: Troubleshoot CPU Pressure in AKS Clusters Using PSI Metrics |
| 3 | +description: Provides troubleshoot guidance on CPU pressure using PSI metrics in an AKS cluster. |
| 4 | +ms.date: 05/21/2025 |
| 5 | +ms.reviewer: aritraghosh, dafell, alvinli, v-weizhu |
| 6 | +ms.service: azure-kubernetes-service |
| 7 | +ms.custom: sap:Node/node pool availability and performance |
| 8 | +--- |
| 9 | +# Troubleshoot CPU pressure in AKS clusters using PSI metrics |
| 10 | + |
| 11 | +CPU pressure is a more accurate indicator of resource contention than traditional CPU utilization metrics. While high CPU usage shows resource consumption, it doesn't necessarily indicate performance problems. In an Azure Kubernetes Service (AKS) cluster, understanding CPU pressure through Pressure Stall Information (PSI) metrics helps identify true resource contention issues. |
| 12 | + |
| 13 | +When a node in an AKS cluster experiences CPU pressure, applications might suffer from poor performance even when CPU utilization appears moderate. PSI metrics provide insight into actual resource contention by measuring task delays rather than just resource consumption. |
| 14 | + |
| 15 | +This article helps you monitor CPU pressure using PSI metrics and provides best practices to resolve resource contention issues. |
| 16 | + |
| 17 | +## Symptoms |
| 18 | + |
| 19 | +The following table outlines the common symptoms of CPU pressure: |
| 20 | + |
| 21 | +|Symptom | Description | |
| 22 | +|---|---| |
| 23 | +|Increased application latency|Services respond slower even when CPU utilization appears moderate.| |
| 24 | +|Throttled containers|Containers experience delays in processing despite having CPU resources available on the node.| |
| 25 | +|Degraded performance|Applications experience unpredictable performance variations that don't correlate with CPU usage percentages.| |
| 26 | + |
| 27 | +## Troubleshooting checklist |
| 28 | + |
| 29 | +To identify and resolve CPU pressure issues, follow these steps: |
| 30 | + |
| 31 | +### Step 1: Enable and monitor PSI metrics |
| 32 | + |
| 33 | +Use one of the following methods to access PSI metrics: |
| 34 | + |
| 35 | +- In a web browser, use Azure Monitoring Managed Prometheus or other monitoring solution to query PSI metrics. |
| 36 | +- In a console, use the Kubernetes command-line tool (`kubectl`). |
| 37 | + |
| 38 | +### [Browser](#tab/browser) |
| 39 | + |
| 40 | +Azure Monitoring Managed Prometheus provides a way to monitor PSI metrics: |
| 41 | + |
| 42 | +1. Enable Azure Monitoring Managed Prometheus for your AKS cluster by following the instructions in [Enable Prometheus and Grafana](/azure/azure-monitor/containers/kubernetes-monitoring-enable#enable-prometheus-and-grafana). |
| 43 | + |
| 44 | + To enable customized scrape metrics for Prometheus, see [Scrape configs](/azure/azure-monitor/containers/prometheus-metrics-scrape-configuration#scrape-configs). We recommend setting `minumum ingestion profile` to `false` and `node-exporter` to `true`. |
| 45 | + |
| 46 | +2. Navigate to the Azure Monitor workspace associated with the AKS cluster from the [Azure portal](https://portal.azure.com). |
| 47 | + |
| 48 | + :::image type="content" source="media/troubleshoot-node-cpu-pressure-psi/configure-azure-monitor-for-containers.png" alt-text="Screenshot that shows how to navigate to the Azure Monitor workspace." lightbox="media/troubleshoot-node-cpu-pressure-psi/configure-azure-monitor-for-containers.png"::: |
| 49 | + |
| 50 | +3. Under **Monitoring**, select **Metrics**. |
| 51 | + |
| 52 | +4. Select **Prometheus metrics** as the data source. |
| 53 | + |
| 54 | + > [!NOTE] |
| 55 | + > To use the metrics, you need to enable them in Azure Monitoring Managed Prometheus. These metrics are exposed by Node Exporter or cAdvisor. |
| 56 | +
|
| 57 | +5. Query specific PSI metrics in Prometheus explorer: |
| 58 | + |
| 59 | + - For node-level CPU pressure, use the `node_pressure_cpu_waiting_seconds_total` Prometheus Query Language (PromQL). |
| 60 | + |
| 61 | + :::image type="content" source="media/troubleshoot-node-cpu-pressure-psi/node-level-cpu-pressure.png" alt-text="Screenshot that shows how to query node-level CPU pressure." lightbox="media/troubleshoot-node-cpu-pressure-psi/node-level-cpu-pressure.png"::: |
| 62 | + |
| 63 | + - For pod-level CPU pressure, use the `container_cpu_cfs_throttled_seconds_total` PromQL. |
| 64 | + |
| 65 | +6. Calculate the PSI-some percentage (percentage of time at least one task is stalled on CPU): |
| 66 | + |
| 67 | + `rate(node_pressure_cpu_waiting_seconds_total[5m]) * 100` |
| 68 | + |
| 69 | +> [!NOTE] |
| 70 | +> Some of the container level metrics such as `container_pressure_cpu_waiting_seconds_total` and `container_pressure_cpu_stalled_seconds_total` aren't available in AKS as they're part of the Kubelet PSI feature gate that is in alpha state. AKS begins supporting the use of the feature when it reaches beta stage. |
| 71 | +
|
| 72 | +### [Command Line](#tab/command-line) |
| 73 | + |
| 74 | +Access PSI metrics safely using kubectl without requiring Secure Shell (SSH) access: |
| 75 | + |
| 76 | +1. Use kubernetes proxy and node metrics: |
| 77 | + |
| 78 | + ```bash |
| 79 | + # Start the kubernetes proxy in a separate terminal |
| 80 | + kubectl proxy |
| 81 | + |
| 82 | + # Access node metrics API |
| 83 | + kubectl get --raw /apis/metrics.k8s.io/v1beta1/nodes |
| 84 | + ``` |
| 85 | + |
| 86 | +2. For more detailed PSI metrics, use the `kubectl debug` feature to create a temporary debug pod: |
| 87 | + |
| 88 | + ```bash |
| 89 | + # Create a debug pod that mounts the host filesystem |
| 90 | + kubectl debug node/<node_name> -it --image=busybox |
| 91 | + |
| 92 | + # Once inside the debug pod, check PSI metrics |
| 93 | + cat /host/proc/pressure/cpu |
| 94 | + ``` |
| 95 | + |
| 96 | + Here's an example command output: |
| 97 | + |
| 98 | + ```output |
| 99 | + some avg10=0.00 avg60=0.00 avg300=0.00 total=0 |
| 100 | + full avg10=0.00 avg60=0.00 avg300=0.00 total=0 |
| 101 | + ``` |
| 102 | + |
| 103 | + - The `some` line indicates the percentage of time at least one task is stalled on CPU. |
| 104 | + - The `full` line indicates the percentage of time all tasks are stalled on CPU. |
| 105 | + |
| 106 | +--- |
| 107 | + |
| 108 | +### Step 2: Review best practices to prevent CPU pressure |
| 109 | + |
| 110 | +Review the following table to learn how to implement best practices for avoiding CPU pressure: |
| 111 | + |
| 112 | +| Best practice | Description | |
| 113 | +|---|---| |
| 114 | +|Focus on PSI metrics instead of utilization|Use PSI metrics as your primary indicator of resource contention rather than CPU utilization percentages. For more information, see [PSI - Pressure Stall Information](https://docs.kernel.org/accounting/psi.html).| |
| 115 | +|Identify pods utilizing the most CPU|Isolate the pods that are utilizing the most CPU and identify solutions to reduce pressure. For more information, see [Troubleshoot high CPU usage in AKS clusters](./identify-high-cpu-consuming-containers-aks.md).| |
| 116 | +|Minimize CPU limits|Consider removing CPU limits and rely on [Linux's Completely Fair Scheduler](https://docs.kernel.org/scheduler/sched-design-CFS.html) with CPU shares based on requests. For more information, see [Resource Management for Pods and Containers](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/).| |
| 117 | +|Use appropriate Quality of Service (QoS) classes|Set the right QoSclass for each pod based on its importance and contention sensitivity. For more information, see [Configure Quality of Service for Pods](https://kubernetes.io/docs/tasks/configure-pod-container/quality-service-pod/).| |
| 118 | +|Optimize pod placement|Use pod anti-affinity rules to avoid placing CPU-intensive workloads on the same nodes. For more information, see [Assigning Pods to Nodes](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/).| |
| 119 | +|Monitor for brief pressure spikes|Short pressure spikes can indicate issues even when average utilization appears acceptable. For more information, see [Resource metrics pipeline](https://kubernetes.io/docs/tasks/debug/debug-cluster/resource-metrics-pipeline/).| |
| 120 | + |
| 121 | +## Key PSI metrics to monitor |
| 122 | + |
| 123 | +> [!NOTE] |
| 124 | +> If a node's CPU usage is moderate but the containers on the node experience CFS throttling, increase the resource limits, or remove them and follow [Linux's Completely Fair Scheduler (CFS)](https://docs.kernel.org/scheduler/sched-design-CFS.html) algorithm. |
| 125 | +
|
| 126 | +### Node-level PSI metrics |
| 127 | + |
| 128 | +- `node_pressure_cpu_waiting_seconds_total`: Cumulative time tasks wait for CPU. |
| 129 | +- `node_cpu_seconds_total`: Traditional CPU utilization for comparison. |
| 130 | + |
| 131 | +### Container-level PSI indicators |
| 132 | + |
| 133 | +- `container_cpu_cfs_throttled_periods_total`: The number of periods a container is throttled. |
| 134 | +- `container_cpu_cfs_throttled_seconds_total`: Total time a container is throttled. |
| 135 | +- Throttling percentage: `rate(container_cpu_cfs_throttled_periods_total[5m]) / rate(container_cpu_cfs_periods_total[5m]) * 100` |
| 136 | + |
| 137 | +## Why using PSI metrics? |
| 138 | + |
| 139 | +AKS uses PSI metrics as an indicator for CPU pressure instead of load average for several reasons: |
| 140 | + |
| 141 | +- In oversized and multi-core nodes, load average often underreports CPU saturation. |
| 142 | +- On chattier and containerized nodes, load average can over-signal, leading to alert fatigue. |
| 143 | +- Since load average doesn't have per-cgroup visibility, noisy pods can hide behind a low system average. |
| 144 | + |
| 145 | +## References |
| 146 | + |
| 147 | +- [Linux PSI documentation](https://docs.kernel.org/accounting/psi.html) |
| 148 | +- [Kubernetes resource management](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/) |
| 149 | +- [AKS performance best practices](/azure/aks/concepts-clusters-workloads) |
| 150 | +- [Enable Prometheus and Grafana](/azure/azure-monitor/containers/kubernetes-monitoring-enable#enable-prometheus-and-grafana) |
| 151 | +- [Quality of Service in Kubernetes](https://kubernetes.io/docs/tasks/configure-pod-container/quality-service-pod/) |
| 152 | +- [Linux Completely Fair Scheduler](https://docs.kernel.org/scheduler/sched-design-CFS.html) |
| 153 | + |
| 154 | +[!INCLUDE [Azure Help Support](../../../includes/azure-help-support.md)] |
0 commit comments