Skip to content

Commit f78f361

Browse files
committed
Add resorucequota related monitoring alerts
Signed-off-by: Jian Wang <[email protected]>
1 parent c89b016 commit f78f361

2 files changed

Lines changed: 49 additions & 0 deletions

File tree

docs/rancher/resource-quota.md

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -118,3 +118,52 @@ When a `ResourceQuota` object has the annotation `harvesterhci.io/skipResourceQu
118118
You must set the annotation before the migration starts. If the annotation is set while the values are already being adjusted, Harvester is unable to automatically restore the previous configuration.
119119

120120
:::
121+
122+
## Monitoring ResourceQuota Usage
123+
124+
When the [rancher-monitoring](../advanced/addons) addon is enabled, it installs the CRD `prometheusrules` object `rancher-monitoring-kubernetes-resources` in `cattle-monitoring-system`. This includes the following pre-configured resourcequota alert rules.
125+
126+
### Alert Rule Definitions
127+
128+
| Alert Name | Condition | Duration |
129+
| :--- | :--- | :--- |
130+
| **KubeCPUQuotaOvercommit** | CPU usage > 150% | 5 minutes |
131+
| **KubeMemoryQuotaOvercommit** | Memory usage > 150% | 5 minutes |
132+
| **KubeQuotaAlmostFull** | CPU/Memory usage between 90% and 100% | 15 minutes |
133+
| **KubeQuotaFullyUsed** | CPU/Memory usage = 100% | 15 minutes |
134+
| **KubeQuotaExceeded** | CPU/Memory usage > 100% | 15 minutes |
135+
136+
### Verifying ResourceQuota Status
137+
138+
To inspect your current resource usage, use the `kubectl get resourcequota -n <namespace>` command. This helps correlate your manual observations with the alert triggers.
139+
140+
**Example Output:**
141+
142+
For a resource quota named `default-kflsw` in the `quota-test` namespace:
143+
144+
```sh
145+
NAMESPACE NAME REQUEST LIMIT
146+
quota-test default-kflsw limits.cpu: 2015m/3, limits.memory: 4797464313/5000Mi
147+
```
148+
149+
In this instance, the memory usage is approximately 95% of the 5000Mi limit, which triggers the **KubeQuotaAlmostFull** alert once the 15-minute duration threshold is surpassed.
150+
151+
### Observing Alerts in Harvester
152+
153+
Once threshold conditions are met and the duration has elapsed, alerts transition to the "Firing" state. Use the following steps to navigate to your monitoring dashboard:
154+
155+
1. **Access Addons:** Go to the **Advanced** section of the Harvester dashboard and select **Addons**.
156+
2. **Open Monitoring:** Click on the **rancher-monitoring** addon.
157+
3. **Launch Prometheus:** Switch to the **Prometheus** tab and click the **Prometheus Graph** link.
158+
4. **Filter Alerts:** Within the Prometheus interface, click the **Alerts** tab.
159+
5. **Locate Rules:** Search for `kubernetes-resources` to view the status of your ResourceQuota alerts.
160+
161+
*Figure: Alert status for the resourcequota `default-kflsw`.*
162+
163+
![](/img/v1.9/rancher/resourcequota-alerts.png)
164+
165+
### Additional Resources
166+
167+
For more information on managing alerts, see [View and Manage Alerts](../monitoring/harvester-monitoring#view-and-manage-alerts).
168+
169+
To integrate external notifications, refer to [Configure AlertmanagerConfig](../monitoring/harvester-monitoring#configure-alertmanagerconfig-from-webui) to forward alerts to a central monitoring or notification server.
367 KB
Loading

0 commit comments

Comments
 (0)