Add resorucequota related monitoring alerts

[w13915984028]

Problem:

How to observe the monitoring and alerts upon resourcequota.

Solution:

Explain why and how to observe resourcequota related alerts.

Related Issue(s):

harvester/harvester#10093
harvester/harvester#10267
#981

Test plan:

Additional documentation or context

A simple webhook server to receive the alerts from Harvester cluster, it got below alert, with key information:

'namespace': 'quota-test2'
'resourcequota': 'default-5k2gw'
'description': 'Namespace quota-test2 is using 91.5% of its limits.memory quota on cluster .`

{'receiver': 'quota-test2/amc-example2/amc-webhook-receiver', 'status': 'firing', 'alerts': [{'status': 'firing', 'labels': {'alertname': 'KubeQuotaAlmostFull', 'container': 'kube-state-metrics', 'endpoint': 'http', 'namespace': 'quota-test2', 'pod': 'rancher-monitoring-kube-state-metrics-9585db8c8-2kbbq', 'prometheus': 'cattle-monitoring-system/rancher-monitoring-prometheus', 'resource': 'limits.memory', 'resourcequota': 'default-5k2gw', 'service': 'rancher-monitoring-kube-state-metrics', 'severity': 'info'}, 'annotations': {'description': 'Namespace quota-test2 is using 91.5% of its limits.memory quota on cluster .', 'runbook_url': 'https://runbooks.prometheus-operator.dev/runbooks/kubernetes/kubequotaalmostfull', 'summary': 'Namespace quota is going to be full.'}, 'startsAt': '2026-06-03T16:23:00.904Z', 'endsAt': '0001-01-01T00:00:00Z', 'generatorURL': 'https://192.168.122.141/api/v1/namespaces/cattle-monitoring-system/services/http:rancher-monitoring-prometheus:9090/proxy/graph?g0.expr=kube_resourcequota%7Bjob%3D%22kube-state-metrics%22%2Ctype%3D%22used%22%7D+%2F+ignoring+%28instance%2C+job%2C+type%29+%28kube_resourcequota%7Bjob%3D%22kube-state-metrics%22%2Ctype%3D%22hard%22%7D+%3E+0%29+%3E+0.9+%3C+1&g0.tab=1', 'fingerprint': '686d97b5cf5e722f'}], 'groupLabels': {'alertname': 'KubeQuotaAlmostFull'}, 'commonLabels': {'alertname': 'KubeQuotaAlmostFull', 'container': 'kube-state-metrics', 'endpoint': 'http', 'namespace': 'quota-test2', 'pod': 'rancher-monitoring-kube-state-metrics-9585db8c8-2kbbq', 'prometheus': 'cattle-monitoring-system/rancher-monitoring-prometheus', 'resource': 'limits.memory', 'resourcequota': 'default-5k2gw', 'service': 'rancher-monitoring-kube-state-metrics', 'severity': 'info'}, 'commonAnnotations': {'description': 'Namespace quota-test2 is using 91.5% of its limits.memory quota on cluster .', 'runbook_url': 'https://runbooks.prometheus-operator.dev/runbooks/kubernetes/kubequotaalmostfull', 'summary': 'Namespace quota is going to be full.'}, 'externalURL': 'https://192.168.122.141/api/v1/namespaces/cattle-monitoring-system/services/http:rancher-monitoring-alertmanager:9093/proxy', 'version': '4', 'groupKey': '{}/{namespace="quota-test2"}:{alertname="KubeQuotaAlmostFull"}', 'truncatedAlerts': 0}

[Copilot]

Pull request overview

This PR updates the Resource Quotas documentation to explain how to observe ResourceQuota monitoring/alerts when the rancher-monitoring addon is enabled.

Changes:

• Adds a new section describing pre-configured ResourceQuota-related Prometheus alert rules.
• Adds step-by-step instructions for viewing ResourceQuota alerts in the Harvester UI / Prometheus.
• Adds an example kubectl output snippet and a screenshot reference for alert status.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[github-actions]

Name	Link
🔨 Latest commit	a6645b3
😎 Deploy Preview	https://6a208a633ab7478dea5b4a3e--harvester-preview.netlify.app

[Copilot]

Pull request overview

Copilot reviewed 1 out of 2 changed files in this pull request and generated 4 comments.

[Copilot]

Pull request overview

Copilot reviewed 1 out of 2 changed files in this pull request and generated 2 comments.

[jillian-maroket]

Review done

[jillian-maroket]

Suggested change

	Monitoring resource usage is critical to cluster stability, as high `ResourceQuota` consumption can cause VM startup failures or prevent successful VM migrations. Real-time visibility and proactive notifications ensure you can address capacity constraints before they impact your workloads.
	When the [rancher-monitoring](../advanced/addons.md) addon is enabled, it creates the `PrometheusRule` resource `rancher-monitoring-kubernetes-resources` in `cattle-monitoring-system`. This includes the following pre-configured ResourceQuota alert rules.
	Resource usage monitoring is critical to cluster stability, as high `ResourceQuota` consumption can prevent virtual machines from starting or migrating. Real-time visibility and proactive notifications enable you to address capacity constraints before they impact active workloads.
	When the [rancher-monitoring](../advanced/addons.md) add-on is enabled, it creates a `PrometheusRule` resource named `rancher-monitoring-kubernetes-resources` in the `cattle-monitoring-system` namespace. This includes the following pre-configured `ResourceQuota` alert rules:

[jillian-maroket]

Suggested change

	### Alert Rule Definitions
	| Alert Name | Condition | Duration |
	| :--- | :--- | :--- |
	| **KubeCPUQuotaOvercommit** | CPU usage > 150% | 5 minutes |
	| **KubeMemoryQuotaOvercommit** | Memory usage > 150% | 5 minutes |
	| **KubeQuotaAlmostFull** | CPU/Memory usage between 90% and 100% | 15 minutes |
	| **KubeQuotaFullyUsed** | CPU/Memory usage = 100% | 15 minutes |
	| **KubeQuotaExceeded** | CPU/Memory usage > 100% | 15 minutes |
	| Alert Name | Condition | Duration |
	| :--- | :--- | :--- |
	| **KubeCPUQuotaOvercommit** | CPU usage > 150% | 5 minutes |
	| **KubeMemoryQuotaOvercommit** | Memory usage > 150% | 5 minutes |
	| **KubeQuotaAlmostFull** | CPU or memory usage is between 90% and 100% | 15 minutes |
	| **KubeQuotaFullyUsed** | CPU or memory usage = 100% | 15 minutes |
	| **KubeQuotaExceeded** | CPU or memory usage > 100% | 15 minutes |

[jillian-maroket]

Suggested change

	To inspect your current resource usage, use the `kubectl get resourcequota -A` command. This helps correlate your manual observations with the alert triggers.
	**Example Output:**
	For a `ResourceQuota` named `default-kflsw` in the `quota-test` namespace:
	```sh
	NAMESPACE NAME REQUEST LIMIT
	quota-test default-kflsw limits.cpu: 2015m/3, limits.memory: 4797464313/5000Mi
	...
	```
	In this instance, the memory usage 4797464313(4575Mi) is approximately 91.5% of the 5000Mi limit, which triggers the **KubeQuotaAlmostFull** alert once the 15-minute duration threshold is surpassed.
	You can run the command `kubectl get resourcequota -A` to check your current resource usage. Use the output to correlate your observations with the alert triggers.
	**Example:**
	A `ResourceQuota` object named `default-kflsw` was created in the `quota-test` namespace.
	```sh
	NAMESPACE NAME REQUEST LIMIT
	quota-test default-kflsw limits.cpu: 2015m/3, limits.memory: 4797464313/5000Mi
	...
	```
	In this example, the memory usage value `4797464313` (equivalent to 4575Mi) is approximately 91.5% of the 5000Mi limit. This triggers the **KubeQuotaAlmostFull** alert once the 15-minute threshold is reached.

[jillian-maroket]

Suggested change

	### Observing Alerts in Harvester
	Once threshold conditions are met and the duration has elapsed, alerts transition to the "Firing" state. Use the following steps to navigate to your monitoring dashboard:
	1. **Access Addons:** Go to the **Advanced** section of the Harvester dashboard and select **Addons**.
	2. **Open Monitoring:** Click on the **rancher-monitoring** addon.
	3. **Launch Prometheus:** Switch to the **Prometheus** tab and click the **Prometheus Graph** link.
	4. **Filter Alerts:** Within the Prometheus interface, click the **Alerts** tab.
	5. **Locate Rules:** Search for `kubernetes-resources` to view the status of your ResourceQuota alerts.
	### Viewing Alert Details
	Once threshold conditions are met and the duration has elapsed, alerts transition to the _Firing_ state.
	To view alert details, perform the following steps:
	1. On the Harvester UI, go to **Advanced > Add-ons**.
	1. Click **rancher-monitoring**.
	1. Go to the **Prometheus** tab and click **Prometheus Graph**.
	1. On the Prometheus UI, go to **Alerts**.
	1. Search for `kubernetes-resources` to view the status of your ResourceQuota alerts.

[jillian-maroket]

Suggested change

	*Figure: Alert status for the `ResourceQuota` `default-kflsw`.*
	
	To integrate external notifications, refer to [Configure AlertmanagerConfig](../monitoring/harvester-monitoring.md#configure-alertmanagerconfig-from-webui) to forward alerts to a central monitoring or notification server.
	For more information on managing alerts, see [View and Manage Alerts](../monitoring/harvester-monitoring.md#view-and-manage-alerts).
	For troubleshooting guidance, refer to the [Prometheus Operator Runbooks](https://runbooks.prometheus-operator.dev/runbooks/kubernetes/kubequotaalmostfull/).
	The following image shows the alert status for the `ResourceQuota` object named `default-kflsw`.
	
	:::tip
	You can forward alerts to a central monitoring server by configuring an [`AlertmanagerConfig`](../monitoring/harvester-monitoring.md#configure-alertmanagerconfig-from-webui) resource.
	:::
	For more information, see [View and Manage Alerts](../monitoring/harvester-monitoring.md#view-and-manage-alerts) in the **Monitoring** section of this document and [KubeQuotaAlmostFull](https://runbooks.prometheus-operator.dev/runbooks/kubernetes/kubequotaalmostfull/) in the `kube-prometheus` runbooks documentation.