Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
52 changes: 52 additions & 0 deletions docs/rancher/resource-quota.md
Original file line number Diff line number Diff line change
Expand Up @@ -140,3 +140,55 @@ Harvester automatically manages this bottleneck through the following workflow:
To minimize reliance on automatic adjustments, manually update `ResourceQuota` settings whenever overhead configurations are changed. Ultimately, the VM's final memory footprint must always comply with defined namespace limits, regardless of whether it is migrated or cold-rebooted.

:::

## Monitoring ResourceQuota Usage and Alerts

Monitoring resource usage is critical to cluster stability, as high `ResourceQuota` consumption can cause VM startup failures or prevent successful VM migrations. Real-time visibility and proactive notifications ensure you can address capacity constraints before they impact your workloads.

When the [rancher-monitoring](../advanced/addons.md) addon is enabled, it creates the `PrometheusRule` resource `rancher-monitoring-kubernetes-resources` in `cattle-monitoring-system`. This includes the following pre-configured ResourceQuota alert rules.
Comment on lines +146 to +148

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Monitoring resource usage is critical to cluster stability, as high `ResourceQuota` consumption can cause VM startup failures or prevent successful VM migrations. Real-time visibility and proactive notifications ensure you can address capacity constraints before they impact your workloads.
When the [rancher-monitoring](../advanced/addons.md) addon is enabled, it creates the `PrometheusRule` resource `rancher-monitoring-kubernetes-resources` in `cattle-monitoring-system`. This includes the following pre-configured ResourceQuota alert rules.
Resource usage monitoring is critical to cluster stability, as high `ResourceQuota` consumption can prevent virtual machines from starting or migrating. Real-time visibility and proactive notifications enable you to address capacity constraints before they impact active workloads.
When the [rancher-monitoring](../advanced/addons.md) add-on is enabled, it creates a `PrometheusRule` resource named `rancher-monitoring-kubernetes-resources` in the `cattle-monitoring-system` namespace. This includes the following pre-configured `ResourceQuota` alert rules:


### Alert Rule Definitions

| Alert Name | Condition | Duration |
| :--- | :--- | :--- |
| **KubeCPUQuotaOvercommit** | CPU usage > 150% | 5 minutes |
| **KubeMemoryQuotaOvercommit** | Memory usage > 150% | 5 minutes |
| **KubeQuotaAlmostFull** | CPU/Memory usage between 90% and 100% | 15 minutes |
| **KubeQuotaFullyUsed** | CPU/Memory usage = 100% | 15 minutes |
| **KubeQuotaExceeded** | CPU/Memory usage > 100% | 15 minutes |
Comment on lines +150 to +158

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
### Alert Rule Definitions
| Alert Name | Condition | Duration |
| :--- | :--- | :--- |
| **KubeCPUQuotaOvercommit** | CPU usage > 150% | 5 minutes |
| **KubeMemoryQuotaOvercommit** | Memory usage > 150% | 5 minutes |
| **KubeQuotaAlmostFull** | CPU/Memory usage between 90% and 100% | 15 minutes |
| **KubeQuotaFullyUsed** | CPU/Memory usage = 100% | 15 minutes |
| **KubeQuotaExceeded** | CPU/Memory usage > 100% | 15 minutes |
| Alert Name | Condition | Duration |
| :--- | :--- | :--- |
| **KubeCPUQuotaOvercommit** | CPU usage > 150% | 5 minutes |
| **KubeMemoryQuotaOvercommit** | Memory usage > 150% | 5 minutes |
| **KubeQuotaAlmostFull** | CPU or memory usage is between 90% and 100% | 15 minutes |
| **KubeQuotaFullyUsed** | CPU or memory usage = 100% | 15 minutes |
| **KubeQuotaExceeded** | CPU or memory usage > 100% | 15 minutes |


### Verifying ResourceQuota Status

To inspect your current resource usage, use the `kubectl get resourcequota -A` command. This helps correlate your manual observations with the alert triggers.

**Example Output:**

For a `ResourceQuota` named `default-kflsw` in the `quota-test` namespace:

```sh
NAMESPACE NAME REQUEST LIMIT
quota-test default-kflsw limits.cpu: 2015m/3, limits.memory: 4797464313/5000Mi
Comment thread
w13915984028 marked this conversation as resolved.
Comment thread
w13915984028 marked this conversation as resolved.
...
```

In this instance, the memory usage 4797464313(4575Mi) is approximately 91.5% of the 5000Mi limit, which triggers the **KubeQuotaAlmostFull** alert once the 15-minute duration threshold is surpassed.
Comment on lines +162 to +174

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
To inspect your current resource usage, use the `kubectl get resourcequota -A` command. This helps correlate your manual observations with the alert triggers.
**Example Output:**
For a `ResourceQuota` named `default-kflsw` in the `quota-test` namespace:
```sh
NAMESPACE NAME REQUEST LIMIT
quota-test default-kflsw limits.cpu: 2015m/3, limits.memory: 4797464313/5000Mi
...
```
In this instance, the memory usage 4797464313(4575Mi) is approximately 91.5% of the 5000Mi limit, which triggers the **KubeQuotaAlmostFull** alert once the 15-minute duration threshold is surpassed.
You can run the command `kubectl get resourcequota -A` to check your current resource usage. Use the output to correlate your observations with the alert triggers.
**Example:**
A `ResourceQuota` object named `default-kflsw` was created in the `quota-test` namespace.
```sh
NAMESPACE NAME REQUEST LIMIT
quota-test default-kflsw limits.cpu: 2015m/3, limits.memory: 4797464313/5000Mi
...
```
In this example, the memory usage value `4797464313` (equivalent to 4575Mi) is approximately 91.5% of the 5000Mi limit. This triggers the **KubeQuotaAlmostFull** alert once the 15-minute threshold is reached.


### Observing Alerts in Harvester

Once threshold conditions are met and the duration has elapsed, alerts transition to the "Firing" state. Use the following steps to navigate to your monitoring dashboard:

1. **Access Addons:** Go to the **Advanced** section of the Harvester dashboard and select **Addons**.
2. **Open Monitoring:** Click on the **rancher-monitoring** addon.
3. **Launch Prometheus:** Switch to the **Prometheus** tab and click the **Prometheus Graph** link.
4. **Filter Alerts:** Within the Prometheus interface, click the **Alerts** tab.
5. **Locate Rules:** Search for `kubernetes-resources` to view the status of your ResourceQuota alerts.
Comment on lines +176 to +184

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
### Observing Alerts in Harvester
Once threshold conditions are met and the duration has elapsed, alerts transition to the "Firing" state. Use the following steps to navigate to your monitoring dashboard:
1. **Access Addons:** Go to the **Advanced** section of the Harvester dashboard and select **Addons**.
2. **Open Monitoring:** Click on the **rancher-monitoring** addon.
3. **Launch Prometheus:** Switch to the **Prometheus** tab and click the **Prometheus Graph** link.
4. **Filter Alerts:** Within the Prometheus interface, click the **Alerts** tab.
5. **Locate Rules:** Search for `kubernetes-resources` to view the status of your ResourceQuota alerts.
### Viewing Alert Details
Once threshold conditions are met and the duration has elapsed, alerts transition to the _Firing_ state.
To view alert details, perform the following steps:
1. On the Harvester UI, go to **Advanced > Add-ons**.
1. Click **rancher-monitoring**.
1. Go to the **Prometheus** tab and click **Prometheus Graph**.
1. On the Prometheus UI, go to **Alerts**.
1. Search for `kubernetes-resources` to view the status of your ResourceQuota alerts.


*Figure: Alert status for the `ResourceQuota` `default-kflsw`.*

![](/img/v1.9/rancher/resourcequota-alerts.png)

To integrate external notifications, refer to [Configure AlertmanagerConfig](../monitoring/harvester-monitoring.md#configure-alertmanagerconfig-from-webui) to forward alerts to a central monitoring or notification server.

For more information on managing alerts, see [View and Manage Alerts](../monitoring/harvester-monitoring.md#view-and-manage-alerts).

For troubleshooting guidance, refer to the [Prometheus Operator Runbooks](https://runbooks.prometheus-operator.dev/runbooks/kubernetes/kubequotaalmostfull/).
Comment on lines +186 to +194

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
*Figure: Alert status for the `ResourceQuota` `default-kflsw`.*
![](/img/v1.9/rancher/resourcequota-alerts.png)
To integrate external notifications, refer to [Configure AlertmanagerConfig](../monitoring/harvester-monitoring.md#configure-alertmanagerconfig-from-webui) to forward alerts to a central monitoring or notification server.
For more information on managing alerts, see [View and Manage Alerts](../monitoring/harvester-monitoring.md#view-and-manage-alerts).
For troubleshooting guidance, refer to the [Prometheus Operator Runbooks](https://runbooks.prometheus-operator.dev/runbooks/kubernetes/kubequotaalmostfull/).
The following image shows the alert status for the `ResourceQuota` object named `default-kflsw`.
![](/img/v1.9/rancher/resourcequota-alerts.png)
:::tip
You can forward alerts to a central monitoring server by configuring an [`AlertmanagerConfig`](../monitoring/harvester-monitoring.md#configure-alertmanagerconfig-from-webui) resource.
:::
For more information, see [View and Manage Alerts](../monitoring/harvester-monitoring.md#view-and-manage-alerts) in the **Monitoring** section of this document and [KubeQuotaAlmostFull](https://runbooks.prometheus-operator.dev/runbooks/kubernetes/kubequotaalmostfull/) in the `kube-prometheus` runbooks documentation.

Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
52 changes: 52 additions & 0 deletions versioned_docs/version-v1.5/rancher/resource-quota.md
Original file line number Diff line number Diff line change
Expand Up @@ -118,3 +118,55 @@ When a `ResourceQuota` object has the annotation `harvesterhci.io/skipResourceQu
You must set the annotation before the migration starts. If the annotation is set while the values are already being adjusted, Harvester is unable to automatically restore the previous configuration.

:::

## Monitoring ResourceQuota Usage and Alerts

Monitoring resource usage is critical to cluster stability, as high `ResourceQuota` consumption can cause VM startup failures or prevent successful VM migrations. Real-time visibility and proactive notifications ensure you can address capacity constraints before they impact your workloads.

When the [rancher-monitoring](../advanced/addons.md) addon is enabled, it creates the `PrometheusRule` resource `rancher-monitoring-kubernetes-resources` in `cattle-monitoring-system`. This includes the following pre-configured ResourceQuota alert rules.

### Alert Rule Definitions

| Alert Name | Condition | Duration |
| :--- | :--- | :--- |
| **KubeCPUQuotaOvercommit** | CPU usage > 150% | 5 minutes |
| **KubeMemoryQuotaOvercommit** | Memory usage > 150% | 5 minutes |
| **KubeQuotaAlmostFull** | CPU/Memory usage between 90% and 100% | 15 minutes |
| **KubeQuotaFullyUsed** | CPU/Memory usage = 100% | 15 minutes |
| **KubeQuotaExceeded** | CPU/Memory usage > 100% | 15 minutes |

### Verifying ResourceQuota Status

To inspect your current resource usage, use the `kubectl get resourcequota -A` command. This helps correlate your manual observations with the alert triggers.

**Example Output:**

For a `ResourceQuota` named `default-kflsw` in the `quota-test` namespace:

```sh
NAMESPACE NAME REQUEST LIMIT
quota-test default-kflsw limits.cpu: 2015m/3, limits.memory: 4797464313/5000Mi
...
```

In this instance, the memory usage 4797464313(4575Mi) is approximately 91.5% of the 5000Mi limit, which triggers the **KubeQuotaAlmostFull** alert once the 15-minute duration threshold is surpassed.

### Observing Alerts in Harvester

Once threshold conditions are met and the duration has elapsed, alerts transition to the "Firing" state. Use the following steps to navigate to your monitoring dashboard:

1. **Access Addons:** Go to the **Advanced** section of the Harvester dashboard and select **Addons**.
2. **Open Monitoring:** Click on the **rancher-monitoring** addon.
3. **Launch Prometheus:** Switch to the **Prometheus** tab and click the **Prometheus Graph** link.
4. **Filter Alerts:** Within the Prometheus interface, click the **Alerts** tab.
5. **Locate Rules:** Search for `kubernetes-resources` to view the status of your ResourceQuota alerts.

*Figure: Alert status for the `ResourceQuota` `default-kflsw`.*

![](/img/v1.9/rancher/resourcequota-alerts.png)

To integrate external notifications, refer to [Configure AlertmanagerConfig](../monitoring/harvester-monitoring.md#configure-alertmanagerconfig-from-webui) to forward alerts to a central monitoring or notification server.

For more information on managing alerts, see [View and Manage Alerts](../monitoring/harvester-monitoring.md#view-and-manage-alerts).

For troubleshooting guidance, refer to the [Prometheus Operator Runbooks](https://runbooks.prometheus-operator.dev/runbooks/kubernetes/kubequotaalmostfull/).
52 changes: 52 additions & 0 deletions versioned_docs/version-v1.6/rancher/resource-quota.md
Original file line number Diff line number Diff line change
Expand Up @@ -118,3 +118,55 @@ When a `ResourceQuota` object has the annotation `harvesterhci.io/skipResourceQu
You must set the annotation before the migration starts. If the annotation is set while the values are already being adjusted, Harvester is unable to automatically restore the previous configuration.

:::

## Monitoring ResourceQuota Usage and Alerts

Monitoring resource usage is critical to cluster stability, as high `ResourceQuota` consumption can cause VM startup failures or prevent successful VM migrations. Real-time visibility and proactive notifications ensure you can address capacity constraints before they impact your workloads.

When the [rancher-monitoring](../advanced/addons.md) addon is enabled, it creates the `PrometheusRule` resource `rancher-monitoring-kubernetes-resources` in `cattle-monitoring-system`. This includes the following pre-configured ResourceQuota alert rules.

### Alert Rule Definitions

| Alert Name | Condition | Duration |
| :--- | :--- | :--- |
| **KubeCPUQuotaOvercommit** | CPU usage > 150% | 5 minutes |
| **KubeMemoryQuotaOvercommit** | Memory usage > 150% | 5 minutes |
| **KubeQuotaAlmostFull** | CPU/Memory usage between 90% and 100% | 15 minutes |
| **KubeQuotaFullyUsed** | CPU/Memory usage = 100% | 15 minutes |
| **KubeQuotaExceeded** | CPU/Memory usage > 100% | 15 minutes |

### Verifying ResourceQuota Status

To inspect your current resource usage, use the `kubectl get resourcequota -A` command. This helps correlate your manual observations with the alert triggers.

**Example Output:**

For a `ResourceQuota` named `default-kflsw` in the `quota-test` namespace:

```sh
NAMESPACE NAME REQUEST LIMIT
quota-test default-kflsw limits.cpu: 2015m/3, limits.memory: 4797464313/5000Mi
...
```

In this instance, the memory usage 4797464313(4575Mi) is approximately 91.5% of the 5000Mi limit, which triggers the **KubeQuotaAlmostFull** alert once the 15-minute duration threshold is surpassed.

### Observing Alerts in Harvester

Once threshold conditions are met and the duration has elapsed, alerts transition to the "Firing" state. Use the following steps to navigate to your monitoring dashboard:

1. **Access Addons:** Go to the **Advanced** section of the Harvester dashboard and select **Addons**.
2. **Open Monitoring:** Click on the **rancher-monitoring** addon.
3. **Launch Prometheus:** Switch to the **Prometheus** tab and click the **Prometheus Graph** link.
4. **Filter Alerts:** Within the Prometheus interface, click the **Alerts** tab.
5. **Locate Rules:** Search for `kubernetes-resources` to view the status of your ResourceQuota alerts.

*Figure: Alert status for the `ResourceQuota` `default-kflsw`.*

![](/img/v1.9/rancher/resourcequota-alerts.png)

To integrate external notifications, refer to [Configure AlertmanagerConfig](../monitoring/harvester-monitoring.md#configure-alertmanagerconfig-from-webui) to forward alerts to a central monitoring or notification server.

For more information on managing alerts, see [View and Manage Alerts](../monitoring/harvester-monitoring.md#view-and-manage-alerts).

For troubleshooting guidance, refer to the [Prometheus Operator Runbooks](https://runbooks.prometheus-operator.dev/runbooks/kubernetes/kubequotaalmostfull/).
52 changes: 52 additions & 0 deletions versioned_docs/version-v1.7/rancher/resource-quota.md
Original file line number Diff line number Diff line change
Expand Up @@ -118,3 +118,55 @@ When a `ResourceQuota` object has the annotation `harvesterhci.io/skipResourceQu
You must set the annotation before the migration starts. If the annotation is set while the values are already being adjusted, Harvester is unable to automatically restore the previous configuration.

:::

## Monitoring ResourceQuota Usage and Alerts

Monitoring resource usage is critical to cluster stability, as high `ResourceQuota` consumption can cause VM startup failures or prevent successful VM migrations. Real-time visibility and proactive notifications ensure you can address capacity constraints before they impact your workloads.

When the [rancher-monitoring](../advanced/addons.md) addon is enabled, it creates the `PrometheusRule` resource `rancher-monitoring-kubernetes-resources` in `cattle-monitoring-system`. This includes the following pre-configured ResourceQuota alert rules.

### Alert Rule Definitions

| Alert Name | Condition | Duration |
| :--- | :--- | :--- |
| **KubeCPUQuotaOvercommit** | CPU usage > 150% | 5 minutes |
| **KubeMemoryQuotaOvercommit** | Memory usage > 150% | 5 minutes |
| **KubeQuotaAlmostFull** | CPU/Memory usage between 90% and 100% | 15 minutes |
| **KubeQuotaFullyUsed** | CPU/Memory usage = 100% | 15 minutes |
| **KubeQuotaExceeded** | CPU/Memory usage > 100% | 15 minutes |

### Verifying ResourceQuota Status

To inspect your current resource usage, use the `kubectl get resourcequota -A` command. This helps correlate your manual observations with the alert triggers.

**Example Output:**

For a `ResourceQuota` named `default-kflsw` in the `quota-test` namespace:

```sh
NAMESPACE NAME REQUEST LIMIT
quota-test default-kflsw limits.cpu: 2015m/3, limits.memory: 4797464313/5000Mi
...
```

In this instance, the memory usage 4797464313(4575Mi) is approximately 91.5% of the 5000Mi limit, which triggers the **KubeQuotaAlmostFull** alert once the 15-minute duration threshold is surpassed.

### Observing Alerts in Harvester

Once threshold conditions are met and the duration has elapsed, alerts transition to the "Firing" state. Use the following steps to navigate to your monitoring dashboard:

1. **Access Addons:** Go to the **Advanced** section of the Harvester dashboard and select **Addons**.
2. **Open Monitoring:** Click on the **rancher-monitoring** addon.
3. **Launch Prometheus:** Switch to the **Prometheus** tab and click the **Prometheus Graph** link.
4. **Filter Alerts:** Within the Prometheus interface, click the **Alerts** tab.
5. **Locate Rules:** Search for `kubernetes-resources` to view the status of your ResourceQuota alerts.

*Figure: Alert status for the `ResourceQuota` `default-kflsw`.*

![](/img/v1.9/rancher/resourcequota-alerts.png)

To integrate external notifications, refer to [Configure AlertmanagerConfig](../monitoring/harvester-monitoring.md#configure-alertmanagerconfig-from-webui) to forward alerts to a central monitoring or notification server.

For more information on managing alerts, see [View and Manage Alerts](../monitoring/harvester-monitoring.md#view-and-manage-alerts).

For troubleshooting guidance, refer to the [Prometheus Operator Runbooks](https://runbooks.prometheus-operator.dev/runbooks/kubernetes/kubequotaalmostfull/).
52 changes: 52 additions & 0 deletions versioned_docs/version-v1.8/rancher/resource-quota.md
Original file line number Diff line number Diff line change
Expand Up @@ -118,3 +118,55 @@ When a `ResourceQuota` object has the annotation `harvesterhci.io/skipResourceQu
You must set the annotation before the migration starts. If the annotation is set while the values are already being adjusted, Harvester is unable to automatically restore the previous configuration.

:::

## Monitoring ResourceQuota Usage and Alerts

Monitoring resource usage is critical to cluster stability, as high `ResourceQuota` consumption can cause VM startup failures or prevent successful VM migrations. Real-time visibility and proactive notifications ensure you can address capacity constraints before they impact your workloads.

When the [rancher-monitoring](../advanced/addons.md) addon is enabled, it creates the `PrometheusRule` resource `rancher-monitoring-kubernetes-resources` in `cattle-monitoring-system`. This includes the following pre-configured ResourceQuota alert rules.

### Alert Rule Definitions

| Alert Name | Condition | Duration |
| :--- | :--- | :--- |
| **KubeCPUQuotaOvercommit** | CPU usage > 150% | 5 minutes |
| **KubeMemoryQuotaOvercommit** | Memory usage > 150% | 5 minutes |
| **KubeQuotaAlmostFull** | CPU/Memory usage between 90% and 100% | 15 minutes |
| **KubeQuotaFullyUsed** | CPU/Memory usage = 100% | 15 minutes |
| **KubeQuotaExceeded** | CPU/Memory usage > 100% | 15 minutes |

### Verifying ResourceQuota Status

To inspect your current resource usage, use the `kubectl get resourcequota -A` command. This helps correlate your manual observations with the alert triggers.

**Example Output:**

For a `ResourceQuota` named `default-kflsw` in the `quota-test` namespace:

```sh
NAMESPACE NAME REQUEST LIMIT
quota-test default-kflsw limits.cpu: 2015m/3, limits.memory: 4797464313/5000Mi
...
```

In this instance, the memory usage 4797464313(4575Mi) is approximately 91.5% of the 5000Mi limit, which triggers the **KubeQuotaAlmostFull** alert once the 15-minute duration threshold is surpassed.

### Observing Alerts in Harvester

Once threshold conditions are met and the duration has elapsed, alerts transition to the "Firing" state. Use the following steps to navigate to your monitoring dashboard:

1. **Access Addons:** Go to the **Advanced** section of the Harvester dashboard and select **Addons**.
2. **Open Monitoring:** Click on the **rancher-monitoring** addon.
3. **Launch Prometheus:** Switch to the **Prometheus** tab and click the **Prometheus Graph** link.
4. **Filter Alerts:** Within the Prometheus interface, click the **Alerts** tab.
5. **Locate Rules:** Search for `kubernetes-resources` to view the status of your ResourceQuota alerts.

*Figure: Alert status for the `ResourceQuota` `default-kflsw`.*

![](/img/v1.9/rancher/resourcequota-alerts.png)

To integrate external notifications, refer to [Configure AlertmanagerConfig](../monitoring/harvester-monitoring.md#configure-alertmanagerconfig-from-webui) to forward alerts to a central monitoring or notification server.

For more information on managing alerts, see [View and Manage Alerts](../monitoring/harvester-monitoring.md#view-and-manage-alerts).

For troubleshooting guidance, refer to the [Prometheus Operator Runbooks](https://runbooks.prometheus-operator.dev/runbooks/kubernetes/kubequotaalmostfull/).
Loading