| title | Private Cloud Maintenance |
|---|---|
| description | Ensuring seamless, reliably maintaintenance of Azure VMware Solution private cloud |
| ms.topic | concept-article |
| ms.service | azure-vmware |
| ms.date | 03/25/2025 |
Azure VMware Solution undertakes periodic maintenance of the private cloud. This maintenance includes security patches, minor and major updates to VMware software stack. This page describes the host monitoring, remediation, and mandatory steps that keep the private cloud ready for maintenance.
[!INCLUDE vmware-software-update-frequency]
Azure VMware Solution continuously monitors the health of both the VMware components and underlay. When Azure VMware Solution detects a failure, it takes action to repair the failed components. When Azure VMware Solution detects a degradation or failure on an Azure VMware Solution node, it triggers the host remediation process.
Host remediation involves replacing the faulty node with a new healthy node in the cluster. Then, when possible, the faulty host is placed in VMware vSphere maintenance mode. VMware vSphere vMotion moves the VMs off the faulty host to other available servers in the cluster, potentially allowing zero downtime for live migration of workloads. If the faulty host can't be placed in maintenance mode, the host is removed from the cluster. Before the faulty host is removed, the customer workloads are migrated to a newly added host.
Tip
Customer communication: An email is sent to the customer's email address before the replacement is initiated and again after the replacement is successful.
To receive emails related to host replacement, you need to be added to any of the following Azure Role-Based Access Control (RBAC) roles in the subscription: 'ServiceAdmin', 'CoAdmin', 'Owner', 'Contributor'.
Azure VMware Solution monitors the following conditions on the host:
- Processor status
- Memory status
- Connection and power state
- Hardware fan status
- Network connectivity loss
- Hardware system board status
- Errors occurred on one or more disks of a vSAN host
- Hardware voltage
- Hardware temperature status
- Hardware power status
- Storage status
- Connection failure
The following actions are necessary for ensuring host maintenance operations are carried out successfully:
- vSAN storage utilization: To maintain Service Level Agreement (SLA), ensure that your vSphere cluster's storage space utilization remains below 75%. If the utilization exceeds 75%, upgrades may take longer than expected or fail entirely. If your storage utilization exceeds 75%, consider adding a node to expand the cluster and prevent potential downtime during upgrades.
- Distributed Resource Scheduler (DRS) rules: DRS VM-VM anti-affinity rules must be configured in a way to have at least (N+1) hosts in the cluster, where N is the number of VMs part of DRS rule.
- Failures To Tolerate (FTT) violation: To prevent data loss, change VMs configured with a vSAN storage policy for Failures to Tolerate (FTT) of 0 to a vSAN storage policy compliant with Microsoft SLA (FTT=1 for up to five hosts in a cluster and FTT=2 for six or more hosts in a cluster) and ensure host maintenance operations can carried out seamlessly.
- Remove VM CD-ROM mounts: VMs mounted with "Emulate mode" CD-ROMs block host maintenance. Ensure CD-ROMs are mounted in "Passthrough mode".
- Serial/parallel port or external device: If you're using an image file (ISO, FLP, etc.), ensure that it's accessible from all ESXi hosts in the cluster. Store the files on a datastore that are shared between all ESXi Servers that participate in the vMotion of the virtual machine. For more information, see Broadcom KB article.
- Orphaned VMs: In the case of an orphaned virtual machine, the Virtual Machine (VM) needs to be either re-registered if possible (if it hasn't been deleted) or removed from inventory. For more information, see Broadcom KB article.
- SCSI shared controller: When using SCSI bus sharing, use with bus type as "Physical" for VMs. VMs connected to Virtual SCSI controllers will be powered-off. For more information, see Broadcom KB article.
- Third-party VMs & applications: For third-party VMs & applications:
- Ensure that third-party solutions deployed on Azure VMware Solution are compliant and don't interfere with maintenance operations.
- Ensure that the VM isn’t installed with a VM-Host "Must run" DRS rule. Additionally, verify that these applications are compatible with upcoming versions of the VMware stack.
- Consult with your solution vendor and update in advance if necessary to maintain compatibility post-upgrade.
Important
If any of these maintenance blocking configurations exist on an Azure VMware Solution host, you'll receive alerts on your Resource Health dashboard. To ensure unhealthy hosts are replaced and upgrades succeed, such blocking configurations will be mitigated by taking appropriate remediation steps to maintain the availability of your private cloud. In some cases, these remediation steps would include powering off a VM and migrating it to another host and then powering it on, which might briefly disrupt the application running on the VM.
| Error Code | Error Details | Recommended Action |
|---|---|---|
| EPC_CDROM_EMULATEMODE | This error is encountered when CD-ROM on the Virtual Machine uses emulate mode, whose ISO image isn't accessible. | Follow this KB article for the removal of any CDROM mounted on customer's workload Virtual Machines in emulate mode or detach ISO. It's recommended to use "Passthrough mode" for mounting any CD-ROM. |
| EPC_DRSOVERRIDERULE | This error is encountered when there's a Virtual Machine with DRS Override set to "Disabled" mode. | VM shouldn't block vMotion while putting host into maintenance. Set Partially Automated DRS rules for the VM. Refer to this document to know more about VM placement policies. |
| EPC_SCSIDEVICE_SHARINGMODE | This error is encountered when a Virtual Machine is configured to use SCSI controller with bus-sharing in "virtual" mode. | Follow this KB article for the removal of any SCSI controller engaged in bus-sharing in virtual mode is attached to VMs. |
| EPC_DATASTORE_INACCESSIBLE | This error is encountered when any external datastore attached to AVS Private Cloud becomes inaccessible. | Follow this article for the removal of any stale Datastore attached to cluster |
| EPC_NWADAPTER_STALE | This error is encountered when connected Network interface on the Virtual Machine uses network adapter, which becomes inaccessible. | Follow this KB article for the removal of any stale N/W adapters attached to Virtual Machines. |
| EPC_SERIAL_PORT | This error is encountered when a Virtual Machine’s serial port is connected to a device that can't be accessed on the destination host. | If you're using an image file (ISO, FLP, and so on), ensure that it's accessible from all ESXi servers on the cluster. Store the files on a data store that is shared between all ESXi servers that participate in vMotion of the virtual machine. Refer to this KB article from Broadcom for more information. |
| EPC_HARDWARE_DEVICE | This error is encountered when a Virtual Machine’s parallel Port/USB Device is connected to a device can't be accessed on the destination host. | If you're using an image file (ISO, FLP, and so on), ensure that it's accessible from all ESXi servers of the cluster. Store the files on a data store that is shared between all ESXi servers that participate in the vMotion of the virtual machine. Refer to this KB article from Broadcom for more information. |
| EPC_INVALIDVM / EPC_ORPHANVM | This error is encountered when there's an orphaned or Invalid VM is present in the inventory. | Ensure all your Virtual Machines are accessible to the vCenter. Refer to this KB article for more information. |
| EPC_VMHOSTDRSRULE | This error is encountered when there's a Virtual Machine with Host affinity/anti-affinity DRS rule. | VM shouldn't block VMware vMotion while putting a host into maintenance mode. Set "should rules" for VM-Host affinity. Refer to this document for more information. |
| EPC_FTT_ZERO | This error is encountered when a virtual machine has "Failures to Tolerate" as 0 or "No data redundancy". | Follow this KB article to configure FTT as 1 or 2 for the Virtual Machine. |
| EPC_FTTVIOLATION | This error is encountered when a cluster does not have the minimum number of hosts that the storage policy needs. | Add hosts as needed by the storage policy or change the VM FTT policy to support putting the host into maintenance mode. Refer to this KB article to know more about FTT policy. |
| ERECOMMENDATION_CLUSTER_SIZE | This recommendation indicates a cluster in the private cloud has 14 or more hosts. AVS supports a maximum of 16 hosts in a cluster. | Create a new cluster for new any new hosts that may be required. |
| ERECOMMENDATION_PRIVATECLOUD_SIZE | This recommendation indicates a private cloud has 90 or more hosts. AVS supports a maximum of 96 hosts in a private cloud. | Consider creating a new private cloud for any new hosts and distribute hosts across the private clouds as necessary. |
| ERECOMMENDATION_VCENTER_SCALE | This recommendation identifies that the vCenter virtual machine is provisioned with fewer CPU cores or less memory than recommended for the current virtual machine count within the private cloud. | Open up a support request to have the vCenter memory and CPU increased. |
Note
Azure VMware Solution tenant admins must not edit or delete the previously defined VMware vCenter Server alarms because they're managed by the Azure VMware Solution control plane on vCenter Server. These alarms are used by Azure VMware Solution monitoring to trigger the Azure VMware Solution host remediation process.
Now that you've learnt how to ensure seamless Azure VMware Solution private cloud maintenance, you might want to learn about: