Skip to content

Commit a933248

Browse files
v-liannav-lianna
committed
Create hyper-v-virtual-machine-backup-checkpoint-storage.md
1 parent 3888125 commit a933248

1 file changed

Lines changed: 190 additions & 0 deletions

File tree

‎support/windows-server/virtualization/hyper-v-virtual-machine-backup-checkpoint-storage.md‎

Lines changed: 190 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,190 @@
1+
---
2+
title: Troubleshooting Hyper-V Virtual Machine Backup, Checkpoint, and Storage Failures
3+
description: Provides a comprehensive guide to troubleshooting common issues encountered with Hyper-V virtual machine (VM) backup, checkpoint management, and storage failures in Windows Server environments.
4+
ms.date: 08/19/2025
5+
manager: dcscontentpm
6+
audience: itpro
7+
ms.topic: troubleshooting
8+
ms.reviewer: kaushika, jeffhuge, v-lianna
9+
ms.custom:
10+
- sap:virtualization and hyper-v\backup and restore of virtual machines
11+
- pcy:WinComm Storage High Avail
12+
---
13+
# Troubleshooting Hyper-V Virtual Machine Backup, Checkpoint, and Storage Failures
14+
15+
This article provides a comprehensive guide to troubleshooting common issues encountered with Hyper-V virtual machine (VM) backup, checkpoint management, and storage failures in Windows Server environments (2016, 2019, and 2022). These issues often arise during backup/restore operations, checkpoint creation or merging, VM startup, and storage connectivity. They can impact both production and disaster recovery workflows in standalone Hyper-V hosts, clustered setups, and environments integrated with third-party backup tools. Timely identification and resolution are critical to maintaining VM availability, data integrity, and backup reliability.
16+
17+
## Symptoms
18+
19+
You might encounter the following symptoms in affected environments:
20+
21+
### Backup and restore failures
22+
23+
- Backups fail to complete or hang indefinitely.
24+
- Restore operations from checkpoints or images fail.
25+
- Error messages from backup software (such as Veeam, Druva, Rubrik, or Veritas) including:
26+
27+
- *"Failed to create VM recovery checkpoint … error code: 32768"*
28+
- *"Could not create backup checkpoint for virtual machine - Error -0x800423F2"*
29+
- *"There was a problem recovering your PC"*
30+
- *"Import failed. … The process cannot access the file because it is being used by another process. (0x80070020)"*
31+
32+
- Event log entries such as Event ID 8229 (host) and 12293 (guest) indicating Volume Shadow Copy Service (VSS) writer errors.
33+
34+
### Checkpoint and snapshot issues
35+
36+
- Unable to create new checkpoints or merge existing ones.
37+
- Accumulation of AVHDX files on disk, even when not visible in Hyper-V Manager.
38+
- Errors such as: *"The process cannot access the file because it is being used by another process. (0x80070020)"*.
39+
- Orphaned or hidden checkpoints blocking backup or VM operations.
40+
- VMs stuck in *Saved*, *Paused*, or *Critical* states.
41+
42+
### VM startup or migration failures
43+
44+
- VMs fail to start after host reboots, patching, or disk migration.
45+
- Errors such as:
46+
- *"The system cannot find the file specified. (0x2)"*
47+
- *"An error occurred while attempting to start the selected VM. Unavailable could not initialize and update VM configuration failed."*
48+
- VMs not visible in Hyper-V Manager or Failover Cluster Manager.
49+
- VM migration, export, or import fails due to missing or corrupted files.
50+
51+
### Storage and file system errors
52+
53+
- Event logs showing IDs 9, 39, 129, 3280, 55, 51, or 513 related to storage, disk, or file system corruption.
54+
- Storage volume running out of space due to large AVHDX files.
55+
- VHD/VHDX files missing, inaccessible, or corrupted.
56+
- Errors such as:
57+
- *"Cannot connect to virtual machine configuration storage"*
58+
- *"The virtual machine is not in a valid state to perform the operation."*
59+
60+
### User experience
61+
62+
- Inability to perform backups or restores, leading to potential data loss.
63+
- Unexpected VM downtime or loss of access.
64+
- Orphaned disk files causing storage exhaustion.
65+
- Persistent warning or error states in management consoles.
66+
67+
## Cause
68+
69+
The root causes of these issues can be grouped into the following categories:
70+
71+
### Checkpoint and VSS-related issues
72+
73+
- **Orphaned or unmerged checkpoints**: Caused by incomplete backup operations, hidden/invisible checkpoints, or AVHDX chain corruption.
74+
- **VSS writer failures or timeouts**: VSS writers in failed or timed-out states, often due to overlapping backup jobs or VSS operations.
75+
- **Third-party backup integration**: Misconfigurations or incompatibilities between Hyper-V and backup tools such as Veeam, Druva, or Veritas.
76+
77+
### File system and storage problems
78+
79+
- **Disk or volume corruption**: Metadata errors on NTFS/ReFS volumes triggered by failed merges or storage I/O issues.
80+
- **Missing or corrupted VM files**: Loss or corruption of VHD, AVHDX, or configuration files due to failed merges or hardware issues.
81+
- **Storage connectivity problems**: Issues with iSCSI, Cluster Shared Volumes (CSV), or Storage Spaces Direct (S2D) leading to unavailability or driver/firmware failures.
82+
83+
### Permissions and policy misconfiguration
84+
85+
- **Insufficient permissions**: Missing "Log on as a Service" rights for NT Virtual Machine accounts or incorrect NTFS permissions.
86+
- **Antivirus interference**: AV filter drivers blocking access to disk files or interfering with checkpoint/backup operations.
87+
- **Group Policy misconfigurations**: Changes removing required permissions or service rights.
88+
89+
### Cluster and configuration issues
90+
91+
- **Cluster role or metadata corruption**: Caused by improper failovers, role removal, or unsynchronized cluster states.
92+
- **Configuration version mismatch**: VM config versions below the required minimum after migrations or upgrades.
93+
- **Incorrect VM or storage path settings**: Misconfigured paths or mismatched network adapters after a migration.
94+
95+
### Operational and environmental causes
96+
97+
- **Improper maintenance**: Rebooting cluster nodes without draining roles or placing them in maintenance mode.
98+
- **Simultaneous backup conflicts**: Overlapping backup jobs causing VSS contention or file corruption.
99+
- **Hardware or driver issues**: Outdated network/storage drivers or hardware instability.
100+
101+
## Resolution
102+
103+
Follow these steps to resolve issues based on their root causes:
104+
105+
### Checkpoint and AVHDX file management
106+
107+
1. **Identify orphaned checkpoints**: - Run the following PowerShell command:powershellGet-VMSnapshot -VMName
108+
- Inspect the VM disk folder for AVHDX files not listed in Hyper-V Manager.
109+
2. **Merge orphaned AVHDX files**: - Shut down the affected VM.
110+
- In Hyper-V Manager: - Go to **Actions > Edit Disk > Select AVHDX > Merge > To Parent Disk**.
111+
- Repeat the process until only the parent VHDX file remains.
112+
3. **Advanced PowerShell merge** (if needed):
113+
114+
```plaintext
115+
Merge-VHD -Path <AVHDXPath> -DestinationPath <VHDXPath>
116+
```
117+
118+
4. **Reattach and start the VM**: - Verify the VM points to the correct (merged) VHDX file.
119+
- Start the VM and confirm functionality.
120+
121+
### Backup/restore and VSS writer troubleshooting
122+
123+
1. **Restart VSS services**:
124+
125+
```plaintext
126+
net stop vss
127+
net start vss
128+
net stop swprv
129+
net start swprv
130+
```
131+
132+
2. **Check VSS writers**: - Run: vssadmin list writers and resolve failed writers by restarting related services or rebooting the host.
133+
3. **Clear stuck checkpoints**: - Use vendor-specific tools, such as nbhypervtool.exe deleteNbuCheckpoints -vmguid <vmguid> for Veritas.
134+
4. **Stagger backup schedules** to avoid overlapping VSS operations.
135+
136+
### File system and storage repair
137+
138+
1. **Run disk and file system repairs**:
139+
140+
```plaintext
141+
chkdsk <drive:> /f
142+
143+
Or for ReFS volumes:
144+
```powershell
145+
Repair-Volume <drive:>
146+
```
147+
148+
2. **Restore missing files**: - Recover from the latest backup if files are irrecoverable.
149+
3. **Validate storage health**: - Check iSCSI, CSV, or S2D connectivity and ensure sufficient space.
150+
- Update storage drivers and firmware.
151+
152+
### Permission and policy corrections
153+
154+
1. **Assign required rights**: - Ensure "NT Virtual Machine\Virtual Machines" has "Log on as a Service" rights via Group Policy or Local Security Policy.
155+
2. **Update NTFS permissions**: - Use icacls to verify and grant Full Control to VM service accounts.
156+
3. **Configure antivirus exclusions**: - Follow the [Microsoft antivirus exclusions for Hyper-V](https://learn.microsoft.com/troubleshoot/windows-server/virtualization/antivirus-exclusions-for-hyper-v-hosts).
157+
158+
### Cluster and VM configuration fixes
159+
160+
1. **Resynchronize cluster metadata**: - Use Failover Cluster Manager for VM operations.
161+
- Perform live migrations to force metadata sync and clear warnings.
162+
2. **Restore or import missing VMs**: - In Hyper-V Manager, use **Import Virtual Machine** and locate the exported/imported files.
163+
3. **Update VM configuration version**: - In Hyper-V Manager: Right-click the VM > **Upgrade Configuration Version**.Or use PowerShell:```powershellUpdate-VMVersion
164+
4. **Correct disk or network settings**: - Edit VM settings to ensure correct disk and network adapter assignments.
165+
166+
### Other repairs and checks
167+
168+
1. **Clean up saved states**: - Delete .vmrs and .vmgs files in the VM folder if stuck in a "Saved" state.
169+
2. **Check and fix cluster resource states**: - Use PowerShell cmdlets like Get-ClusterResource to inspect and repair cluster roles.
170+
3. **Reboot the host** to clear stale file locks if needed.
171+
172+
## Data collection
173+
174+
To assist with troubleshooting, collect the following:
175+
176+
- **Event logs**: Application, System, and Hyper-V-specific logs.
177+
- **PowerShell diagnostic commands**: - Get-VHDChain, Get-VMSnapshot, Get-VM, Update-VMVersion, icacls, Merge-VHD, Set-VHD.
178+
- **VSS tools**: - vssadmin list writers, vssadmin list providers.
179+
- **Diagnostic tools**: - Process Monitor (Procmon), Handle.exe, Process Explorer, and Troubleshooting Support Script (TSS).
180+
- **Cluster logs**: Failover clustering logs and System Center Virtual Machine Manager (SCVMM) job history.
181+
- **Vendor-specific tools**: For checkpoint/snapshot cleanup.
182+
183+
## References
184+
185+
- [Microsoft Docs: Export and import virtual machines](https://learn.microsoft.com/windows-server/virtualization/hyper-v/deploy/export-and-import-virtual-machines)
186+
- [Hyper-V backup and restore best practices](https://learn.microsoft.com/windows-server/virtualization/hyper-v/manage/backup-and-restore)
187+
- [Antivirus exclusions for Hyper-V](https://learn.microsoft.com/troubleshoot/windows-server/virtualization/antivirus-exclusions-for-hyper-v-hosts)
188+
- [PowerShell Hyper-V cmdlets](https://learn.microsoft.com/powershell/module/hyper-v/?view=windowsserver2022-ps)
189+
190+
**Note**: Always back up critical VMs before performing file merges, deletions, or major configuration changes. For further assistance, consult Microsoft Support or your backup vendor.

0 commit comments

Comments
 (0)