A VMware backup job may fail with the error message 'An error occurred while quiescing the virtual machine'. As this issue is primarily one with VMware technology it is difficult for Infrascale to identify and resolve the problem. In this article, we have provided some potential resolutions as well as some last-resort workarounds. We strongly recommend, however, that you reach out to VMware support for a permanent solution if all of the potential resolutions do not work.
Message Log Sample:
04-Apr-22 09:13:55: BeforeJob: run command "/raider/etc/runBeforeJob.sh 34490 VMware_VM+VM:Backup.2022-04-04_01 D7468DC7-738A-B7F4-8335-AB8C06E2B642 Backup Incremental"
04-Apr-22 09:13:55: VMware Backup Started
04-Apr-22 09:15:41: VMwareBackupJobRunner: error encountered during VM backup: Snapshot of VMware_VM unsuccessful, result='error', error='An error occurred while quiescing the virtual machine. See the virtual machine's event log for details.'
04-Apr-22 09:15:41: VMware Backup Finished
04-Apr-22 09:15:41: Job started, firmware version: 126.96.36.199, client id: D7468DC7-738A-B7F4-8335-AB8C06E2B642, client ip: localhost, agent id: Unknown
04-Apr-22 09:15:41: BeforeJob: A failure has occured, Job is terminating.
04-Apr-22 09:15:41: Runscript: BeforeJob returned non-zero status=1. ERR=Child exited with code 1
What is Quiescing
Quiescing a file system is a process of bringing the on-disk data of a physical or virtual computer into a state suitable for backups. This process might include such operations as flushing dirty buffers from the operating system's in-memory cache to disk, or other higher-level application-specific tasks.
This VMware technology allows the creation of application-consistent backups. If the VMware snapshot with quiescing enabled has failed, it means that the VSS snapshot for the guest OS of the VM failed.
You can find additional information about VMware Snapshots in the following article:
Overview of virtual machine snapshots in vSphere (1015180)
VMware Tools is used to quiesce the file system in the virtual machine. Because of this, VMware Tools must be installed, up to date, and running in order for quiescing to function properly.
You can find the steps on how to install VMware Tools in the following article:
How to install VMware Tools (1014294)
Free Space on the VM and VSS Shadow Storage
In order to run a VSS snapshot, the VM should have enough free space on both the disks and inside VSS Shadow Storage. Check that the VM has at least 15% of free space on the disks and VSS Shadow storage.
You can find the steps on how to check the space in VSS Shadow Storage, increase the size of the VSS Shadow Storage, or remove old VSS Shadow Copies in the following article:
How To Resolve an 'Insufficient Storage Space' Error
Reboot the VM or Restart the VSS services
In some cases, the services may be stuck on the VM. It is recommended to try rebooting the VM to make sure that there are no frozen processes. Another approach is to try restarting the VSS services manually.
You can find additional troubleshooting steps published by VMware in the following article:
Troubleshooting Volume Shadow Copy (VSS) quiesce related issues (1007696)
Articles created by VMware
There are a few additional KBAs provided by VMware regarding this error:
- Unable to take a quiesced snapshot of Windows Server virtual machine with GPT disks without MSR partition (71130)
- Cannot create a quiesced snapshot because the snapshot operation exceeded the time limit for holding off I/O in the frozen virtual machine
- Quiescing snapshots in an ESXi host fails with the error: The guest OS has reported an error during quiescing. The error code was: 5 The error message was: Asynchronous operation failed: >VssSyncStart (2090545)
Turn off the VM before running a backup
If the VM is not running, quiescing is not needed. The reason for this is that all the data is on the disks and no data is present in the cache or RAM. Shutting down a VM and running a manual backup job should succeed, but require you to shut down the VM every time you would like to run a backup.
Warning: This approach is NOT recommended by the Infrascale team unless it is a last resort.
It is possible to disable the option "Quiescing" on the appliance UI on the Clients → Edit tab. This will stop doing VSS snapshots inside the VM guest OS. The backup job will succeed but will be crash-consistent instead of application-consistent, which means that live data that is in the RAM and cache will not be backed up.