Quiescing a file system is a process of bringing the on-disk data of a physical or virtual computer into a state suitable for backups. This process might include such operations as flushing dirty buffers from the operating system's in-memory cache to disk, or other higher-level application-specific tasks.
This VMware technology allows the creation of application-consistent backups. If the VMware snapshot with quiescing enabled has failed, it means that the VSS snapshot for the guest OS of the VM failed.
You can find additional information about VMware Snapshots in the following article:
Overview of virtual machine snapshots in vSphere (1015180)
VMware Tools is used to quiesce the file system in the virtual machine. Because of this, VMware Tools must be installed, up to date, and running in order for quiescing to function properly.
You can find the steps on how to install VMware Tools in the following article:
How to install VMware Tools (1014294)
In order to run a VSS snapshot, the VM should have enough free space on both the disks and inside VSS Shadow Storage. Check that the VM has at least 15% of free space on the disks and VSS Shadow storage.
You can find the steps on how to check the space in VSS Shadow Storage, increase the size of the VSS Shadow Storage, or remove old VSS Shadow Copies in the following article:
How To Resolve an 'Insufficient Storage Space' Error
In some cases, the services may be stuck on the VM. It is recommended to try rebooting the VM to make sure that there are no frozen processes. Another approach is to try restarting the VSS services manually.
You can find additional troubleshooting steps published by VMware in the following article:
Troubleshooting Volume Shadow Copy (VSS) quiesce related issues (1007696)
There are a few additional KBAs provided by VMware regarding this error:
If the VM is not running, quiescing is not needed. The reason for this is that all the data is on the disks and no data is present in the cache or RAM. Shutting down a VM and running a manual backup job should succeed but require you to shut down the VM every time you would like to run a backup.