Unable to Compact a VHD Due to a File System Limitation Explained

You may get the following error when attempting to compact a Hyper-V virtual hard disk (VHD):

The system failed to compact disk.vhd. Error Code: The requested operation could not be completed due to a file system limitation

This can be happen for several of reasons. 

1. Windows Server 2008 R2 does not support hosting VHDs on compressed NTFS volumes. See the article, Virtual Hard Disk (VHD) Architecture Explained.
2. The NTFS volume where the VHD resides is very heavily fragmented.  See Microsoft KB967351.

Both of these scenarios are fairly well explained, but I also came across another reason this can happen.  You cannot compact a VHD where the VHD contains a VSS backup.

The VHD in question was an Exchange 2010 data volume where I had run an Exchange-aware VSS backup using the native Windows Server Backup utility.  See my article, How to Backup Exchange 2010 RTM at Release Timeframe.

In order to compact the VHD, all VSS shadows must be deleted from the VHD volume within the VM.  This can be done by running the vssadmin utility, as such:

vssadmin delete shadows /all

Now you can compact the VHD to reduce the physical size of the VHD file and regain space on the Hyper-V host.