Recently a number of things have come to light together, the combination of which is remarkable.
- The System Volume Information folder, on very many slow Windows machines, has a lot of junk files within, files that serve no purpose. These are called “orphan shadows”.
- Orphan shadows occur when the RAID or other drive capability of a Windows machine, cannot keep up with demands on the Volume Shadow Services (VSS) subsystem of Windows, and/or has to abandon an effort in midstream. VSS is used by a huge variety of Windows and application operations.
- The best-understood examples of events causing orphan shadows, are hard poweroffs in the middle of many different operations large and small. Another is when block-level backups fail.
- Other examples are simultaneous heavy demands which are too high for the RAID or drive capability. For instance, if demands on RAID are already fairly high, a large database request happening at the same moment as the incremental for a backup will cause one or both to fail, and will create orphan shadow(s) for one or both.
- As orphan shadows build up, they take more and more space in System Volume Information, and when SVI has more than 20 gigabytes or so of these things, all VSS operations get slower and slower, presumably because the VSS system has to choose to bypass all of the orphans over and over again.
- SVI has been seen to rise to hundreds of gigabytes, sometimes taking hours to clear.
The short-term step, to get things working, is to clear SVI. On a client OS, this is done thus:
vssadmin delete shadows /all
On a server OS, we do the above, and then we also run
diskshadow, and within that little environment, we do:
delete shadows all
Alternatively, we can replace all of the above steps with:
wmic shadowcopy delete /nointeractive
One situation has been seen thus far, where the above two do not do the job, where SVI remains huge. No standard solution has been identified for this as of yet. Up until now, the symptoms have been clear and obvious, but causality a lot more mysterious, and we have often alleviated the situation via removal of OEMware, BIOS, firmware, and driver updates, and use of tweaking tools. On a server, one can remove Windows Defender:
and this helps a lot (solved one big problem so far) on server, but not on a client OS, Microsoft doesn’t allow it. Also, methods to disable bits of Windows Defender on client OS have been disallowed increasingly, as build upgrades have been given.
One step which has helped is to
CHKDSK /F the volumes and reboot, and then try the standard cleanout steps again.
Just today, some VSS tweaks have manifest:
HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\VSS\Settings DWORD MaxShadowCopies
HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\VolSnap DWORD MinDiffAreaFileSize
but it will be a while of testing before any confidence is available as to how to use these to help. Thus far, MaxShadowCopies of 8 and MinDiffAreaFileSize of 128 seems to be helping.