Recently a number of things have come to light together, the combination of which is remarkable.
- The System Volume Information folder, on very many slow Windows machines, has a lot of junk files within, files that serve no purpose. These are called “orphan shadows”.
- Orphan shadows occur when the RAID or drive capability of a Windows machine, cannot keep up with demands on the Volume Shadow Services (VSS) subsystem of Windows, or has to abandon an effort in midstream. VSS is used by a huge variety of Windows and application operations.
- Examples of events causing orphan shadows, are hard poweroffs in the middle of many different operations large and small.
- Other examples are simultaneous heavy demands which are too high for the RAID or drive capability. For instance, if demands on RAID are already fairly high, a large database request happening at the same moment as the incremental for a backup will cause one or both to fail, and will create orphan shadow(s) for one or both.
- As orphan shadows build up, they take more and more space in System Volume Information, and when SVI has more than 20 gigabytes or so of these things, all VSS operations get slower and slower, presumably because the VSS system has to choose to bypass all of the orphans over and over again.
- SVI has been seen to rise to hundreds of gigabytes, sometimes taking hours to clear.
The short-term solution is to clear SVI. On a client OS, this is done thus:
vssadmin delete shadows /all
On a server OS, we do the above, and then we also run
diskshadow, and within that little environment, we do:
delete shadows all
One situation has been seen thus far, where the above two do not do the job. No standard solution has been identified for this as of yet. Up until now, the symptoms have been clear and obvious, but causality a lot more mysterious, and we have often alleviated the situation via removal of OEMware, BIOS, firmware, and driver updates, and use of tweaking tools. On a server, one can remove Windows Defender:
and this helps a lot (solved one big problem so far) on server, but not on a client OS, Microsoft doesn’t allow it. Also, methods to disable bits of Windows Defender on client OS have been disallowed increasingly, as build upgrades have been given.
Just today, some VSS tweaks have manifest:
HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\VSS\Settings DWORD MaxShadowCopies
HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\VolSnap DWORD MinDiffAreaFileSize
but it will be a while of testing before any confidence is available as to how to use these to help and not harm.