Some Notes on the Machines

The hidden NTFS “System Volume Information” folders on Windows machines, can build up and up and up in size. I’ve seen instances ranging from 20G to hundreds of gigabytes, and every time this occurs, the overall system slows down, and often slows down a whole lot. SpaceSniffer is my favorite method of identifying this situation, but there are many. The only preventative I have been able to identify so far, is here.

But here we are discussing cleanup. If you have SentinelOne (S1) installed on this machine, you need to know that S1 considers deletion of volume shadows to be very bad actor behavior. This is because it often is a way that cryptolockers and others delete last-known-good checkpoints. S1 will not let you clear SVI, unless you disable it first, and it will complain very loudly if you try. Instructions for disabling S1, are at the end of this article. There may well be other security tools which will behave similarly, and need similar interaction beforehand.

General cleanup steps:

1. The easiest one to remember may be: vssadmin delete shadows /all
2. The more comprehensive, probably: wmic shadowcopy delete /nointeractive
3. Every once in a while on a server, the above two don’t get it done, and this is needed:
   
   diskshadow
   then within diskshadow’s command line: delete shadows all
   

Any of these can take a while, especially if SVI is big, e.g., more than 20-30 gigabytes. It can get huge occasionally, hundreds of gigabytes. I recently saw 1,022 shadow copies deleted (the first and third methods tell you the count) from one server.

Special case cleanup steps

Special cases do occur. Here are steps which can help a lot.

• Sometimes the steps above quit in the middle. Start them over again. Often they’ll complete.
• If the above does not completely solve the situation (if the SVI folder is still huge), do vssadmin resize shadowstorage for the relevant drive(s) (try /? for syntax…), first to 10%, then back to whatever it was. Sometimes Windows will do a lot of steady cleanup for you, sometimes over hours of time. You’ll see it by watching File Explorer.
• Run CHKDSK /F at reboot.

To disable SentinelOne:

1. First get the Passphrase for the machine, from the S1 console. It’s under Actions, you can choose Show Passphrase. Do be aware that your S1 admin may receive a notice that you have asked for this.
2. cd "C:\Program Files\SentinelOne\Sentinel*"
3. Please put the actual passphrase in, and the quotes are necessary:
   .\sentinelctl.exe unload -slam -k "<passphrase>"

Then, and only then, will the cleanup commands above work.

To reenable S1:

4. .\sentinelctl.exe load -slam

If you should need to reenable S1 and your command prompt is not where you need it, here’s a paste:

cd "C:\Program Files\SentinelOne\Sentinel*"
.\sentinelctl.exe load -slam

Found by the profound Brigg Bush:

$writers = vssadmin list writers | 
    Select-String -Context 0,4 'Writer name:' | 
    ? {$_.Context.PostContext[2].Trim() -ne "Last error: No error"} | 
    Select Line | 
    % {$_.Line.tostring().Split("'")[1]}



 $ServiceNames = $writers | 
    ForEach-Object {
        switch ($_) {
            'ASR Writer' { $Result = 'VSS' }
            'Bits Writer' { $Result = 'BITS'}
             'Certificate Authority' { $Result = 'EventSystem'}
            'COM+ REGDB Writer' { $Result = 'VSS'}
            'DFS Replication service writer' { $Result = 'DFSR'}
            'Dhcp Jet Writer' { $Result = 'DHCPServer'}
            'FRS Writer' { $Result = 'NtFrs' }
            'IIS Config Writer' { $Result = 'AppHostSvc'}
            'IIS Metabase Writer' { $Result = 'IISADMIN'}
            'Microsoft Exchange Writer' { $Result = 'MSExchangeIS'}
            'Microsoft Hyper-V VSS Writer' { $Result = 'vmms'}
            'MS Search Service Writer' { $Result = 'EventSystem'}
            'NPS VSS Writer' { $Result = 'EventSystem'}
            'NTDS' { 'EventSystem'}
            'OSearch VSS Writer' { $Result = 'OSearch'}
            'OSearch14 VSS Writer' { $Result = 'OSearch14'}
            'Registry Writer' { $Result = 'VSS'}
            'Shadow Copy Optimization Writer' { $Result = 'VSS'}
            'Sharepoint Services Writer' { $Result = 'SPWriter'}
            'SPSearch VSS Writer' { $Result = 'SPSearch'}
            'SPSearch4 VSS Writer' { $Result = 'SPSearch4'}
            'SqlServerWriter' { $Result = 'SQLWriter'}
            'System Writer' { $Result = 'CryptSvc'}
            'WMI Writer' { $Result = 'Winmgmt'}
            'TermServLicensing' { $Result = 'TermServLicensing'}
        }
        $result
    }

If ($ServiceNames) { Restart-Service -Name ($ServiceNames | Select-Object -Unique) -WhatIf }

If ($Result) { Restart-Service -Name ($ServiceNames | Select-Object -Unique) -WhatIf }

When trying to get SVI contents out and vssadmin and diskshadow don’t work, this may:

wmic
(then it its own command prompt) shadowcopy delete

It will require “Y” and “Enter” to be pressed, it will do one at a time. One can also have it delete all noninteractively:

wmic shadowcopy delete /nointeractive

and in Powershell, one can run that noninteractive as a job:

Start-Job -ScriptBlock { wmic shadowcopy delete /nointeractive }

There is some definite undocumented mystery concerning Volume Shadow Services in Windows. In general we are told to use VSSADMIN to do maintenance, and it does a lot, and helps a lot. But recently there was a Server 2012 R2 machine using 280G of space for System Volume Information on C:, and after CHKDSK and various DISMs it still was using 280G. So I tried removing all orphan shadows with VSSADMIN, and it found one and removed it; almost zero change comparatively. And then I searched a little deeper.

DISKSHADOW is built into Windows 2012 R2 and later, and earlier too I think, not sure how early. It may be a successor to VSHADOW which was an SDK add-on to 2003. Regardless, DISKSHADOW is a command-line environment of its own sort of like NSLOOKUP and DISKPART (!), not a simple command, can run a script of its own commands, and one of its commands is:

DELETE SHADOWS ALL

Now VSSADMIN DELETE SHADOWS /ALL deletes all orphan shadows, all VSS shadow copy sets which Windows knows are good to delete. The above within DISKSHADOW is a different animal altogether: it deletes them all. And does not appear to report anything to event logs (!). And definitely frees up a whole lot of space. And also, definitely not least, is flagged as infection activity by certain high-test super-anti-malware tools, when run! That was amazing, a Windows built-in being run with one of its own recommended commands, flagged. But I’ll think that that means this is to be used only when very needed. There may be gotchas I don’t know about yet.

As I write, the System Volume Information on this C: drive has been shrunken 290 (two hundred ninety) gigabytes, and everything is still running fine. There were originally 522 (five hundred twenty-two) shadow copies hanging out there of many different sizes, and DISKSHADOW was able to delete them all, all server services appear AOK.

A good thing to do, is to do cleanup/improvement steps, and then restart related services:

These steps can improve Windows performance a whole lot. It works because a vast array of different applications and services in Windows utilize VSS on their backends. All of the below, except for one server-only step sometimes needed, is encapsulated in PowerShell script (3.0 and up) OVSS.ps1 , part of the windows-tools project.

To do the VSS optimization interactively, start an administrative CMD, and then…

Step 1:

vssadmin Delete Shadows /All

If there are orphan shadows, you will be asked whether you want to delete them. If there are and you delete them, you will see immediate performance benefit. Reportedly, Windows autodeletes them only after there are 64 per volume. We prefer to see zero! These build up as a result of bad shutdowns, drive and drive controller issues, and insufficient RAID resources to serve demands.

Step 2:

We now improve any existing preassociation of disk space for VSS. On some machines, this will increase performance very impressively, immediately. In general it keeps them smooth and stable and prevents hesitations. This does not reserve or take up the space, it just “associates” it, makes it ready for use, so that whenever Windows wants to do any of the bajillions of things it does with VSS, things ranging from tiny to enormous, it can skip that step.

It is worthwhile to know that C: on all workstation installs and many server installs, has a minimal preassociation already set up. And we should check to see if it has been done. So the first step is to check it. Do the below:

vssadmin list shadowstorage

If it gives you something like this:

vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2013 Microsoft Corp.

Shadow Copy Storage association
   For volume: (\\?\Volume{84214e3c-0000-0000-0000-100000000000}\)\\?\Volume{84214e3c-0000-0000-0000-100000000000}\
   Shadow Copy Storage volume: (\\?\Volume{84214e3c-0000-0000-0000-100000000000}\)\\?\Volume{84214e3c-0000-0000-0000-100000000000}\
   Used Shadow Copy Storage space: 0 bytes (0%)
   Allocated Shadow Copy Storage space: 0 bytes (0%)
   Maximum Shadow Copy Storage space: 100 MB (20%)

Shadow Copy Storage association
   For volume: (C:)\\?\Volume{84214e3c-0000-0000-0000-501f00000000}\
   Shadow Copy Storage volume: (C:)\\?\Volume{84214e3c-0000-0000-0000-501f00000000}\
   Used Shadow Copy Storage space: 0 bytes (0%)
   Allocated Shadow Copy Storage space: 0 bytes (0%)
   Maximum Shadow Copy Storage space: 373 GB (20%)

where “Maximum Shadow Copy Storage space: “ for each volume is set to 20%, the rest has been done, you are fully optimized. Otherwise, if this is a desktop OS, we resize the existing association for each volume. For volumes without letters, and to pull a list of all VSS-ready volumes, see the note towards the end of this document.

So for the C drive, do the below in administrative CMD:

vssadmin Resize ShadowStorage /For=C: /On=C: /MaxSize=20%

Do repeat for any other active hard drive partitions, D:, E:, et cetera. Don’t worry if you get an error, the next step covers it.

Step 3:

It may well throw an error, saying there is no such association. If this is a workstation OS, vssadmin lacks two commands which we need for any further steps, so in that case we are done. But on any Windows Server OS from 2008R2, if the error was thrown, we do an Add:

vssadmin Add ShadowStorage /For=E: /On=E: /MaxSize=20%

Step 4:

And finally (server only), one more thing which can help if, for instance, C: is almost full but E: has plenty of space:

vssadmin Delete ShadowStorage /For=C: /On=C:
vssadmin Add ShadowStorage /For=C: /On=E: /MaxSize=20%

This maximizes overall performance, and also prevents possible backup failures and other issues due to insufficient disk space on C:.

Note:

On some machines, the volumes do not have letters. For these you will need to use the volume GUID path. In vssadmin list shadowstorage, they look like this:

Shadow Copy Storage association
   For volume: (\\?\Volume{99ac05c7-c06b-11e0-b883-806e6f6e6963}\)\\?\Volume{99a
c05c7-c06b-11e0-b883-806e6f6e6963}\
   Shadow Copy Storage volume: (\\?\Volume{99ac05c7-c06b-11e0-b883-806e6f6e6963}
\)\\?\Volume{99ac05c7-c06b-11e0-b883-806e6f6e6963}\
   Used Shadow Copy Storage space: 0 B (0%)
   Allocated Shadow Copy Storage space: 0 B (0%)
   Maximum Shadow Copy Storage space: 32 MB (32%)

For such a situation, substitute \\?\Volume{99ac05c8-c06b-11e0-b883-806e6f6e6963} (the whole long string) for C: in the above command lines.

PowerShell will give GUI paths for all volumes thusly:

GWMI -namespace root\cimv2 -class win32_volume

References are here:

https://technet.microsoft.com/en-us/library/cc788050.aspx

https://www.storagecraft.com/support/kb/article/289

http://backupchain.com/i/how-to-delete-all-vss-shadows-and-orphaned-shadows

http://www.tech-no.org/?p=898

‘vssadmin delete shadows /all’ gets a lot of gunk out, but DevNodeClean:

https://www.microsoft.com/en-us/download/confirmation.aspx?id=42286

eliminates related registry entries, including Device Manager grayed-out items, for even more improvement. This has existed for a while, but not available in public download very long.

From the amazing Matt Quick:

http://www.shadowexplorer.com/

Some very interesting info:

https://social.technet.microsoft.com/Forums/windowsserver/en-US/1b926c85-b866-464b-9590-70506ba9d8ce/vss-issue-event-id-22-12289-12293-error-0×8004230f-failed-to-retrieve-volumes-that-are-eligible?forum=windowsbackup

https://msdn.microsoft.com/en-us/library/windows/desktop/bb891959#mindiffareafilesize

Here are some steps and info:

http://www.storagecraft.com/support/kb/article/32

Here is a Fix-It from Microsoft for Server 2003, which runs quite a few steps automatically, and does not (this is new) necessarily require a reboot:

http://support.microsoft.com/kb/940184

And lastly, here is a patch from Microsoft which can help a lot in Server 2003, even on fully-updated machines:

http://support.microsoft.com/default.aspx?scid=kb;en-us;891957

Here is a product, free for commercial and non-profit use, which does the job of VSS under Linux:

http://www.r1soft.com/tools/linux-hot-copy/