Some Notes on the Machines

SentinelOne (S1) considers deletion of volume shadows to be bad actor behavior, because it often is a way that cryptolockers delete last-known-good checkpoints. Unfortunately, there is a lot of cleanup which ends up being required, as the hidden NTFS System Volume Information folders on Windows machines build up and up and up in size. I’ve seen instances ranging from 20G to hundreds of gigabytes, and every time this occurs, the overall system slows down, and often slows down a whole lot. WizTree is my favorite method of identifying this situation, but there are many.

In order to clear a huge SVI folder with S1 installed, one must do this:

1. First get the Passphrase for the machine, from the S1 console. It’s under Actions, you can choose Show Passphrase. Do be aware that your S1 admin will probably receive a notice that you have asked for this.
2. cd "C:\Program Files\SentinelOne\Sentinel*"
3. Please put the actual passphrase in, and the quotes are necessary:
   .\sentinelctl.exe unload -slam -k "<passphrase>"
4. vssadmin delete shadows /all
5. Only if this is a server, check System Volume Information size again. If it’s still big, we have three options.
   1. The first is to do these two:
      diskshadow
      (within diskshadow’s command line:) delete shadows all
      
      This can take a while, especially if SVI is big, e.g., more than 20-30 gigabytes. It can get huge occasionally, hundreds of gigabytes. I recently saw 1,022 shadow copies deleted (it tells you the count at the end) from one server.
   2. The second is to do this:
      wmic shadowcopy delete /nointeractive
      which runs the cleanup nicely, and possibly a bit faster than the other.
   3. The third is to do this, in Powershell:
      Start-Job -ScriptBlock { wmic shadowcopy delete /nointeractive }
      which does #2, but in the background.

6. .\sentinelctl.exe load -slam

And you are done.

If you should need to reenable S1 after work such as the above, here’s a paste:

cd "C:\Program Files\SentinelOne\Sentinel*"
.\sentinelctl.exe load -slam

When trying to get SVI contents out and vssadmin and diskshadow don’t work, this may:

wmic
(then it its own command prompt) shadowcopy delete

It will require “Y” and “Enter” to be pressed, it will do one at a time. One can also have it delete all noninteractively:

wmic shadowcopy delete /nointeractive

and in Powershell, one can run that noninteractive as a job:

Start-Job -ScriptBlock { wmic shadowcopy delete /nointeractive }

There is some definite undocumented mystery concerning Volume Shadow Services in Windows. In general we are told to use VSSADMIN to do maintenance, and it does a lot, and helps a lot. But recently there was a Server 2012 R2 machine using 280G of space for System Volume Information on C:, and after CHKDSK and various DISMs it still was using 280G. So I tried removing all orphan shadows with VSSADMIN, and it found one and removed it; almost zero change comparatively. And then I searched a little deeper.

DISKSHADOW is built into Windows 2012 R2 and later, and earlier too I think, not sure how early. It may be a successor to VSHADOW which was an SDK add-on to 2003. Regardless, DISKSHADOW is a command-line environment of its own sort of like NSLOOKUP and DISKPART (!), not a simple command, can run a script of its own commands, and one of its commands is:

DELETE SHADOWS ALL

Now VSSADMIN DELETE SHADOWS /ALL deletes all orphan shadows, all VSS shadow copy sets which Windows knows are good to delete. The above within DISKSHADOW is a different animal altogether: it deletes them all. And does not appear to report anything to event logs (!). And definitely frees up a whole lot of space. And also, definitely not least, is flagged as infection activity by certain high-test super-anti-malware tools, when run! That was amazing, a Windows built-in being run with one of its own recommended commands, flagged. But I’ll think that that means this is to be used only when very needed. There may be gotchas I don’t know about yet.

As I write, the System Volume Information on this C: drive has been shrunken 290 (two hundred ninety) gigabytes, and everything is still running fine. There were originally 522 (five hundred twenty-two) shadow copies hanging out there of many different sizes, and DISKSHADOW was able to delete them all, all server services appear AOK.

A good thing to do, is to do cleanup/improvement steps, and then restart related services:

These steps can improve Windows performance a whole lot. It works because a vast array of different applications and services in Windows utilize VSS on their backends. All of the below, except for one server-only step sometimes needed, is encapsulated in PowerShell script (3.0 and up) OVSS.ps1 , part of the windows-tools project.

To do the VSS optimization interactively, start an administrative CMD, and then…

Step 1:

vssadmin Delete Shadows /All

If there are orphan shadows, you will be asked whether you want to delete them. If there are and you delete them, you will see immediate performance benefit. Reportedly, Windows autodeletes them only after there are 64 per volume. We prefer to see zero! These build up as a result of bad shutdowns, drive and drive controller issues, and insufficient RAID resources to serve demands.

Step 2:

We now improve any existing preassociation of disk space for VSS. On some machines, this will increase performance very impressively, immediately. In general it keeps them smooth and stable and prevents hesitations. This does not reserve or take up the space, it just “associates” it, makes it ready for use, so that whenever Windows wants to do any of the bajillions of things it does with VSS, things ranging from tiny to enormous, it can skip that step.

It is worthwhile to know that C: on all workstation installs and many server installs, has a minimal preassociation already set up. And we should check to see if it has been done. So the first step is to check it. Do the below:

vssadmin list shadowstorage

If it gives you something like this:

vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2013 Microsoft Corp.

Shadow Copy Storage association
   For volume: (\\?\Volume{84214e3c-0000-0000-0000-100000000000}\)\\?\Volume{84214e3c-0000-0000-0000-100000000000}\
   Shadow Copy Storage volume: (\\?\Volume{84214e3c-0000-0000-0000-100000000000}\)\\?\Volume{84214e3c-0000-0000-0000-100000000000}\
   Used Shadow Copy Storage space: 0 bytes (0%)
   Allocated Shadow Copy Storage space: 0 bytes (0%)
   Maximum Shadow Copy Storage space: 100 MB (20%)

Shadow Copy Storage association
   For volume: (C:)\\?\Volume{84214e3c-0000-0000-0000-501f00000000}\
   Shadow Copy Storage volume: (C:)\\?\Volume{84214e3c-0000-0000-0000-501f00000000}\
   Used Shadow Copy Storage space: 0 bytes (0%)
   Allocated Shadow Copy Storage space: 0 bytes (0%)
   Maximum Shadow Copy Storage space: 373 GB (20%)

where “Maximum Shadow Copy Storage space: “ for each volume is set to 20%, the rest has been done, you are fully optimized. Otherwise, if this is a desktop OS, we resize the existing association for each volume. For volumes without letters, and to pull a list of all VSS-ready volumes, see the note towards the end of this document.

So for the C drive, do the below in administrative CMD:

vssadmin Resize ShadowStorage /For=C: /On=C: /MaxSize=20%

Do repeat for any other active hard drive partitions, D:, E:, et cetera. Don’t worry if you get an error, the next step covers it.

Step 3:

It may well throw an error, saying there is no such association. If this is a workstation OS, vssadmin lacks two commands which we need for any further steps, so in that case we are done. But on any Windows Server OS from 2008R2, if the error was thrown, we do an Add:

vssadmin Add ShadowStorage /For=E: /On=E: /MaxSize=20%

Step 4:

And finally (server only), one more thing which can help if, for instance, C: is almost full but E: has plenty of space:

vssadmin Delete ShadowStorage /For=C: /On=C:
vssadmin Add ShadowStorage /For=C: /On=E: /MaxSize=20%

This maximizes overall performance, and also prevents possible backup failures and other issues due to insufficient disk space on C:.

Note:

On some machines, the volumes do not have letters. For these you will need to use the volume GUID path. In vssadmin list shadowstorage, they look like this:

Shadow Copy Storage association
   For volume: (\\?\Volume{99ac05c7-c06b-11e0-b883-806e6f6e6963}\)\\?\Volume{99a
c05c7-c06b-11e0-b883-806e6f6e6963}\
   Shadow Copy Storage volume: (\\?\Volume{99ac05c7-c06b-11e0-b883-806e6f6e6963}
\)\\?\Volume{99ac05c7-c06b-11e0-b883-806e6f6e6963}\
   Used Shadow Copy Storage space: 0 B (0%)
   Allocated Shadow Copy Storage space: 0 B (0%)
   Maximum Shadow Copy Storage space: 32 MB (32%)

For such a situation, substitute \\?\Volume{99ac05c8-c06b-11e0-b883-806e6f6e6963} (the whole long string) for C: in the above command lines.

PowerShell will give GUI paths for all volumes thusly:

GWMI -namespace root\cimv2 -class win32_volume

References are here:

https://technet.microsoft.com/en-us/library/cc788050.aspx

https://www.storagecraft.com/support/kb/article/289

http://backupchain.com/i/how-to-delete-all-vss-shadows-and-orphaned-shadows

http://www.tech-no.org/?p=898

‘vssadmin delete shadows /all’ gets a lot of gunk out, but DevNodeClean:

https://www.microsoft.com/en-us/download/confirmation.aspx?id=42286

eliminates related registry entries, including Device Manager grayed-out items, for even more improvement. This has existed for a while, but not available in public download very long.

From the amazing Matt Quick:

http://www.shadowexplorer.com/

Some very interesting info:

https://social.technet.microsoft.com/Forums/windowsserver/en-US/1b926c85-b866-464b-9590-70506ba9d8ce/vss-issue-event-id-22-12289-12293-error-0×8004230f-failed-to-retrieve-volumes-that-are-eligible?forum=windowsbackup

https://msdn.microsoft.com/en-us/library/windows/desktop/bb891959#mindiffareafilesize

Here are some steps and info:

http://www.storagecraft.com/support/kb/article/32

Here is a Fix-It from Microsoft for Server 2003, which runs quite a few steps automatically, and does not (this is new) necessarily require a reboot:

http://support.microsoft.com/kb/940184

And lastly, here is a patch from Microsoft which can help a lot in Server 2003, even on fully-updated machines:

http://support.microsoft.com/default.aspx?scid=kb;en-us;891957

Here is a product, free for commercial and non-profit use, which does the job of VSS under Linux:

http://www.r1soft.com/tools/linux-hot-copy/

Windows Backup on Server 2008, is very different than previous backup tools provided by Microsoft. It is block-level, meaning that it handles only whole volumes (drives) at once. But it works very well, it has much lower impact on resources. It uses VSS as its basis.

Data difficult to identify, but useful for setting up monitoring, follow.

Service name: Block Level Backup Engine Service
Executable (on 32-bit, default): “C:\Windows\system32\wbengine.exe”
Command-line utility (runs from elevated CMD): wbadmin