Category: VSS

Clear System Volume Information (and Disable Sentinel One)
article #1438, updated 74 days ago

The hidden NTFS “System Volume Information” folders on Windows machines, can build up and up and up in size. I’ve seen instances ranging from 20G to hundreds of gigabytes, and every time this occurs, the overall system slows down, and often slows down a whole lot. SpaceSniffer is my favorite method of identifying this situation, but there are many. The only preventative I have been able to identify so far, is here.

But here we are discussing cleanup. If you have SentinelOne (S1) installed on this machine, you need to know that S1 considers deletion of volume shadows to be very bad actor behavior. This is because it often is a way that cryptolockers and others delete last-known-good checkpoints. S1 will not let you clear SVI, unless you disable it first, and it will complain very loudly if you try. Instructions for disabling S1, are at the end of this article. There may well be other security tools which will behave similarly, and need similar interaction beforehand.

General cleanup steps:

  1. This command usually gets all of them: wmic shadowcopy delete /nointeractive
  2. Very rarely, this will get a few more: vssadmin delete shadows /all
  3. And even more rarely on a server only, the above two don’t get it done, and this is needed:

    diskshadow
    then within diskshadow’s command line: delete shadows all

Any of these can take a while, especially if SVI is big, e.g., more than 20-30 gigabytes. It can get huge occasionally, hundreds of gigabytes. I recently saw 1,022 shadow copies deleted (the first and third methods tell you the count) from one server.

Special case cleanup steps

Special cases do occur. Here are steps which can help a lot.

  • Sometimes the steps above quit in the middle. Start them over again. Often they’ll complete.
  • If the above does not completely solve the situation (if the SVI folder is still huge), do vssadmin resize shadowstorage for the relevant drive(s) (try /? for syntax…), first to 10%, then back to whatever it was. Sometimes Windows will do a lot of steady cleanup for you, sometimes over hours of time. You’ll see it by watching File Explorer.
  • Run CHKDSK /F at reboot. Then start from the beginning :-)

To disable SentinelOne:

  1. First get the Passphrase for the machine, from the S1 console. It’s under Actions, you can choose Show Passphrase. Do be aware that your S1 admin may receive a notice that you have asked for this.
  2. cd "C:\Program Files\SentinelOne\Sentinel*"
  3. Please put the actual passphrase in, and the quotes are necessary:
    .\sentinelctl.exe unload -slam -k "<passphrase>"

Then, and only then, will the cleanup commands above work.

To reenable S1:

  1. .\sentinelctl.exe load -slam

If you should need to reenable S1 and your command prompt is not where you need it, here’s a paste:

cd "C:\Program Files\SentinelOne\Sentinel*"
.\sentinelctl.exe load -slam

Categories:      

==============

Restart All Failed VSS Writers
article #1513, updated 398 days ago

Found by the profound Brigg Bush:

$writers = vssadmin list writers | 
    Select-String -Context 0,4 'Writer name:' | 
    ? {$_.Context.PostContext[2].Trim() -ne "Last error: No error"} | 
    Select Line | 
    % {$_.Line.tostring().Split("'")[1]}



 $ServiceNames = $writers | 
    ForEach-Object {
        switch ($_) {
            'ASR Writer' { $Result = 'VSS' }
            'Bits Writer' { $Result = 'BITS'}
             'Certificate Authority' { $Result = 'EventSystem'}
            'COM+ REGDB Writer' { $Result = 'VSS'}
            'DFS Replication service writer' { $Result = 'DFSR'}
            'Dhcp Jet Writer' { $Result = 'DHCPServer'}
            'FRS Writer' { $Result = 'NtFrs' }
            'IIS Config Writer' { $Result = 'AppHostSvc'}
            'IIS Metabase Writer' { $Result = 'IISADMIN'}
            'Microsoft Exchange Writer' { $Result = 'MSExchangeIS'}
            'Microsoft Hyper-V VSS Writer' { $Result = 'vmms'}
            'MS Search Service Writer' { $Result = 'EventSystem'}
            'NPS VSS Writer' { $Result = 'EventSystem'}
            'NTDS' { 'EventSystem'}
            'OSearch VSS Writer' { $Result = 'OSearch'}
            'OSearch14 VSS Writer' { $Result = 'OSearch14'}
            'Registry Writer' { $Result = 'VSS'}
            'Shadow Copy Optimization Writer' { $Result = 'VSS'}
            'Sharepoint Services Writer' { $Result = 'SPWriter'}
            'SPSearch VSS Writer' { $Result = 'SPSearch'}
            'SPSearch4 VSS Writer' { $Result = 'SPSearch4'}
            'SqlServerWriter' { $Result = 'SQLWriter'}
            'System Writer' { $Result = 'CryptSvc'}
            'WMI Writer' { $Result = 'Winmgmt'}
            'TermServLicensing' { $Result = 'TermServLicensing'}
        }
        $result
    }

If ($ServiceNames) { Restart-Service -Name ($ServiceNames | Select-Object -Unique) -WhatIf }

If ($Result) { Restart-Service -Name ($ServiceNames | Select-Object -Unique) -WhatIf }

Categories:      

==============

Delete shadow copies by WMIC
article #1449, updated 778 days ago

When trying to get SVI contents out and vssadmin and diskshadow don’t work, this may:

wmic
(then it its own command prompt) shadowcopy delete

It will require “Y” and “Enter” to be pressed, it will do one at a time. One can also have it delete all noninteractively:

wmic shadowcopy delete /nointeractive

and in Powershell, one can run that noninteractive as a job:

Start-Job -ScriptBlock { wmic shadowcopy delete /nointeractive }

Categories:      

==============

Shrink A Huge "System Volume Information" Folder with DISKSHADOW
article #1423, updated 1042 days ago

There is some definite undocumented mystery concerning Volume Shadow Services in Windows. In general we are told to use VSSADMIN to do maintenance, and it does a lot, and helps a lot. But recently there was a Server 2012 R2 machine using 280G of space for System Volume Information on C:, and after CHKDSK and various DISMs it still was using 280G. So I tried removing all orphan shadows with VSSADMIN, and it found one and removed it; almost zero change comparatively. And then I searched a little deeper.

DISKSHADOW is built into Windows 2012 R2 and later, and earlier too I think, not sure how early. It may be a successor to VSHADOW which was an SDK add-on to 2003. Regardless, DISKSHADOW is a command-line environment of its own sort of like NSLOOKUP and DISKPART (!), not a simple command, can run a script of its own commands, and one of its commands is:

DELETE SHADOWS ALL

Now VSSADMIN DELETE SHADOWS /ALL deletes all orphan shadows, all VSS shadow copy sets which Windows knows are good to delete. The above within DISKSHADOW is a different animal altogether: it deletes them all. And does not appear to report anything to event logs (!). And definitely frees up a whole lot of space. And also, definitely not least, is flagged as infection activity by certain high-test super-anti-malware tools, when run! That was amazing, a Windows built-in being run with one of its own recommended commands, flagged. But I’ll think that that means this is to be used only when very needed. There may be gotchas I don’t know about yet.

As I write, the System Volume Information on this C: drive has been shrunken 290 (two hundred ninety) gigabytes, and everything is still running fine. There were originally 522 (five hundred twenty-two) shadow copies hanging out there of many different sizes, and DISKSHADOW was able to delete them all, all server services appear AOK.

Categories:      

==============

When VSS Writers Fail
article #1304, updated 1743 days ago

A good thing to do, is to do cleanup/improvement steps, and then restart related services:

VSS Writer Service, Short ID & Binary Service, Long Name
ASR Writer VSS Volume Shadow Copy
BITS Writer BITS Background Intelligent Transfer Service
COM+ REGDB Writer VSS Volume Shadow Copy
DFS Replication Service Writer DFSR DFS Replication
DHCP Jet Writer DHCPServer DHCP Server
FRS Writer NtFrs File Replication
FRSM Writer srmsvc File Server Resource Manager
IIS Config Writer AppHostSvc Application Host Helper Service
IIS Metabase Writer IISADMIN IIS Admin Service
Microsoft Exchange Replica Writer MSExchangeRepl Microsoft Exchange Replication Service
Microsoft Exchange Writer MSExchangeIS Microsoft Exchange Information Store
Microsoft Hyper-V VSS Writer vmms Hyper-V Virtual Machine Management
MSMQ Writer MSMQ Message Queuing
MSSearch Service Writer WSearch Windows Search
NTDS NTDS Active Directory Domain Services
O_Search VSS Writer OSearch Office SharePoint Server Search
O_Search 14 VSS Writer OSearch14 SharePoint Server Search 14
Registry Writer VSS Volume Shadow Copy
Shadow Copy Optimization Writer VSS Volume Shadow Copy
SMS Writer SMSSITEVSSWRITER SMS_SITE_VSS_WRITER
SPSearch VSS Writer SPSearch Windows SharePoint Services Search
SPSearch4 VSS Writer SPSearch4 SharePoint Foundation Search V4
SqlServerWriter SQLWriter SQL Server VSS Writer
System Writer CryptSvc Cryptographic Services
TermServLicensing TermServLicensing Remote Desktop Licensing
WIDWriter WIDWriter Windows Internal Database VSS Writer
WINS Jet Writer WINS Windows Internet Name Service (WINS)
WMI Writer Winmgmt Windows Management Instrumentation

Categories:      

==============

Windows Performance through VSS Cleanup and Preassociation
article #1004, updated 2104 days ago

These steps can improve Windows performance a whole lot. It works because a vast array of different applications and services in Windows utilize VSS on their backends. All of the below, except for one server-only step sometimes needed, is encapsulated in PowerShell script (3.0 and up) OVSS.ps1 , part of the windows-tools project.

To do the VSS optimization interactively, start an administrative CMD, and then…

Step 1:

vssadmin Delete Shadows /All

If there are orphan shadows, you will be asked whether you want to delete them. If there are and you delete them, you will see immediate performance benefit. Reportedly, Windows autodeletes them only after there are 64 per volume. We prefer to see zero! These build up as a result of bad shutdowns, drive and drive controller issues, and insufficient RAID resources to serve demands.

Step 2:

We now improve any existing preassociation of disk space for VSS. On some machines, this will increase performance very impressively, immediately. In general it keeps them smooth and stable and prevents hesitations. This does not reserve or take up the space, it just “associates” it, makes it ready for use, so that whenever Windows wants to do any of the bajillions of things it does with VSS, things ranging from tiny to enormous, it can skip that step.

It is worthwhile to know that C: on all workstation installs and many server installs, has a minimal preassociation already set up. And we should check to see if it has been done. So the first step is to check it. Do the below:

vssadmin list shadowstorage

If it gives you something like this:

vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2013 Microsoft Corp.

Shadow Copy Storage association
   For volume: (\\?\Volume{84214e3c-0000-0000-0000-100000000000}\)\\?\Volume{84214e3c-0000-0000-0000-100000000000}\
   Shadow Copy Storage volume: (\\?\Volume{84214e3c-0000-0000-0000-100000000000}\)\\?\Volume{84214e3c-0000-0000-0000-100000000000}\
   Used Shadow Copy Storage space: 0 bytes (0%)
   Allocated Shadow Copy Storage space: 0 bytes (0%)
   Maximum Shadow Copy Storage space: 100 MB (20%)

Shadow Copy Storage association
   For volume: (C:)\\?\Volume{84214e3c-0000-0000-0000-501f00000000}\
   Shadow Copy Storage volume: (C:)\\?\Volume{84214e3c-0000-0000-0000-501f00000000}\
   Used Shadow Copy Storage space: 0 bytes (0%)
   Allocated Shadow Copy Storage space: 0 bytes (0%)
   Maximum Shadow Copy Storage space: 373 GB (20%)

where “Maximum Shadow Copy Storage space: “ for each volume is set to 20%, the rest has been done, you are fully optimized. Otherwise, if this is a desktop OS, we resize the existing association for each volume. For volumes without letters, and to pull a list of all VSS-ready volumes, see the note towards the end of this document.

So for the C drive, do the below in administrative CMD:

vssadmin Resize ShadowStorage /For=C: /On=C: /MaxSize=20%

Do repeat for any other active hard drive partitions, D:, E:, et cetera. Don’t worry if you get an error, the next step covers it.

Step 3:

It may well throw an error, saying there is no such association. If this is a workstation OS, vssadmin lacks two commands which we need for any further steps, so in that case we are done. But on any Windows Server OS from 2008R2, if the error was thrown, we do an Add:

vssadmin Add ShadowStorage /For=E: /On=E: /MaxSize=20%

Step 4:

And finally (server only), one more thing which can help if, for instance, C: is almost full but E: has plenty of space:

vssadmin Delete ShadowStorage /For=C: /On=C:
vssadmin Add ShadowStorage /For=C: /On=E: /MaxSize=20%

This maximizes overall performance, and also prevents possible backup failures and other issues due to insufficient disk space on C:.

Note:

On some machines, the volumes do not have letters. For these you will need to use the volume GUID path. In vssadmin list shadowstorage, they look like this:

Shadow Copy Storage association
   For volume: (\\?\Volume{99ac05c7-c06b-11e0-b883-806e6f6e6963}\)\\?\Volume{99a
c05c7-c06b-11e0-b883-806e6f6e6963}\
   Shadow Copy Storage volume: (\\?\Volume{99ac05c7-c06b-11e0-b883-806e6f6e6963}
\)\\?\Volume{99ac05c7-c06b-11e0-b883-806e6f6e6963}\
   Used Shadow Copy Storage space: 0 B (0%)
   Allocated Shadow Copy Storage space: 0 B (0%)
   Maximum Shadow Copy Storage space: 32 MB (32%)

For such a situation, substitute \\?\Volume{99ac05c8-c06b-11e0-b883-806e6f6e6963} (the whole long string) for C: in the above command lines.

PowerShell will give GUI paths for all volumes thusly:

GWMI -namespace root\cimv2 -class win32_volume

References are here:

https://technet.microsoft.com/en-us/library/cc788050.aspx

https://www.storagecraft.com/support/kb/article/289

http://backupchain.com/i/how-to-delete-all-vss-shadows-and-orphaned-shadows

http://www.tech-no.org/?p=898

Categories:      

==============

Download DevNodeClean for further VSS cleanup
article #1156, updated 2206 days ago

‘vssadmin delete shadows /all’ gets a lot of gunk out, but DevNodeClean:

https://www.microsoft.com/en-us/download/confirmation.aspx?id=42286

eliminates related registry entries, including Device Manager grayed-out items, for even more improvement. This has existed for a while, but not available in public download very long.

Categories:      

==============

Browse shadow copies (VSS)!
article #747, updated 2377 days ago

From the amazing Matt Quick:

http://www.shadowexplorer.com/

Categories:      

==============

Troubleshooting and Fixing Windows VSS
article #274, updated 2728 days ago

Some very interesting info:

https://social.technet.microsoft.com/Forums/windowsserver/en-US/1b926c85-b866-464b-9590-70506ba9d8ce/vss-issue-event-id-22-12289-12293-error-0×8004230f-failed-to-retrieve-volumes-that-are-eligible?forum=windowsbackup

https://msdn.microsoft.com/en-us/library/windows/desktop/bb891959#mindiffareafilesize

Here are some steps and info:

http://www.storagecraft.com/support/kb/article/32

Here is a Fix-It from Microsoft for Server 2003, which runs quite a few steps automatically, and does not (this is new) necessarily require a reboot:

http://support.microsoft.com/kb/940184

And lastly, here is a patch from Microsoft which can help a lot in Server 2003, even on fully-updated machines:

http://support.microsoft.com/default.aspx?scid=kb;en-us;891957

Categories:      

==============

VSS for Linux
article #363, updated 4507 days ago

Here is a product, free for commercial and non-profit use, which does the job of VSS under Linux:

http://www.r1soft.com/tools/linux-hot-copy/

Categories: