Some Notes on the Machines

Recently a number of things have come to light together, the combination of which is remarkable.

1. The System Volume Information folder, on very many slow Windows machines, has a lot of junk files within, files that serve no purpose. These are called “orphan shadows”.
2. Orphan shadows occur when the RAID or other drive capability of a Windows machine, cannot keep up with demands on the Volume Shadow Services (VSS) subsystem of Windows, and/or has to abandon an effort in midstream. VSS is used by a huge variety of Windows and application operations.
3. The best-understood examples of events causing orphan shadows, are hard poweroffs in the middle of many different operations large and small. Another is when block-level backups fail.
4. Other examples are simultaneous heavy demands which are too high for the RAID or drive capability. For instance, if demands on RAID are already fairly high, a large database request happening at the same moment as the incremental for a backup will cause one or both to fail, and will create orphan shadow(s) for one or both.
5. As orphan shadows build up, they take more and more space in System Volume Information, and when SVI has more than 20 gigabytes or so of these things, all VSS operations get slower and slower, presumably because the VSS system has to choose to bypass all of the orphans over and over again.
6. SVI has been seen to rise to hundreds of gigabytes, sometimes taking hours to clear.

The short-term step, to get things working, is to clear SVI. On a client OS, this is done thus:

vssadmin delete shadows /all

On a server OS, we do the above, and then we also run diskshadow, and within that little environment, we do:

delete shadows all

Alternatively, we can replace all of the above steps with:

wmic shadowcopy delete /nointeractive

One situation has been seen thus far, where the above two do not do the job, where SVI remains huge. No standard solution has been identified for this as of yet. Up until now, the symptoms have been clear and obvious, but causality a lot more mysterious, and we have often alleviated the situation via removal of OEMware, BIOS, firmware, and driver updates, and use of tweaking tools. On a server, one can remove Windows Defender:

Uninstall-WindowsFeature Windows-Defender

and this helps a lot (solved one big problem so far) on server, but not on a client OS, Microsoft doesn’t allow it. Also, methods to disable bits of Windows Defender on client OS have been disallowed increasingly, as build upgrades have been given.

One step which has helped is to CHKDSK /F the volumes and reboot, and then try the standard cleanout steps again.

Just today, some VSS tweaks have manifest:

https://docs.microsoft.com/en-us/windows/win32/backup/registry-keys-for-backup-and-restore?redirectedfrom=MSDN#maxshadowcopies

specifically this:

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\VSS\Settings
DWORD MaxShadowCopies

and this:

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\VolSnap
DWORD MinDiffAreaFileSize

but it will be a while of testing before any confidence is available as to how to use these to help. Thus far, MaxShadowCopies of 8 and MinDiffAreaFileSize of 128 seems to be helping.

Recently it was discovered that a Windows server was running very slow because the Security log’s maximum size was set to 40 gigabytes. Here is a Powershell bit which will look at all event logs, set their max size to 2.5M if set larger, and clear them. Seems to free up a nice healthy dollop of performance in general.

wevtutil el | Foreach-Object {
	$LogObject = Get-WinEvent -ListLog $_
	If ( $LogObject.MaximumSizeInBytes -gt 2500KB ) {
		"$_ has max set to larger than 2.5M.  Setting to 2.5M."
		$LogObject.MaximumSizeInBytes = 2500KB
		$LogObject.SaveChanges()
		}
	wevtutil cl $_
	"$_ cleared."
}

There are times when an operation, a software install or configure perhaps, will error with “Cannot open log for source ———-. You may not have write access.” The below will do the above, and also give read/write to every local admin.

wevtutil el | Foreach-Object {
	wevtutil sl $_ "/ca:O:BAG:SYD:(A;;0x1;;;SY)(A;;0x5;;;BA)(A;;0x1;;;LA)(A;;0x3;;;LA)"
	$LogObject = Get-WinEvent -ListLog $_
	If ( $LogObject.MaximumSizeInBytes -gt 2500KB ) {
		"$_ has max set to larger than 2.5M.  Setting to 2.5M."
		$LogObject.MaximumSizeInBytes = 2500KB
		$LogObject.SaveChanges()
		}
	wevtutil cl $_
	"$_ cleared."
}

The security string is written in something called SDDL. Some more info:

https://itconnect.uw.edu/wares/msinf/other-help/understanding-sddl-syntax/

Appears excellent on the desktop.

http://www.puransoftware.com/Puran-Defrag.html

The journal can, reportedly, get huge. Here’s how to delete it for C:

fsutil usn deletejournal /n C:

Warnings and info is here, but haven’t seen any signs of the warned items, perhaps they happen mostly with very large and/or very slow machines:

docs.microsoft.com/en-us/windows-server/administration/windows-commands/fsutil-usn

Here is a useful guide to HPE RAID model numbers:

medium.com/@ITsolutions/selecting-the-best-hpe-smart-array-controller-for-your-server-da10b7424c8c

UltraDefrag still comes in its free open source version:

sourceforge.net/projects/ultradefrag/

as well as a newer, more developed paid version:

ultradefrag.net

Something new. Reported to help a lot, especially with Internet connections higher than 10Mbps.

Basically, we browse here in the registry:

Computer\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\LanmanServer\Parameters

we create (or edit) DWORD IRPStackSize, and we set it to 32. Default is 15, and Microsoft reportedly says that values 33-38 are fraught with peril.

Something relatively new. Very interesting changes from some Microsoft documentation, searchable here. Performance improvements have been visible in general behavior of all machines tested for this so far. On some, reported CPU speed does still change over time. BIOS changes are likely to assist as well.

The lines below work in administrative Powershell. It creates a new power scheme called “Maximum Performance”, based on the hidden built-in “Ultimate Performance” if it exists on your system. To revert, just go into Power in the Control Panel and reselect your original power scheme.

powercfg -duplicatescheme e9a42b02-d5df-448d-aa00-03f14749eb61

$oldpower = powercfg -getactivescheme
"Creating power scheme:  Maximum Performance"

$newpower = powercfg -duplicatescheme scheme_current
$newpower = ($newpower[19..54] -join "")
powercfg -changename $newpower "Maximum Performance"
powercfg -setactive $newpower

# Makes maximum CPU speeds available, by default they're not
powercfg -setacvalueindex scheme_current sub_processor PERFBOOSTMODE 2
powercfg -setacvalueindex scheme_current sub_processor PERFBOOSTMODE1 2
powercfg -setacvalueindex scheme_current sub_processor PERFINCTHRESHOLD 0
powercfg -setacvalueindex scheme_current sub_processor PERFINCTHRESHOLD1 0
powercfg -setacvalueindex scheme_current sub_processor PERFINCTIME 0
powercfg -setacvalueindex scheme_current sub_processor PERFINCTIME1 0
powercfg -setacvalueindex scheme_current sub_processor PERFDECTHRESHOLD 100
powercfg -setacvalueindex scheme_current sub_processor PERFDECTHRESHOLD1 100
powercfg -setacvalueindex scheme_current sub_processor LATENCYHINTPERF 0
powercfg -setacvalueindex scheme_current sub_processor LATENCYHINTPERF1 0
powercfg -setacvalueindex scheme_current sub_processor PERFAUTONOMOUS 0
powercfg -setacvalueindex scheme_current sub_processor PERFDUTYCYCLING 0

# Sets overall throttles to maximum
powercfg -setacvalueindex scheme_current sub_processor PROCTHROTTLEMAX 100
powercfg -setacvalueindex scheme_current sub_processor PROCTHROTTLEMAX1 100
powercfg -setacvalueindex scheme_current sub_processor PROCTHROTTLEMIN 100
powercfg -setacvalueindex scheme_current sub_processor PROCTHROTTLEMIN1 100
powercfg -setacvalueindex scheme_current sub_processor HETEROCLASS1INITIALPERF 100
powercfg -setacvalueindex scheme_current sub_processor HETEROCLASS0FLOORPERF 100

# Turns off CPU core controls, tells OS to just use them all.
powercfg -setacvalueindex scheme_current sub_processor CPMAXCORES 100
powercfg -setacvalueindex scheme_current sub_processor CPMINCORES 100
powercfg -setacvalueindex scheme_current sub_processor DISTRIBUTEUTIL 0
powercfg -setacvalueindex scheme_current sub_processor CPDISTRIBUTION 0

# Minimizes CPU spinup time, and maximizes spindown time, just in case
powercfg -setacvalueindex scheme_current sub_processor CPINCREASETIME 0
powercfg -setacvalueindex scheme_current sub_processor CPDECREASETIME 100
powercfg -setacvalueindex scheme_current sub_processor CPHEADROOM 0
powercfg -setacvalueindex scheme_current sub_processor CPCONCURRENCY 0
powercfg -setacvalueindex scheme_current sub_processor LATENCYHINTUNPARK 0
powercfg -setacvalueindex scheme_current sub_processor LATENCYHINTUNPARK1 0

# Sets energy savings preference to zero
powercfg -setacvalueindex scheme_current sub_processor PERFEPP 0

# Commits all above changes to current power plan
powercfg -setactive scheme_current

Some detail can be had here and here.

These changes are disrecommended for cooling-poor laptops. And one might want to watch the temperatures of poorly built desktops and even some poorly built servers, too.

A version of the above which alters the original power scheme, and runs in administrative CMD, is below. The below is not very easily reversible. It does not use the “Ultimate Performance” builtin.

REM Makes maximum CPU speeds available, by default they're not
powercfg -setacvalueindex scheme_current sub_processor PERFBOOSTMODE 2
powercfg -setacvalueindex scheme_current sub_processor PERFBOOSTMODE1 2
powercfg -setacvalueindex scheme_current sub_processor PERFINCTHRESHOLD 0
powercfg -setacvalueindex scheme_current sub_processor PERFINCTHRESHOLD1 0
powercfg -setacvalueindex scheme_current sub_processor PERFINCTIME 0
powercfg -setacvalueindex scheme_current sub_processor PERFINCTIME1 0
powercfg -setacvalueindex scheme_current sub_processor PERFDECTHRESHOLD 100
powercfg -setacvalueindex scheme_current sub_processor PERFDECTHRESHOLD1 100
powercfg -setacvalueindex scheme_current sub_processor LATENCYHINTPERF 0
powercfg -setacvalueindex scheme_current sub_processor LATENCYHINTPERF1 0
powercfg -setacvalueindex scheme_current sub_processor PERFAUTONOMOUS 0
powercfg -setacvalueindex scheme_current sub_processor PERFDUTYCYCLING 0

REM Sets overall throttles to maximum
powercfg -setacvalueindex scheme_current sub_processor PROCTHROTTLEMAX 100
powercfg -setacvalueindex scheme_current sub_processor PROCTHROTTLEMAX1 100
powercfg -setacvalueindex scheme_current sub_processor PROCTHROTTLEMIN 100
powercfg -setacvalueindex scheme_current sub_processor PROCTHROTTLEMIN1 100
powercfg -setacvalueindex scheme_current sub_processor HETEROCLASS1INITIALPERF 100
powercfg -setacvalueindex scheme_current sub_processor HETEROCLASS0FLOORPERF 100

REM Turns off CPU core controls, tells OS to just use them all.
powercfg -setacvalueindex scheme_current sub_processor CPMAXCORES 100
powercfg -setacvalueindex scheme_current sub_processor CPMINCORES 100
powercfg -setacvalueindex scheme_current sub_processor DISTRIBUTEUTIL 0
powercfg -setacvalueindex scheme_current sub_processor CPDISTRIBUTION 0

REM Minimizes CPU spinup time, and maximizes spindown time, just in case
powercfg -setacvalueindex scheme_current sub_processor CPINCREASETIME 0
powercfg -setacvalueindex scheme_current sub_processor CPDECREASETIME 100
powercfg -setacvalueindex scheme_current sub_processor CPHEADROOM 0
powercfg -setacvalueindex scheme_current sub_processor CPCONCURRENCY 0
powercfg -setacvalueindex scheme_current sub_processor LATENCYHINTUNPARK 0
powercfg -setacvalueindex scheme_current sub_processor LATENCYHINTUNPARK1 0

REM Sets energy savings preference to zero
powercfg -setacvalueindex scheme_current sub_processor PERFEPP 0

REM Commits all above changes to current power plan
powercfg -setactive scheme_current

There is a built-in, hidden, “Ultimate Performance” power scheme in Windows 10, 11, and probably others:

powercfg -duplicatescheme e9a42b02-d5df-448d-aa00-03f14749eb61

Does not include everything in a page on this site, but integration will follow soon :-)

Not something always to do — I’m kind of allergic to removing standard components — but certainly if you have anemic RAID or a slow hard drive, this will help, even before the necessary reboot:

Uninstall-WindowsFeature Windows-Defender

Servers only, alas, and this may go away in later server builds.