Some Notes on the Machines

Here is a useful guide to HPE RAID model numbers:

medium.com/@ITsolutions/selecting-the-best-hpe-smart-array-controller-for-your-server-da10b7424c8c

UltraDefrag still comes in its free open source version:

sourceforge.net/projects/ultradefrag/

as well as a newer, more developed paid version:

ultradefrag.net

Recently a number of things have come to light together, the combination of which is remarkable.

1. The System Volume Information folder, on very many slow Windows machines, has a lot of junk files within, files that serve no purpose. These are called “orphan shadows”.
2. Orphan shadows occur when the RAID or other drive capability of a Windows machine, cannot keep up with demands on the Volume Shadow Services (VSS) subsystem of Windows, and/or has to abandon an effort in midstream. VSS is used by a huge variety of Windows and application operations.
3. The best-understood examples of events causing orphan shadows, are hard poweroffs in the middle of many different operations large and small. Another is when block-level backups fail.
4. Other examples are simultaneous heavy demands which are too high for the RAID or drive capability. For instance, if demands on RAID are already fairly high, a large database request happening at the same moment as the incremental for a backup will cause one or both to fail, and will create orphan shadow(s) for one or both.
5. As orphan shadows build up, they take more and more space in System Volume Information, and when SVI has more than 20 gigabytes or so of these things, all VSS operations get slower and slower, presumably because the VSS system has to choose to bypass all of the orphans over and over again.
6. SVI has been seen to rise to hundreds of gigabytes, sometimes taking hours to clear.

The short-term step, to get things working, is to clear SVI. On a client OS, this is done thus:

vssadmin delete shadows /all

On a server OS, we do the above, and then we also run diskshadow, and within that little environment, we do:

delete shadows all

Alternatively, we can replace all of the above steps with:

wmic shadowcopy delete /nointeractive

One situation has been seen thus far, where the above two do not do the job, where SVI remains huge. No standard solution has been identified for this as of yet. Up until now, the symptoms have been clear and obvious, but causality a lot more mysterious, and we have often alleviated the situation via removal of OEMware, BIOS, firmware, and driver updates, and use of tweaking tools. On a server, one can remove Windows Defender:

Uninstall-WindowsFeature Windows-Defender

and this helps a lot (solved one big problem so far) on server, but not on a client OS, Microsoft doesn’t allow it. Also, methods to disable bits of Windows Defender on client OS have been disallowed increasingly, as build upgrades have been given.

One step which has helped is to CHKDSK /F the volumes and reboot, and then try the standard cleanout steps again.

Just today, some VSS tweaks have manifest:

https://docs.microsoft.com/en-us/windows/win32/backup/registry-keys-for-backup-and-restore?redirectedfrom=MSDN#maxshadowcopies

specifically this:

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\VSS\Settings
DWORD MaxShadowCopies

and this:

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\VolSnap
DWORD MinDiffAreaFileSize

but it will be a while of testing before any confidence is available as to how to use these to help. Thus far, MaxShadowCopies of 8 and MinDiffAreaFileSize of 128 seems to be helping.

Something new. Reported to help a lot, especially with Internet connections higher than 10Mbps.

Basically, we browse here in the registry:

Computer\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\LanmanServer\Parameters

we create (or edit) DWORD IRPStackSize, and we set it to 32. Default is 15, and Microsoft reportedly says that values 33-38 are fraught with peril.

Something relatively new. Very interesting changes from some Microsoft documentation, searchable here. Performance improvements have been visible in general behavior of all machines tested for this so far. On some, reported CPU speed does still change over time. BIOS changes are likely to assist as well.

The lines below work in administrative Powershell. It creates a new power scheme called “Maximum Performance”, based on the hidden built-in “Ultimate Performance” if it exists on your system. To revert, just go into Power in the Control Panel and reselect your original power scheme.

powercfg -duplicatescheme e9a42b02-d5df-448d-aa00-03f14749eb61

$oldpower = powercfg -getactivescheme
"Creating power scheme:  Maximum Performance"

$newpower = powercfg -duplicatescheme scheme_current
$newpower = ($newpower[19..54] -join "")
powercfg -changename $newpower "Maximum Performance"
powercfg -setactive $newpower

# Makes maximum CPU speeds available, by default they're not
powercfg -setacvalueindex scheme_current sub_processor PERFBOOSTMODE 2
powercfg -setacvalueindex scheme_current sub_processor PERFBOOSTMODE1 2
powercfg -setacvalueindex scheme_current sub_processor PERFINCTHRESHOLD 0
powercfg -setacvalueindex scheme_current sub_processor PERFINCTHRESHOLD1 0
powercfg -setacvalueindex scheme_current sub_processor PERFINCTIME 0
powercfg -setacvalueindex scheme_current sub_processor PERFINCTIME1 0
powercfg -setacvalueindex scheme_current sub_processor PERFDECTHRESHOLD 100
powercfg -setacvalueindex scheme_current sub_processor PERFDECTHRESHOLD1 100
powercfg -setacvalueindex scheme_current sub_processor LATENCYHINTPERF 0
powercfg -setacvalueindex scheme_current sub_processor LATENCYHINTPERF1 0
powercfg -setacvalueindex scheme_current sub_processor PERFAUTONOMOUS 0
powercfg -setacvalueindex scheme_current sub_processor PERFDUTYCYCLING 0

# Sets overall throttles to maximum
powercfg -setacvalueindex scheme_current sub_processor PROCTHROTTLEMAX 100
powercfg -setacvalueindex scheme_current sub_processor PROCTHROTTLEMAX1 100
powercfg -setacvalueindex scheme_current sub_processor PROCTHROTTLEMIN 100
powercfg -setacvalueindex scheme_current sub_processor PROCTHROTTLEMIN1 100
powercfg -setacvalueindex scheme_current sub_processor HETEROCLASS1INITIALPERF 100
powercfg -setacvalueindex scheme_current sub_processor HETEROCLASS0FLOORPERF 100

# Turns off CPU core controls, tells OS to just use them all.
powercfg -setacvalueindex scheme_current sub_processor CPMAXCORES 100
powercfg -setacvalueindex scheme_current sub_processor CPMINCORES 100
powercfg -setacvalueindex scheme_current sub_processor DISTRIBUTEUTIL 0
powercfg -setacvalueindex scheme_current sub_processor CPDISTRIBUTION 0

# Minimizes CPU spinup time, and maximizes spindown time, just in case
powercfg -setacvalueindex scheme_current sub_processor CPINCREASETIME 0
powercfg -setacvalueindex scheme_current sub_processor CPDECREASETIME 100
powercfg -setacvalueindex scheme_current sub_processor CPHEADROOM 0
powercfg -setacvalueindex scheme_current sub_processor CPCONCURRENCY 0
powercfg -setacvalueindex scheme_current sub_processor LATENCYHINTUNPARK 0
powercfg -setacvalueindex scheme_current sub_processor LATENCYHINTUNPARK1 0

# Sets energy savings preference to zero
powercfg -setacvalueindex scheme_current sub_processor PERFEPP 0

# Commits all above changes to current power plan
powercfg -setactive scheme_current

Some detail can be had here and here.

These changes are disrecommended for cooling-poor laptops. And one might want to watch the temperatures of poorly built desktops and even some poorly built servers, too.

A version of the above which alters the original power scheme, and runs in administrative CMD, is below. The below is not very easily reversible. It does not use the “Ultimate Performance” builtin.

REM Makes maximum CPU speeds available, by default they're not
powercfg -setacvalueindex scheme_current sub_processor PERFBOOSTMODE 2
powercfg -setacvalueindex scheme_current sub_processor PERFBOOSTMODE1 2
powercfg -setacvalueindex scheme_current sub_processor PERFINCTHRESHOLD 0
powercfg -setacvalueindex scheme_current sub_processor PERFINCTHRESHOLD1 0
powercfg -setacvalueindex scheme_current sub_processor PERFINCTIME 0
powercfg -setacvalueindex scheme_current sub_processor PERFINCTIME1 0
powercfg -setacvalueindex scheme_current sub_processor PERFDECTHRESHOLD 100
powercfg -setacvalueindex scheme_current sub_processor PERFDECTHRESHOLD1 100
powercfg -setacvalueindex scheme_current sub_processor LATENCYHINTPERF 0
powercfg -setacvalueindex scheme_current sub_processor LATENCYHINTPERF1 0
powercfg -setacvalueindex scheme_current sub_processor PERFAUTONOMOUS 0
powercfg -setacvalueindex scheme_current sub_processor PERFDUTYCYCLING 0

REM Sets overall throttles to maximum
powercfg -setacvalueindex scheme_current sub_processor PROCTHROTTLEMAX 100
powercfg -setacvalueindex scheme_current sub_processor PROCTHROTTLEMAX1 100
powercfg -setacvalueindex scheme_current sub_processor PROCTHROTTLEMIN 100
powercfg -setacvalueindex scheme_current sub_processor PROCTHROTTLEMIN1 100
powercfg -setacvalueindex scheme_current sub_processor HETEROCLASS1INITIALPERF 100
powercfg -setacvalueindex scheme_current sub_processor HETEROCLASS0FLOORPERF 100

REM Turns off CPU core controls, tells OS to just use them all.
powercfg -setacvalueindex scheme_current sub_processor CPMAXCORES 100
powercfg -setacvalueindex scheme_current sub_processor CPMINCORES 100
powercfg -setacvalueindex scheme_current sub_processor DISTRIBUTEUTIL 0
powercfg -setacvalueindex scheme_current sub_processor CPDISTRIBUTION 0

REM Minimizes CPU spinup time, and maximizes spindown time, just in case
powercfg -setacvalueindex scheme_current sub_processor CPINCREASETIME 0
powercfg -setacvalueindex scheme_current sub_processor CPDECREASETIME 100
powercfg -setacvalueindex scheme_current sub_processor CPHEADROOM 0
powercfg -setacvalueindex scheme_current sub_processor CPCONCURRENCY 0
powercfg -setacvalueindex scheme_current sub_processor LATENCYHINTUNPARK 0
powercfg -setacvalueindex scheme_current sub_processor LATENCYHINTUNPARK1 0

REM Sets energy savings preference to zero
powercfg -setacvalueindex scheme_current sub_processor PERFEPP 0

REM Commits all above changes to current power plan
powercfg -setactive scheme_current

There is a built-in, hidden, “Ultimate Performance” power scheme in Windows 10, 11, and probably others:

powercfg -duplicatescheme e9a42b02-d5df-448d-aa00-03f14749eb61

Does not include everything in a page on this site, but integration will follow soon :-)

Not something always to do — I’m kind of allergic to removing standard components — but certainly if you have anemic RAID or a slow hard drive, this will help, even before the necessary reboot:

Uninstall-WindowsFeature Windows-Defender

Servers only, alas, and this may go away in later server builds.

Run it like this from administrative CMD:

cmd.exe /c Cleanmgr /sageset:65535 & Cleanmgr /sagerun:65535

Gives you a lot more! Procedurally it’s a bit different too.

Appears to work in Vista, 7, and 8. A whole lot of web references are out there on this. Just one example:

http://helpdeskgeek.com/windows-vista-tips/manage-irq-settings-windows-vista-7/

One studies a list of IRQs and related hardware, and then choose the hardware to maximize priority upon using registry adds. Use msinfo32 (Hardware Resources, IRQs).

We then add registry entries here:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\PriorityControl

It is reported best to include IRQ zero (0) and eight (8) to start with, this is system timer and real time clock. To do these two, add the following-named items as DWORD in the above area, value 1 for the first, value 2 for the second:

IRQ0Priority
IRQ8Priority

When originally looking at this, I was solving a tendency for my softphone to cut out during any load situation or drive access, and so I checked my PC using Device Manager and msinfo32 as above, and also added subseqeuent priorities:

IRQ7Priority
IRQ20Priority
IRQ21Priority
IRQ4294967288Priority

because on this box, 7, 20, and 21 were USB, and 4294967288 was the active NIC. After you have made the changes, reboot.

The above also produced much better response to VNC and RDP redirection via Labtech.

At least one resource states that one must not set the same priority to multiple IRQs. Duplication may be the source of some reports saying it is placebo effect. Here is a very interesting post with some seriously good-quality testing and results:

https://www.tenforums.com/performance-maintenance/140553-regedit-priority-control-irq.html#6

Spotted these recently. All of these are to be run in administrative Powershell. On one Server 2019 machine, these boosted overall throughput from 7-25M to 600-700M.

File server ( Microsoft docs here )

Set-SmbServerConfiguration -EnableMultiChannel $true -force
Set-SmbServerConfiguration -EnableOplocks $true -force
Set-SmbServerConfiguration -ServerHidden $true -force
Set-SmbServerConfiguration -IrpStackSize 20 -force
Set-SmbServerConfiguration -MaxMpxCount 4096 -force
Set-SmbServerConfiguration -MaxWorkItems 16384 -force
Set-SmbServerConfiguration -MaxSessionPerConnection 16384 -force
Set-SmbServerConfiguration -TreatHostAsStableStorage $true -force

Clients ( Microsoft docs here )

Set-SmbClientConfiguration -EnableBandwidthThrottling $false -force
Set-SmbClientConfiguration -EnableLargeMtu $true -force
Set-SmbClientConfiguration -EnableLoadBalanceScaleOut $true -force
Set-SmbClientConfiguration -EnableMultiChannel $true -force
Set-SmbClientConfiguration -EnableSecuritySignature $false -force
Set-SmbClientConfiguration -MaxCmds 16384 -force
Set-SmbClientConfiguration -MaximumConnectionCountPerServer 32 -force
Set-SmbClientConfiguration -OplocksDisabled $false -force
Set-SmbClientConfiguration -RequireSecuritySignature $false -force
Set-SmbClientConfiguration -UseOpportunisticLocking $true -force
Set-SmbClientConfiguration -WindowSizeThreshold 2 -force

-WindowSizeThreshold may be good to set to 4 or 8.

Set-SmbServerConfiguration -IrpStackSize 15 -force
Set-SmbServerConfiguration -MaxMpxCount 50 -force
Set-SmbServerConfiguration -MaxWorkItems 1 -force
Set-SmbServerConfiguration -TreatAsStableStorage $false -force

Set-SmbServerConfiguration -EnableBandwidthThrottling $true -force
Set-SmbServerConfiguration -MaxCmds 50 -force
Set-SmbServerConfiguration -WindowSizeThreshold 1 -force