Some Notes on the Machines

…is often like a flying carpet in a hurricane.

There is some definite undocumented mystery concerning Volume Shadow Services in Windows. In general we are told to use VSSADMIN to do maintenance, and it does a lot, and helps a lot. But recently there was a Server 2012 R2 machine using 280G of space for System Volume Information on C:, and after CHKDSK and various DISMs it still was using 280G. So I tried removing all orphan shadows with VSSADMIN, and it found one and removed it; almost zero change comparatively. And then I searched a little deeper.

DISKSHADOW is built into Windows 2012 R2 and later, and earlier too I think, not sure how early. It may be a successor to VSHADOW which was an SDK add-on to 2003. Regardless, DISKSHADOW is a command-line environment of its own sort of like NSLOOKUP and DISKPART (!), not a simple command, can run a script of its own commands, and one of its commands is:

DELETE SHADOWS ALL

Now VSSADMIN DELETE SHADOWS /ALL deletes all orphan shadows, all VSS shadow copy sets which Windows knows are good to delete. The above within DISKSHADOW is a different animal altogether: it deletes them all. And does not appear to report anything to event logs (!). And definitely frees up a whole lot of space. And also, definitely not least, is flagged as infection activity by certain high-test super-anti-malware tools, when run! That was amazing, a Windows built-in being run with one of its own recommended commands, flagged. But I’ll think that that means this is to be used only when very needed. There may be gotchas I don’t know about yet.

As I write, the System Volume Information on this C: drive has been shrunken 290 (two hundred ninety) gigabytes, and everything is still running fine. There were originally 522 (five hundred twenty-two) shadow copies hanging out there of many different sizes, and DISKSHADOW was able to delete them all, all server services appear AOK.

Something relatively new. Very interesting changes from some Microsoft documentation, searchable here. Performance improvements have been visible in general behavior of all machines tested for this so far. On some, reported CPU speed does still change over time. BIOS changes are likely to assist as well.

The lines below work in administrative Powershell. They report the current power scheme and create a new one for the new settings. To revert, just go into Power in the Control Panel and reselect your original power scheme.

$oldpower = powercfg -getactivescheme
$oldname = $oldpower[58..100] -join ""
$oldname = $oldname.Substring(0,$oldname.Length-1)
"Current power scheme name:  $oldname"
"Creating power scheme:  CPU Special"

$newpower = powercfg -duplicatescheme scheme_current
$newpower = ($newpower[19..54] -join "")
powercfg -changename $newpower "CPU Special"
powercfg -setactive $newpower

# Makes maximum CPU speeds available, by default they're not
powercfg -setacvalueindex scheme_current sub_processor PERFBOOSTMODE 0
powercfg -setdcvalueindex scheme_current sub_processor PERFBOOSTMODE1 0
powercfg -setacvalueindex scheme_current sub_processor PERFINCTHRESHOLD 1
powercfg -setacvalueindex scheme_current sub_processor PERFINCTHRESHOLD1 1
powercfg -setacvalueindex scheme_current sub_processor PERFINCTIME 1
powercfg -setacvalueindex scheme_current sub_processor PERFINCTIME1 1
powercfg -setacvalueindex scheme_current sub_processor PERFDECTHRESHOLD 100
powercfg -setacvalueindex scheme_current sub_processor PERFDECTHRESHOLD1 100
powercfg -setacvalueindex scheme_current sub_processor LATENCYHINTPERF 0
powercfg -setacvalueindex scheme_current sub_processor LATENCYHINTPERF1 0
powercfg -setacvalueindex scheme_current sub_processor PERFAUTONOMOUS 0
powercfg -setacvalueindex scheme_current sub_processor PERFDUTYCYCLING 0

# Sets overall throttles to maximum
powercfg -setacvalueindex scheme_current sub_processor PROCTHROTTLEMAX 100
powercfg -setacvalueindex scheme_current sub_processor PROCTHROTTLEMAX1 100
powercfg -setacvalueindex scheme_current sub_processor PROCTHROTTLEMIN 100
powercfg -setacvalueindex scheme_current sub_processor PROCTHROTTLEMIN1 100
powercfg -setacvalueindex scheme_current sub_processor HETEROCLASS1INITIALPERF 100
powercfg -setacvalueindex scheme_current sub_processor HETEROCLASS0FLOORPERF 100

# Turns off CPU core controls, tells OS to just use them all.
powercfg -setacvalueindex scheme_current sub_processor CPMAXCORES 100
powercfg -setacvalueindex scheme_current sub_processor CPMINCORES 100
powercfg -setacvalueindex scheme_current sub_processor DISTRIBUTEUTIL 0
powercfg -setacvalueindex scheme_current sub_processor CPDISTRIBUTION 1

# Minimizes CPU spinup time, and maximizes spindown time, just in case
powercfg -setacvalueindex scheme_current sub_processor CPINCREASETIME 1
powercfg -setacvalueindex scheme_current sub_processor CPDECREASETIME 100
powercfg -setacvalueindex scheme_current sub_processor CPHEADROOM 1
powercfg -setacvalueindex scheme_current sub_processor CPCONCURRENCY 1
powercfg -setacvalueindex scheme_current sub_processor LATENCYHINTUNPARK 1
powercfg -setacvalueindex scheme_current sub_processor LATENCYHINTUNPARK1 1

# Sets energy savings preference to zero
powercfg -setacvalueindex scheme_current sub_processor PERFEPP 0

# Commits all above changes to current power plan
powercfg -setactive scheme_current

Some detail can be had here and here.

These changes are disrecommended for cooling-poor laptops. And one might want to watch the temperatures of poorly built desktops and even some poorly built servers, too.

A version of the above which alters the original power scheme, and runs in administrative CMD, is below. The below is not very easily reversible.

REM Makes maximum CPU speeds available, by default they're not
powercfg -setacvalueindex scheme_current sub_processor PERFBOOSTMODE 0
powercfg -setdcvalueindex scheme_current sub_processor PERFBOOSTMODE1 0
powercfg -setacvalueindex scheme_current sub_processor PERFINCTHRESHOLD 1
powercfg -setacvalueindex scheme_current sub_processor PERFINCTHRESHOLD1 1
powercfg -setacvalueindex scheme_current sub_processor PERFINCTIME 1
powercfg -setacvalueindex scheme_current sub_processor PERFINCTIME1 1
powercfg -setacvalueindex scheme_current sub_processor PERFDECTHRESHOLD 100
powercfg -setacvalueindex scheme_current sub_processor PERFDECTHRESHOLD1 100
powercfg -setacvalueindex scheme_current sub_processor LATENCYHINTPERF 0
powercfg -setacvalueindex scheme_current sub_processor LATENCYHINTPERF1 0
powercfg -setacvalueindex scheme_current sub_processor PERFAUTONOMOUS 0
powercfg -setacvalueindex scheme_current sub_processor PERFDUTYCYCLING 0

REM Sets overall throttles to maximum
powercfg -setacvalueindex scheme_current sub_processor PROCTHROTTLEMAX 100
powercfg -setacvalueindex scheme_current sub_processor PROCTHROTTLEMAX1 100
powercfg -setacvalueindex scheme_current sub_processor PROCTHROTTLEMIN 100
powercfg -setacvalueindex scheme_current sub_processor PROCTHROTTLEMIN1 100
powercfg -setacvalueindex scheme_current sub_processor HETEROCLASS1INITIALPERF 100
powercfg -setacvalueindex scheme_current sub_processor HETEROCLASS0FLOORPERF 100

REM Turns off CPU core controls, tells OS to just use them all.
powercfg -setacvalueindex scheme_current sub_processor CPMAXCORES 100
powercfg -setacvalueindex scheme_current sub_processor CPMINCORES 100
powercfg -setacvalueindex scheme_current sub_processor DISTRIBUTEUTIL 0
powercfg -setacvalueindex scheme_current sub_processor CPDISTRIBUTION 1

REM Minimizes CPU spinup time, and maximizes spindown time, just in case
powercfg -setacvalueindex scheme_current sub_processor CPINCREASETIME 1
powercfg -setacvalueindex scheme_current sub_processor CPDECREASETIME 100
powercfg -setacvalueindex scheme_current sub_processor CPHEADROOM 1
powercfg -setacvalueindex scheme_current sub_processor CPCONCURRENCY 1
powercfg -setacvalueindex scheme_current sub_processor LATENCYHINTUNPARK 1
powercfg -setacvalueindex scheme_current sub_processor LATENCYHINTUNPARK1 1

REM Sets energy savings preference to zero
powercfg -setacvalueindex scheme_current sub_processor PERFEPP 0

REM Commits all above changes to current power plan
powercfg -setactive scheme_current

DTX download:
https://www.storagecraft.com/downloads/diagnostic-tool-xplatform

DTX run:
https://support.storagecraft.com/s/article/How-To-Gathering-diagnostics-for-StorageCraft-Support?language=en_US

Enable ImageManager verbose logging:
https://support.storagecraft.com/s/article/Enable-Verbose-logging-for-ImageManager-6-6-and-later?language=en_US

In general Virtual Machine Queues are often disabled on Hyper-V hosts, to solve network slowness. However, if we go here:

HKLM\SYSTEM\CurrentControlSet\Services\VMSMP\Parameters

and add DWORD BelowTenGigVmqEnabled and set it to 1, or TenGigVmqEnabled for 10G, we can enable Virtual Machine Queues and get great speed indeed.

If this is going on, there is more than likely to be a malfunctioning sensor or two. In this situation you’ll need this:

https://www.techpowerup.com/download/techpowerup-throttlestop/

to get the machine to run at good speed. Do be careful with temperatures.

Spotted these recently. All of these are to be run in administrative Powershell. On one Server 2019 machine, these boosted overall throughput from 7-25M to 600-700M.

File server ( Microsoft docs here )

Set-SmbServerConfiguration -EnableMultiChannel $true -force
Set-SmbServerConfiguration -EnableOplocks $true -force
Set-SmbServerConfiguration -ServerHidden $true -force
Set-SmbServerConfiguration -IrpStackSize 20 -force
Set-SmbServerConfiguration -MaxMpxCount 4096 -force
Set-SmbServerConfiguration -MaxWorkItems 16384 -force
Set-SmbServerConfiguration -MaxSessionPerConnection 16384 -force
Set-SmbServerConfiguration -TreatHostAsStableStorage $true -force

Default server resettings:

Set-SmbServerConfiguration -IrpStackSize 15 -force
Set-SmbServerConfiguration -MaxMpxCount 50 -force
Set-SmbServerConfiguration -MaxWorkItems 1 -force
Set-SmbServerConfiguration -TreatAsStableStorage $false -force

Clients ( Microsoft docs here )

Set-SmbClientConfiguration -EnableBandwidthThrottling $false -force
Set-SmbClientConfiguration -EnableLargeMtu $true -force
Set-SmbClientConfiguration -EnableLoadBalanceScaleOut $true -force
Set-SmbClientConfiguration -EnableMultiChannel $true -force
Set-SmbClientConfiguration -EnableSecuritySignature $false -force
Set-SmbClientConfiguration -MaxCmds 16384 -force
Set-SmbClientConfiguration -MaximumConnectionCountPerServer 32 -force
Set-SmbClientConfiguration -OplocksDisabled $false -force
Set-SmbClientConfiguration -RequireSecuritySignature $false -force
Set-SmbClientConfiguration -UseOpportunisticLocking $true -force
Set-SmbClientConfiguration -WindowSizeThreshold 2 -force

-WindowSizeThreshold may be good to set to 4 or 8.

Default client resettings:

Set-SmbServerConfiguration -EnableBandwidthThrottling $true -force
Set-SmbServerConfiguration -MaxCmds 50 -force
Set-SmbServerConfiguration -WindowSizeThreshold 1 -force

There are many ways of doing this. Here is one way to bring everything into a single consistent behavior, a landing place from which you can vary slightly at need.

This presumes that you have Azure/AD sync installed and working in general, and yielding errors in the Synchronization Service window for one or more users. Make sure that you don’t have duplicate email addresses in AD, that could be bad.

The first steps are in Active Directory Users and Computers.

1. Set the user’s email correctly in his/her AD object, in “E-mail” under General.
2. Set proxyAddresses in the Attribute Editor. The primary email address has to be the same, and in proxyAddresses has to be of the format “SMTP:email@domain.com”. There can be others in proxyAddresses but smtp must be lowercase. Also in proxyAddresses, set mailNickname blank.
3. Under Account, either the user login name plus the dropdown domain is to be the same as the above, or it is to be a valid login according to the O365 console. If the dropdown domain list is local only, you can add the Internet domain list in Active Directory Domains and Trusts, with a right-click on the root level in the left pane of that window, and then an add of one or more alternative UPN suffixes. Then restart ADUC and the domain(s) you just added will be available in the dropdown.

Now we do some other things.

4. Run Azure/AD sync, this is CMD, do it as administrator:

repadmin /syncall /AdeP
@"%SystemRoot%\System32\WindowsPowerShell\v1.0\powershell.exe" -NoProfile -InputFormat None -ExecutionPolicy Bypass -Command ^
"Import-Module ADSync; Start-ADSyncSyncCycle -PolicyType Delta"

5. If you see errors in the Synchronization Service window, run something called a “hard match”, which changes enough to connect the AD user of that email address, with the Azure user of that email address. Here’s a script for it, Powershell; you’ll need to log into the O365 tenant. Note that while $AzureUPN is the O365 “primary email address”, $ADUPN is the double item under Profile in AD, the username plus the domain dropdown.

$AzureUPN = "user_email_on_azure@domain.com"
$ADUPN = "user_login_in_ad@domain.com"

"First connect to AzureAD:"
Connect-AzureAD

"Now get Azure ImmutableID:"
$AzureUser = Get-AzureADUser -SearchString $AzureUPN
$AzureUser.ImmutableID
""
""
"Extracting AD GUID..."
ldifde -f export.txt -r "(Userprincipalname=$ADUPN)" -l *
$ADGUID = (-split (type export.txt | select-string "ObjectGUID"))[1]
"Extracted AD GUID:"
$ADGUID
""
""
Set-AzureADuser -ObjectID $AzureUser.ObjectID -ImmutableID $ADGUID

"New Azure ImmutableID re-extracted for confirmation:"
$AzureUser = Get-AzureADUser -SearchString $AzureUPN
$AzureUser.ImmutableID

Things have changed. Best recommend now is CloudFlare’s. General:

1.1.1.1
1.0.0.1

CloudFlare’s, filtered for malware:

1.1.1.2
1.0.0.2

And CloudFlare’s, filtered for malware and “adult content”:

1.1.1.3
1.0.0.3

And one more provider, almost as good in this writer’s experience, Level 3. Unfiltered.

209.244.0.3
209.244.0.4

No other recommends at this writing.

Microsoft is heavily using something called GeoIP, to optimize Internet data routing for its services, including Skype, Office 365, and all of the others.

All of the code below is within ‘nslookup’, running in CMD on Windows.

The way this works, basically, is different IP sets are reported by DNS lookups, depending on the upstream DNS server being polled. So if, like many right now, you were using Google’s DNS (8.8.8.8 and 8.8.4.4) on your LAN, and did nslookup on the recommended test hostname, outlook.office365.com, you would see this:

> outlook.office365.com
Server:  google-public-dns-a.google.com
Address:  8.8.8.8

Non-authoritative answer:
Name:    outlook-namsouth2.office365.com
Addresses:  2603:1036:0:26::2
          2603:1036:102:90::2
          2603:1036:404:a4::2
          2603:1036:102:107::2
          2603:1036:102:b8::2
          2603:1036:404:11b::2
          2603:1036:404:3f::2
          2603:1036:3:12e::2
          2603:1036:102:3e::2
          2603:1036:404:11c::2
          40.97.170.162
          40.97.30.130
          40.97.170.178
          40.97.142.18
          40.97.41.98
          40.97.162.130
          40.97.154.66
          40.97.166.178
          40.97.117.242
          40.97.119.178
Aliases:  outlook.office365.com
          outlook.ha.office365.com
          outlook.office365.com.g.office365.com

>

But on the other hand, if you were using OpenDNS (208.67.220.220/222.222), you would see this:

> outlook.office365.com
Server:  resolver1.opendns.com
Address:  208.67.222.222

Non-authoritative answer:
Name:    outlook-namsouth4.office365.com
Addresses:  2603:1036:d01:2::2
          2603:1036:101:2::2
          2a01:111:f400:31ab::2
          2603:1036:902:a3::2
          2603:1036:906:4d::2
          2603:1036:405:2::2
          2603:1036:405:15::2
          2603:1036:404:67::2
          2603:1036:100::2
          40.97.142.18
          40.97.41.98
          40.97.162.130
          40.97.154.66
          40.97.166.178
          40.97.117.242
          40.97.119.178
          40.97.170.162
          40.97.30.130
          40.97.170.178
Aliases:  outlook.office365.com
          outlook.ha.office365.com
          outlook.office365.com.g.office365.com

>

The most important thing to observe in the above, is that the IP set is different. And if you try pings from your test PC to each of the above IPs, you will notice major differences. In recent testing, most of Google’s results ping much slower (higher, in milliseconds) than OpenDNS’s. But we found OpenDNS’s pings noticeably slower than our current known best of breed, Level3 (209.244.0.3/4):

> outlook.office365.com
Server:  resolver1.level3.net
Address:  209.244.0.3

Non-authoritative answer:
Name:    outlook-namsouth.office365.com
Addresses:  2603:1036:404:16::2
          2603:1036:404:b6::2
          2603:1036:102:16::2
          2603:1036:405:29::2
          2603:1036:906:4f::2
          2603:1036:d00::2
          2603:1036:102:8f::2
          2603:1036:405:4a::2
          2603:1036:4:4c::2
          40.97.133.130
          40.97.132.194
          40.97.125.114
          40.97.132.226
          40.97.126.50
          40.97.31.50
          40.97.164.146
          40.97.136.194
          40.97.166.34
Aliases:  outlook.office365.com
          outlook.ha.office365.com
          outlook.office365.com.g.office365.com

>

We have also noticed that the lists of IPs do not correspond to names, i.e., outlook-namsouth3 does not return the same IP list each time. So there is a lot of highly complex geographically-centered IP routing by DNS, going on, by Microsoft, and Level3 seems to cooperate best.

The upshot is, if you see any Microsoft cloud-based services being slow, hesitating, freezing up, or losing connection regularly, switch your LAN’s DNS forwarders to Level 3, and you may well knock the problem out most easily. CloudFlare’s DNS works as well if not better.