Some Notes on the Machines

This scenario is common, though not necessarily obviously so:

• Windows 8.1/2012R2 through 11/2022 is corrupt.
• DISM (DISM /Online /Cleanup-Image /RestoreHealth) won’t fix it, it reports needed data not available.
• The above DISM adding /Source:...INSTALL.WIM, where INSTALL.WIM resides on a mounted ISO of that Windows, even a freshly downloaded ISO, still fails due to needed data not available.

The fact is that #3 above occurs because freshly downloaded ISOs lack many Windows updates. This is not quickly resolvable, but it is very resolvable given some time. Using this method, we will fix corrupt Windows images. You’ll need many gigabytes of disk space to work with.

So. The first step is to decide which Windows version we’re going to create an up-to-date INSTALL.WIM for, to be used by DISM. For this example we’ll say 64-bit Windows Server 2016. The ISO can be had (at this writing) at this page.

Download that ISO, and unpack it. In the SOURCES folder at the root, there is a large file called INSTALL.WIM, 5.6 gigs at this writing. Create folder D:\Server2016WIM (or whatever works for you) and move that WIM into it.

You will need to go to the Properties of the INSTALL.WIM file, and uncheck “Read-Only”, for some reason.

Now we need a tool called wsusoffline. It downloads the updates which we are going to put in that INSTALL.WIM file. It comes from www.wsusoffline.net. Unpack the zip file and run UpdateGenerator.exe. Set it up like this:

Make sure the “USB medium” target directory is useful and handy, We’re not using USB for this, just a different folder on the same drive that the WIM is in. Click Start, and it will download, verify, and store a whole lot of Windows updates in that folder. The above setup does make a repository for every 64-bit Windows 10-class OS, including Server 2016 and 2019. 2022 isn’t available here yet, we can expect it to be in the next edition of wsusoffline.

Once that download is done, we need to see which install(s) are in that WIM file. We do this thusly:

CD \SERVER2016WIM
DISM /Get-WimInfo /wimFile:install.wim

In the ISO I just downloaded, there are four installs, Server 2016 Standard without (1) and with (2) GUI, and Server 2016 Datacenter without (3) and with (4). Because our server to be repaired is not bare-bones, we want to update 2 and 4 just in case. So we prepare,

D:
CD \SERVER2016WIM
MKDIR Mounted-WIM

and mount the WIM in index 2:

DISM /Mount-WIM /WimFile:INSTALL.WIM /index:2 /MountDir:Mounted-WIM

The above will take a little while. We study D:\wsusoffline-output (see the wsusoffline image above), and we find that the Windows 10-class updates are here:

D:\wsusoffline-output\w100-x64\glb

So when the mount is done, we run the updates, still with current directory as above:

DISM /image:Mounted-Wim /Add-Package /PackagePath:D:\wsusoffline-output\w100-x64\glb

and a lot of nonapplicable updates are reported as errors, and a lot of successful updates are also reported, in a long report stream. This happens because every patch for every build of 10, 2016, and 2019 are all in that one folder. DISM knows which ones it needs, and will report success for those. But it often does not get them all on the first pass. So it can be helpful to run it again. Successful DISM fixes have been had without the second run, and the second run does take time, but the second is necessary for full completion.

Then we unmount and commit the changes:

DISM /Unmount-Wim /MountDir:Mounted-Wim /Commit

And it’s done! That WIM is ready to be used to de-corrupt a Server 2016 machine. We just have to get that file (6.5 gigabytes just now) onto a folder on the server or a network share, make that folder or share the current directory, and run:

DISM /Online /Cleanup-Image /RestoreHealth /Source:wim:Install.wim:2 /limitaccess

That’s index 2, index 4 may be needed if servers have roles only found in Datacenter. And after the DISM, some machines will need SFC /SCANNOW, because although DISM will bring in the needed items, it will not always put them everywhere they are needed.

If power is lost during the WIM procedures, you will probably have to run a cleanup and start over again. Check thus:

dism /get-MountedWiminfo

and cleanup thus:

dism /cleanup-wim

Reportedly, you may need to run those more than once. You may also have to delete subkeys in the registry here:

\HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\WIMMount\Mounted Images

and reboot, and then delete and recreate the folder Mounted-WIM.

At least in version 2.7.2 and close, SIP-ALG is apparently (seen via live logs and behavior…) active within a Watchguard, if you have a TCP-UDP proxy active. Doesn’t matter how much you turn off in that proxy, SIP-ALG is still active. The only way to handle it, apparently, is by disabling the policy altogether and replacing it with a TCP-UDP packet filter.

Try this for Windows 10 etc., from the excellent Brigg Bush:

https://www.heidoc.net/joomla/technology-science/microsoft/67-microsoft-windows-and-office-iso-download-tool

For Server 2022 through 2012R2 (at this writing), and Windows Enterprise 11 and 10, we have this:

www.microsoft.com/en-us/evalcenter

See the menu at the upper left corner to choose.

It’s by registry entry. Create these as DWORDs and set them to value 1:

HKCU\SOFTWARE\Microsoft\Office\15.0\Common\Identity\EnableADAL
HKCU\SOFTWARE\Microsoft\Office\15.0\Common\Identity\Version
HKCU\SOFTWARE\Microsoft\Exchange\AlwaysUseMSOAuthForAutoDiscover

See:

www.winhelponline.com/blog/slipstream-windows-10-integrate-updates-setup-media-iso/

Don’t know yet if it works with Server, or even how well it works yet, but this looks very interesting, most especially for DISMing an existing machine.

So we have the situation where we have multiple network interfaces, possibly including wireless, and we want to set priority, so if one is connected, that one will be used. Here’s a good working procedure, all in Powershell.

1. Get list of interfaces with the Windows index number for each:

Get-NetIPInterface | ft ifINdex,InterfaceAlias,AddressFamily

Now we have a list of interfaces and names. Each interface device may have two listings, one for IPv6 and one for IPv4. What we want is the index numbers for the two. On one machine, “Ethernet” was index 12, and “Wifi” was 18, but there will be wide variation.

2. Priority is higher, for lower numbers. So if we want to set high priority for wired Ethernet when it’s present, we could set priority 10:

Set-NetIPInterface -InterfaceIndex "12" -InterfaceMetric "10"

3. and to make it stick and work predictably, we set Wifi to priority 100:

Set-NetIPInterface -InterfaceIndex "18" -InterfaceMetric "100"

Recently a number of things have come to light together, the combination of which is remarkable.

1. The System Volume Information folder, on very many slow Windows machines, has a lot of junk files within, files that serve no purpose. These are called “orphan shadows”.
2. Orphan shadows occur when the RAID or other drive capability of a Windows machine, cannot keep up with demands on the Volume Shadow Services (VSS) subsystem of Windows, and/or has to abandon an effort in midstream. VSS is used by a huge variety of Windows and application operations.
3. The best-understood examples of events causing orphan shadows, are hard poweroffs in the middle of many different operations large and small. Another is when block-level backups fail.
4. Other examples are simultaneous heavy demands which are too high for the RAID or drive capability. For instance, if demands on RAID are already fairly high, a large database request happening at the same moment as the incremental for a backup will cause one or both to fail, and will create orphan shadow(s) for one or both.
5. As orphan shadows build up, they take more and more space in System Volume Information, and when SVI has more than 20 gigabytes or so of these things, all VSS operations get slower and slower, presumably because the VSS system has to choose to bypass all of the orphans over and over again.
6. SVI has been seen to rise to hundreds of gigabytes, sometimes taking hours to clear.

The short-term step, to get things working, is to clear SVI. On a client OS, this is done thus:

vssadmin delete shadows /all

On a server OS, we do the above, and then we also run diskshadow, and within that little environment, we do:

delete shadows all

Alternatively, we can replace all of the above steps with:

wmic shadowcopy delete /nointeractive

One situation has been seen thus far, where the above two do not do the job, where SVI remains huge. No standard solution has been identified for this as of yet. Up until now, the symptoms have been clear and obvious, but causality a lot more mysterious, and we have often alleviated the situation via removal of OEMware, BIOS, firmware, and driver updates, and use of tweaking tools. On a server, one can remove Windows Defender:

Uninstall-WindowsFeature Windows-Defender

and this helps a lot (solved one big problem so far) on server, but not on a client OS, Microsoft doesn’t allow it. Also, methods to disable bits of Windows Defender on client OS have been disallowed increasingly, as build upgrades have been given.

One step which has helped is to CHKDSK /F the volumes and reboot, and then try the standard cleanout steps again.

Just today, some VSS tweaks have manifest:

https://docs.microsoft.com/en-us/windows/win32/backup/registry-keys-for-backup-and-restore?redirectedfrom=MSDN#maxshadowcopies

specifically this:

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\VSS\Settings
DWORD MaxShadowCopies

and this:

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\VolSnap
DWORD MinDiffAreaFileSize

but it will be a while of testing before any confidence is available as to how to use these to help. Thus far, MaxShadowCopies of 8 and MinDiffAreaFileSize of 128 seems to be helping.

This appears to be rather good.

http://www.softwareok.com/?seite=Microsoft/StressMyPC

Just grab the latest code here:

https://github.com/PKISharp/win-acme/releases/latest

unpack it into a folder you will keep (I used “E:\Let’s Encrypt”), and run letsencrypt.exe in the folder. The multi-site (SAN) mode works only if IIS is set up in certain ways; but the manual mode for a single site is simple and easy, and it sets up a scheduled task in Windows for the needed automatic updates.

Also recently discovered this, have not tested yet but it can come by Chocolatey, which means automatic updates are easy:

https://certifytheweb.com/

A number of friends and I have tried several DISM variations using non-Windows-Update (non-default) sources. Here are known successes. /LimitAccess means don’t use Windows Update.

The first one presumes that a different machine running the same OS is available on the network with C$ being the C: drive share. Reportedly, that share can be simply all of the files from a non-running install.

DISM /online /cleanup-image /restorehealth /source:\\ANOTHER_SERVER\C$\Windows /LimitAccess

The second one uses an OS install ISO; right-click it and choose Mount. G: represents the drive letter given:

DISM /Online /Cleanup-Image /RestoreHealth /Source:wim:G:\Sources\Install.wim:2 /limitaccess

Many ISOs have more than one image within the .WIM file. To get the list of images (and numbers to put after the colon), run this:

DISM /Get-WIMinfo /wimfile:G:\sources\install.wim

You’ll notice, in the /Source:wim line above, that #2 is present, not #1. This is because, in a very common image recently used, it is image #2 which has most of the components, not #1.

After a DISM /RestoreHealth is successful, it appears to be best to run SFC /SCANNOW. To some extent, DISM appears to get the good stuff in, and SFC appears to put it where it needs to go.