Unable to Install Windows Updates or Windows Components

The below post initially stemmed from an Exchange Risk Assessment which noted that all of the customer's Exchange servers were missing critical Windows updates. The customer's security team were slightly upset as they believed (mistakenly) that all assets were properly updated. This was not the case. When we looked into the details, the monthly Windows Server 2012 R2 updates had been failing to install for the last 18 months....

Note that Windows 7 and Windows Server 2008 R2 changed their update methodology in October 2016 to mirror the update distribution behaviour in newer versions of Windows.

The Windows Component Store was introduced in Windows XP to support side by side assemblies and enhanced in Vista track all the OS components. For those who worked on NT or Windows 2000/2003 servers there are always some funny stories around managing the OS over time. For example if you wanted to install an OS feature on a Windows Server 2003 SP2 machine, potentially you needed to provide RTM, SP1 and SP2 source files. This was due to how the installation .inf files were chained as the OS was updated over time with service packs and hotfixes. Windows knew it needed particular files from a particular source and would prompt as needed. Well, not always. Sometimes that went a bit wrong.

NT had its own quirk where there was no chaining. After installing an OS component you had to reapply the service pack. The act of re-applying the service pack would over write the older file from the CD with the new file from the service pack. Yes, that really was a thing.

After having to deal with those issues for 15 years, CBS promised much goodness. Typically though, two things happened to sour an admin on CBS. Firstly they saw the perceived size of the C:\Windows\WinSxS folder, and secondly they experienced CBS corruption.

In may cases, the first indication of a CBS issue was Windows Update failing to install an update. Trying to install the update manually after downloading from the Windows Update Catalog, or using Windows Update would always fail. There are many Windows Update failure codes and each has their own root cause.

Note that you can directly search for a given update in the Windows Update Catalog, add to basket and then download it. This can be very useful when trying to correct the issue – more on that trick later.

Symptoms

As mentioned earlier, there are a range of error codes reported by Windows Update when it cannot service the OS. In addition to CBS preventing updates from installing, I’ve also seen other "interesting" issues such as:

Exchange DAG fails to install as the underlying OS cluster components can not install
SCOM agent crashing on NTdll.dll exception
Exchange IIS app pools crashing on process exceptions
Out of memory and assorted performance issues

The classic WindowsUpdate.log or the new Get-WindowsUpdateLog cmdlet will log the failure from a Windows Update perspective. However if the underlying issue is due to CBS then we need to look at logs which are focused on that area. The best examples of such logs are the DISM and CBS logs which can be found in the respective locations below:

C:\Windows\Logs\DISM\DISM.log
C:\Windows\Logs\CBS\CBS.log

Typically the DISM log will state that there is a CBS issue, and we need to look into the CBS.log to identify the particular package which has the issue.

The below example shows Windows Update failing with error code 0x800f0831.

2018-01-28 13:14:06, Info CBS Store corruption, manifest missing for package: Package_56_for_KB3081320~31bf3856ad364e35~amd64~~6.3.1.1

2018-01-28 13:14:06, Error CBS Failed to resolve package 'Package_56_for_KB3081320~31bf3856ad364e35~amd64~~6.3.1.1' [HRESULT = 0x800f0831 - CBS_E_STORE_CORRUPTION]

2018-01-28 13:14:06, Info CBS Mark store corruption flag because of package: Package_56_for_KB3081320~31bf3856ad364e35~amd64~~6.3.1.1. [HRESULT = 0x800f0831 - CBS_E_STORE_CORRUPTION]

2018-01-28 13:14:06, Info CBS Failed to resolve package [HRESULT = 0x800f0831 - CBS_E_STORE_CORRUPTION]

2018-01-28 13:14:06, Info CBS Failed to get next package to re-evaluate [HRESULT = 0x800f0831 - CBS_E_STORE_CORRUPTION]

2018-01-28 13:14:06, Info CBS Failed to process component watch list. [HRESULT = 0x800f0831 - CBS_E_STORE_CORRUPTION]

Remediation Tools

Depending upon the exact issue, you may need to run one or more of these tools:

CHKDSK
DISM
SFC

If you work for Contoso, the simple commands listed in the first section will suffice. In reality though, things can get quite squirrely and we need to look at more advanced and/or different options.

That is covered in the latter section of this post.

CHKDSK

CHKDSK.exe can check and correct a volume’s file system. This may be required if the NTFS filesystem is suspect after a storage "event".

The CHKDSK tool is documented here.

Below is an example for scanning a local machine’s C:\ drive.

CHKDSK.exe C: /R /F /B

This will require a reboot and considerable time to then perform the scan. There are other options to perform SpotFix and online scans in newer versions of Windows, though that may not help that much for the OS drive depending on the issue.

After the system reboots look for an EventID 1001 in the Application Event Log. The event source will be Wininit and should be an information level event.

DISM Scan

Note that Deployment Image Servicing and Management (DISM) can do multiple tasks on the Windows OS. Here we will focus on the scanning and repairing of CBS issues. DISM is not available for Windows Server 2008 R2 for example and the System Update Readiness (SUR) tool should be used.

There are a some different options that can be used:

DISM.exe /CheckHealth
DISM.exe /ScanHealth
DISM.exe /RestoreHealth

Use /CheckHealth to check whether the image has been flagged as corrupted by a failed process and whether the corruption can be repaired.
Use /ScanHealth to scan the image for component store corruption.
Use /RestoreHealth to scan the image for component store corruption, and then perform repair operations automatically.

Use /Source with /RestoreHealth to specify the location of known good versions of files that can be used for the repair. For more information on specifying a source location, see https://go.microsoft.com/fwlink/?LinkId=243077.
Use /LimitAccess to prevent DISM from contacting WU/WSUS

Note that the /Online specifies we will work with the current Windows installation rather than correct an offline image using the /Image switch.

Example: DISM.exe /Image:C:\offline /Cleanup-Image /RestoreHealth /Source:C:\Test\Mount

Most of the online articles will specify to start with this command to scan and repair CBS:

DISM.exe /Online /Cleanup-Image /RestoreHealth

We will come back to DISM later, with some more advanced options.

SFC

Once DISM has done its work, time to run SFC.

SFC.exe is the System File Checker utility which will scan and verify the integrity of protected system files. Should an incorrect version be found, it can be replaced.

The SFC tool is documented here.

Example syntax to scan and repair the local machine would be:

SFC.exe /SCANNOW

Typically this is run AFTER DISM reports no issues.

IF Squirrel Factor > 10

If the standard DISM commands do not correct the issue, then its time to break out the rough stuff. In many cases a DISM /RestoreHealth ends in an error stating that the source files could not be found. By default DISM will typically try to contact Windows Update to download the required components. If the default location for Features on Demand is specified then that location is used.

The error text is included for accessibility.

C:\>Dism /Online /Cleanup-Image /RestoreHealth

Deployment Image Servicing and Management tool
Version: 10.0.14393.3750

Image Version: 10.0.14393.3241

[==========================100.0%==========================]

Error: 0x800f081f

The source files could not be found.
Use the "Source" option to specify the location of the files that are required to restore the feature. For more information on specifying a source location, see http://go.microsoft.com/fwlink/?LinkId=243077.

The DISM log file can be found at C:\Windows\Logs\DISM\dism.log

Note: There are some known issues with the PowerShell components so you may want to review this article.

So we need to point DISM at a location where the relevant files are available. However in order to determine a suitable location, we need to identify which component is causing the issue.

Is it a base part of the OS or an update that was installed? If it is a base OS component then we could use the install WIM located on the install media. If the required component is an update that was downloaded and installed, i.e. not part of base DVD image, then we need to access a WIM that has that component installed. Alternatively a separate server which has the update in question installed can be used.

How to determine which component is causing the issue?

If we open up the the C:\Windows\Logs\DISM\DISM.log in Notepad, we can see the error 0x800f081f listed there:

2020-08-06 13:35:52, Info DISM DISM Package Manager: PID=6472 TID=6784 Error in operation: source for package or file not found, ResolveSource() unsuccessful. (CBS HRESULT=0x800f081f) - CCbsConUIHandler::Error
2020-08-06 13:35:52, Error DISM DISM Package Manager: PID=6472 TID=13972 Failed finalizing changes. - CDISMPackageManager::Internal_Finalize(hr:0x800f081f)

However, what component is actually failing? let’s look in the CBS.log for those details. In the CBS.log we see:

p) CSI Payload Corrupt amd64_windows-defender-management-powershell_31bf3856ad364e35_10.0.14393.0_none_1137c685b804c9a3\MSFT_MpThreatCatalog.cdxml
Repair failed: Corrupted replacement payload.

In the example above it is an OS PowerShell component which is experiencing an issue. It it were an update, then you would see the relevant KB number listed.

To fix this we need to specify where DISM can get the required files. That may prove to be tricky as not all of the syntax for all options are fully laid out.

We have a few different options to get the required files:

Contact Windows Update
A different Windows install which contains the required files
Mount a WIM image file to the local file system.
Access a WIM image file directly

Chance are that the WU option already failed, else you would not be reading this far down the page. If we want to look at a WIM file to list the images inside, to ensure we have the right source the below command will help:

DISM.exe /Get-ImageInfo /ImageFile:Install.wim

We will use the second image as it is a match. We specify that now in the DISM command – note the :2 syntax after the WIM file name.

DISM /Online /Cleanup-Image /RestoreHealth /Source:wim:E:\Sources\Install.wim:2 /LimitAccess

In the CBS.log file

Checking System Update Readiness.

(p)    CSI Payload Corrupt    (Fixed)    amd64_windows-defender-management-powershell_31bf3856ad364e35_10.0.14393.0_none_1137c685b804c9a3\MSFT_MpThreatCatalog.cdxml
(p)    CSI Payload Corrupt    (Fixed)    amd64_windows-defender-management-powershell_31bf3856ad364e35_10.0.14393.0_none_1137c685b804c9a3\Defender.psd1
(p)    CSI Payload Corrupt    (Fixed)    amd64_windows-defender-management-powershell_31bf3856ad364e35_10.0.14393.0_none_1137c685b804c9a3\MSFT_MpWDOScan.cdxml
(p)    CSI Payload Corrupt    (Fixed)    amd64_windows-defender-management-powershell_31bf3856ad364e35_10.0.14393.0_none_1137c685b804c9a3\MSFT_MpSignature.cdxml
(p)    CSI Payload Corrupt    (Fixed)    amd64_windows-defender-management-powershell_31bf3856ad364e35_10.0.14393.0_none_1137c685b804c9a3\MSFT_MpThreat.cdxml
(p)    CSI Payload Corrupt    (Fixed)    amd64_windows-defender-management-powershell_31bf3856ad364e35_10.0.14393.0_none_1137c685b804c9a3\MSFT_MpComputerStatus.cdxml
(p)    CSI Payload Corrupt    (Fixed)    amd64_windows-defender-management-powershell_31bf3856ad364e35_10.0.14393.0_none_1137c685b804c9a3\MSFT_MpScan.cdxml
(p)    CSI Payload Corrupt    (Fixed)    amd64_windows-defender-management-powershell_31bf3856ad364e35_10.0.14393.0_none_1137c685b804c9a3\MSFT_MpThreatDetection.cdxml
(p)    CSI Payload Corrupt    (Fixed)    amd64_windows-defender-management-powershell_31bf3856ad364e35_10.0.14393.0_none_1137c685b804c9a3\MSFT_MpPreference.cdxml

Summary:
Operation: Detect and Repair
Operation result: 0x0

Running another /RestoreHealth scan shows that CBS is now healthy.

Then we run SFC to validate system files. Note that corrupt files were found and replaced.

For reference, below are the entries from the CBS log showing the activity.

2020-08-06 21:42:14, Info                  CSI    00007d39 Hashes for file member \??\C:\Windows\System32\WindowsPowerShell\v1.0\Modules\Defender\MSFT_MpPreference.cdxml do not match actual file [l:23]'MSFT_MpPreference.cdxml' :
Found: {l:32 3VV5hrUQdPPkt9XAkjkDvxC6kZoSNWBfXLqKSbBe8H0=} Expected: {l:32 zbiTyZoXoGxGmd+AOLbOJqmYHoTwkMkcKkoXw+Uz/ng=}
2020-08-06 21:42:14, Info                  CSI    00007d3a [SR] Repairing corrupted file \??\C:\Windows\System32\WindowsPowerShell\v1.0\Modules\Defender\MSFT_MpPreference.cdxml from store
2020-08-06 21:42:14, Info                  CSI    00007d3b@2020/8/7:01:42:14.130 CSIPERF - FileMapsCreated 113ms
2020-08-06 21:42:14, Info                  CSI    00007d3c CSIPERF - FilePI Commit 116ms
2020-08-06 21:42:14, Info                  CSI    00007d3d@2020/8/7:01:42:14.146 Primitive installers committed for repair
2020-08-06 21:42:14, Info                  CSI    00007d3e [SR] Repair complete
2020-08-06 21:42:14, Info                  CSI    00007d3f [SR] Committing transaction
2020-08-06 21:42:14, Info                  CSI    00007d40 Creating NT transaction (seq 1), objectname '(null)'
2020-08-06 21:42:14, Info                  CSI    00007d41 Created NT transaction (seq 1) result 0x00000000, handle @0x3fc
2020-08-06 21:42:14, Info                  CSI    00007d42@2020/8/7:01:42:14.257 Beginning NT transaction commit...
2020-08-06 21:42:14, Info                  CSI    00007d43@2020/8/7:01:42:14.278 CSI perf trace:
CSIPERF:TXCOMMIT;21310
2020-08-06 21:42:14, Info                  CSI    00007d44 [SR] Verify and Repair Transaction completed. All files and registry keys listed in this transaction have been successfully repaired

DISM - Remote Server

If another machine has the required components, then we can point to that. The below uses a server called Exch-2016 and hits the admin $ share to obtain the files from that other machine's WinSXS folder.

DISM /Online /Cleanup-Image /RestoreHealth /Source:\\Exch-2016\c$\Windows\winsxs /LimitAccess

Note - ensure that you have access to the required resource from the cmd prompt where you are running this. If in doubt, map a drive to provide the credentials and use the drive mapping in your command.
For example, map drive M:\ to \\Exch-2016\c$

DISM /Online /Cleanup-Image /RestoreHealth /Source:M:\Windows\winsxs /LimitAccess

DISM – Inject Package

This process is outlined in KB 4477073.

Using the steps mentioned about, download the update from the Windows Update Catalog and then use DISM to inject the package into the running OS.

Once the package has been installed, then re-run /RestoreHealth command.

That sounds great but you may have to expand the downloaded MSU file and then use DISM to inject the relevant CAB file. This could be summarised as:

Download the required update from Windows Update Catalog
Expand the downloaded file to a folder. For example:
- Expand.exe C:\Temp\FileWeDownloadedNameGoesHere.msu /F:* C:\Temp\Extracted
Use DISM to inject the extracted CAB
- DISM /Online /Add-Package /PackagePath:C:\Temp\Extracted\SomeFileName.cab
DISM /Online /Cleanup-Image /RestoreHealth

Note that the destination directory needs to be created before running the expand.exe command.

Cheers,

Rhoderick