Windows Update wants to install over and over

[Mayo, Bill]

I have a Windows Server 2022 box that keeps indicating that it needs KB5030216. The update has been installed multiple times, both through Windows Update and manually. Update history shows that it installed successfully, but it then shows available again. In the event log, “WindowsUpdateClient” shows that it successfully installed.

 

After doing this a few rounds, I did a DISM /StartComponentCleanup which didn’t help. I also did a DISM /ScanHealth to see if there were any problems and it reported none.

 

At some point after doing the above, it started showing the update as needed, but that it only needed a reboot. You guessed it, doing a reboot doesn’t actually help. I can’t find any errors in the event log that seem relevant. I do note that the update, while showing successfully installed in history, does not show as installed update that can be removed.

 

This server is part of a cluster, which may be relevant, but the other node doesn’t have this problem.

 

I am stumped as to what to do next? Any suggestions?

 

Bill Mayo

[Robert ECEO Townley]

Oh, not only do I feel your pain, but was seriously hoping 2022 would have done away with this - it is a major system integrity and security issue.  

Since there is clearly a problem with the components of windows update, you cannot trust that it correctly compares your files to certified files up on MS servers.   

One MUST have a LOCAL image of win2022 files - a recent ISO like install media  on the machine or (less preferred) a copy from `\\clusterPartner\c$\windows\winsxs\`

So repeat your dism.exe commands with local sources and /limitaccess.  

Much better is repeat from  a known good and trusted win2022 bootable media:

mountinfo.exe (some Mount executable list the local drives instead of writing a batch for loop).   

Look through each drive letter until you find the correct:\users\  folders, lets say drive `A:\` - pass drive letter A: to /image: or is it /offline:a:  

dism.exe /image:a:\  /checkHealth /limitaccess and /source:goldenImageOfWindows2022/windows/ MUST be used.   

chkntfs

sfc.exe /scanNow

Boot into safe mode at least once and logon.   Some people say that fixes it, but did not for me.   

Search the Component Based Servicing CBS LogFiles, somewhere in a log[s | files ] folder and and CBS.

Grep the c:\windows\log*\cbs 

c:\windows\system32\log*\cbs 

Could boot partition be read only?

CheckSUR - System Update Readiness Tool.

Since you seem comfortable dism, use more of its features:

dism.exe /image:a: /check-health /restoreHealth

Warning: Typing this from memory on my tiny iPhone, so I am sure my thumbs made mistakes.

--
You received this message because you are subscribed to the Google Groups "ntsysadmin" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ntsysadmin+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ntsysadmin/a7b6b845b8cb469da68048050a60b186%40pittcountync.gov.

[Brian Illner]

Sometimes in those CBS logs you will discover that an earlier patch actually failed to completely install, which is now causing issues for the current patch.

 

If that is the case, reinstalling that earlier one could resolve the issue.

 

I’ve had to do that a few times in the past.

 

BRIAN ILLNER​​​​  |  Senior Systems Administrator 864.250.9227 Office 864.679.2537 Fax Canal Insurance Company 101 N. Main Street, Suite 400 Greenville, SC 29601 WARNING:  As the information in this transmittal (including attachments, if any) may contain confidential, proprietary, or business trade secret information, it should only be reviewed by those who are the intended recipients.  Unless you are an intended recipient, any review, use, disclosure, distribution or copying of this transmittal (or any attachments) is strictly prohibited.   If you have received this transmittal in error, please notify me immediately by reply email and destroy all copies of the transmittal.  While Canal believes this transmittal to be free of virus or other defect, it is the responsibility of the recipient to ensure that it is virus free and no responsibility is accepted by Canal (or its subsidiaries and affiliates) for any loss or damage arising therefrom.

 

From: ntsys...@googlegroups.com <ntsys...@googlegroups.com> On Behalf Of Robert ECEO Townley
Sent: Monday, October 16, 2023 5:26 PM
To: ntsys...@googlegroups.com
Subject: Re: [ntsysadmin] Windows Update wants to install over and over

 

CAUTION: This message was sent from outside of Canal Insurance. Please do not click links or open attachments unless you recognize the source of this email and know the content is safe. Please report all suspicious emails to "inf...@canal-ins.com" as an attachment.

---

To view this discussion on the web visit https://groups.google.com/d/msgid/ntsysadmin/CACE8FwkVgs%3Dg4SuKyOL6ng%2B9gNGvXMVpQsDgMFhd4-9r7xmYhQ%40mail.gmail.com.

[Mayo, Bill]

Thank you both for the info. I am tied up most of the day, but will dig into this deeper later.

 

From: ntsys...@googlegroups.com <ntsys...@googlegroups.com> On Behalf Of Brian Illner
Sent: Monday, October 16, 2023 6:29 PM
To: ntsys...@googlegroups.com
Subject: RE: [ntsysadmin] Windows Update wants to install over and over

 

EXTERNAL EMAIL: This email originated from outside of Pitt County Government. Do not click any links or open any attachments unless you trust the sender and know the content is safe.

To view this discussion on the web visit https://groups.google.com/d/msgid/ntsysadmin/BN6PR13MB2913974B7B17FA011118446EC5D7A%40BN6PR13MB2913.namprd13.prod.outlook.com.

[Mayo, Bill]

I snuck in some time to work on this. I did the restorehealth pointing to 2022 disc (DISM /Online /Cleanup-Image /RestoreHealth /Source:wim:D:\sources\install.wim:2 /limitaccess), which succeeded. I also note that scanhealth was clean before doing that. I then rebooted and noted that both the Windows and DotNet updates showed as available again. Applied the updates, rebooted, and back to where I started.

 

I note that there are a number of errors in the CBS logs, but I am struggling to figure out how to address. I also note that “Installed Updates” doesn’t show any Microsoft Windows updates being installed since 6/13/2023, and that was Servicing Stack 10.0.20348.1720. I would note that we have applied updates monthly since the server was stood up and there were previously no indications of failed updates.

 

In the aforementioned CBS logs, I note errors related to the files in the SoftwareDistribution\Download folder. I cleared those folders out and restarted Windows Updates before applying, so I am not sure what to make of that. As related to the comment by Brian, I am leaning towards an earlier update not having applied successfully, and am thinking it is the first thing listed, which is a servicing stack update. However, it only provides a filename and I am struggling to correlate it with an actual update. I am posting an extract below in case someone with more experience can infer a next action better than I seem to be able to.

 

2023-10-17 09:06:21, Info                  CBS    Failed to clear CorruptionDetectedDuringAcr store corrupt flag (slow mode trigger). [HRESULT = 0x80070002 - ERROR_FILE_NOT_FOUND]

2023-10-17 09:40:20, Info                  CBS    Failed to clear CorruptionDetectedDuringAcr store corrupt flag (slow mode trigger). [HRESULT = 0x80070002 - ERROR_FILE_NOT_FOUND]

2023-10-17 10:21:25, Info                  DPX    CCabStorage::GetErrorCode(): m_hrFirstError = 0x80070012

2023-10-17 10:21:25, Info                  DPX    CCabStorage::GetErrorCode(): m_hrFirstError = 0x80004004

2023-10-17 10:21:25, Info                  CBS    Failed to extract file TOC.xml from cabinet \\?\C:\Windows\SoftwareDistribution\Download\9b812b246eb28c1864ee84fd9ea7034c\inst\SSU-20348.1960-x64.cab [HRESULT = 0x80070002 - ERROR_FILE_NOT_FOUND]

2023-10-17 10:21:25, Info                  DPX    CCabStorage::GetErrorCode(): m_hrFirstError = 0x80004004

2023-10-17 10:21:25, Info                  DPX    CCabStorage::GetErrorCode(): m_hrFirstError = 0x80070012

2023-10-17 10:21:25, Info                  CBS    Not able to add file to extract: update.ses [HRESULT = 0x80070002 - ERROR_FILE_NOT_FOUND]

2023-10-17 10:21:26, Info                  CBS    Failed to extract file TOC.xml from cabinet \\?\C:\Windows\SoftwareDistribution\Download\9b812b246eb28c1864ee84fd9ea7034c\inst\Windows10.0-KB5030216-x64.cab [HRESULT = 0x80070002 - ERROR_FILE_NOT_FOUND]

2023-10-17 10:21:34, Info                  CBS    Failed to extract file TOC.xml from cabinet \\?\C:\Windows\SoftwareDistribution\Download\47763c217414bba1998ffa88fdbee2cd\Windows10.0-KB5029928-x64-NDP48.cab [HRESULT = 0x80070002 - ERROR_FILE_NOT_FOUND]

 

As a final note, I have not started in safe mode as was suggested by Robert yet.

 

From: ntsys...@googlegroups.com <ntsys...@googlegroups.com> On Behalf Of Mayo, Bill
Sent: Tuesday, October 17, 2023 8:29 AM
To: ntsys...@googlegroups.com
Subject: RE: [ntsysadmin] Windows Update wants to install over and over

 

Thank you both for the info. I am tied up most of the day, but will dig into this deeper later.

 

From: ntsys...@googlegroups.com <ntsys...@googlegroups.com> On Behalf Of Brian Illner
Sent: Monday, October 16, 2023 6:29 PM
To: ntsys...@googlegroups.com
Subject: RE: [ntsysadmin] Windows Update wants to install over and over

 

 

Sometimes in those CBS logs you will discover that an earlier patch actually failed to completely install, which is now causing issues for the current patch.

To view this discussion on the web visit https://groups.google.com/d/msgid/ntsysadmin/eed52df7a2234346be017a68ed8e0536%40pittcountync.gov.

[Mayo, Bill]

If anyone has any other suggestions regarding the below, it would be much appreciated. I still haven’t been able to make any progress. I did start up in safe mode once with no change. The errors sure sound like some kind of disk or catalog corruption, but DISM and SFC come back clean.

 

From: ntsys...@googlegroups.com <ntsys...@googlegroups.com> On Behalf Of Mayo, Bill
Sent: Tuesday, October 17, 2023 10:51 AM
To: ntsys...@googlegroups.com
Subject: RE: [ntsysadmin] Windows Update wants to install over and over

 

EXTERNAL EMAIL: This email originated from outside of Pitt County Government. Do not click any links or open any attachments unless you trust the sender and know the content is safe.

To view this discussion on the web visit https://groups.google.com/d/msgid/ntsysadmin/0013d08e131745998c4b335b2e95f714%40pittcountync.gov.

[Jim Behning]

I had two or three older 2016 servers that bombed on updates 2 months in a row. Hour or more reboots to uninstall failed patches. The latest updates installed without issue. Maybe fingers crossed will work for you next month. 😉

---

From: ntsys...@googlegroups.com <ntsys...@googlegroups.com> on behalf of Mayo, Bill <Bill...@pittcountync.gov>
Sent: Tuesday, October 24, 2023 8:42 AM
To: ntsys...@googlegroups.com <ntsys...@googlegroups.com>

To view this discussion on the web visit https://groups.google.com/d/msgid/ntsysadmin/b2c341c0190c455c8632f9f97a8bf8df%40pittcountync.gov.

[Brian Illner]

For Windows Server 2022 – that SSU appears to be bundled with the September 2023 OS update.

 

September 12, 2023—KB5030216 (OS Build 20348.1970) - Microsoft Support

 

However, since the last update installed shows June 2023, personally I would start with reinstalling those, and then trying a more recent update after checking the logs again to make sure there wasn’t an issue.

 

That seems similar to what we saw. Even though Windows showed that last update as installed, it actually had not installed correctly and prevented the newer CUs from installing afterwards.

 

BRIAN ILLNER​​​​  |  Senior Systems Administrator 864.250.9227 Office 864.679.2537 Fax Canal Insurance Company 101 N. Main Street, Suite 400 Greenville, SC 29601 WARNING:  As the information in this transmittal (including attachments, if any) may contain confidential, proprietary, or business trade secret information, it should only be reviewed by those who are the intended recipients.  Unless you are an intended recipient, any review, use, disclosure, distribution or copying of this transmittal (or any attachments) is strictly prohibited.   If you have received this transmittal in error, please notify me immediately by reply email and destroy all copies of the transmittal.  While Canal believes this transmittal to be free of virus or other defect, it is the responsibility of the recipient to ensure that it is virus free and no responsibility is accepted by Canal (or its subsidiaries and affiliates) for any loss or damage arising therefrom.

 

To view this discussion on the web visit https://groups.google.com/d/msgid/ntsysadmin/LV8PR14MB7536EC309480DF5044C3BE29AFDEA%40LV8PR14MB7536.namprd14.prod.outlook.com.

[Robert ECEO Townley]

Have you watched the machine shutdown and completely restart to install updates?   The old `uptime.exe -s` at an elevated prompt will help quickly find out past restart behaviour.   My event logs will show the shutdown and startup and then another and then during the first startup, it will restart to a few minutes later to uninstall and sometimes restart again.  Depending on speed of machine, `uptime.exe -s `  measured in tens of minutes or less might be indicative of could not install the updates, so reverting behaviour.

It might say something like   “Unable to complete the updates …. reverting applied updates “.   Otherwise, everything looks completely normal.  

Since all past updates are offered every month now, have to wonder how many admins believe they are fully patched but are not at all.  winUpdate is probably the most important piece of sw on a system but has a history of providing false integrity information.

There are some registry key permissions that some have had luck with ( I didn’t).   https://answers.microsoft.com/en-us/windows/forum/all/endless-loop-from-update/7ad33df4-4381-4d81-a710-ef75025fc4c4

Develop a script that reads all the ACL entries (including hidden files) on  both  cluster members root drive and c:\windows\ and c:\programdata\.   Run those through a diff program to look for differences.  

[workingClusterSrv] PS> get-acl c:\* | select-object -property * | fl

Lastly, I failed to mention deleting the softwareDistribution folder.   Important to stop many different services before deleting that folder.  I have PS scripts that try to gather all the recommended actions into but

Google for inplace upgrade from windows server 2022 to windows server 2022.  If that does not return anything, repeat with 2019.  This method is supposed to keep all your files.  Different directions for Windows on Azure.

If you can at all, i highly recommend wipe and reinstall.

To view this discussion on the web visit https://groups.google.com/d/msgid/ntsysadmin/0013d08e131745998c4b335b2e95f714%40pittcountync.gov.

[Mayo, Bill]

Thanks for the response. If I try to reinstall the updates that show installed, Windows just tells me that the update is already installed (or maybe it was the update doesn’t apply). I did manage to install the June update successfully, but just temporarily. It showed as installed in the list of installed updates yesterday, but today it no longer shows as installed. After installing that, I did try the next month and it went through the motions, but didn’t show as installed. I’m not sure when the June update disappeared, but I have done a bunch over the last couple of days working on it.

 

From: ntsys...@googlegroups.com <ntsys...@googlegroups.com> On Behalf Of Brian Illner
Sent: Wednesday, October 25, 2023 1:34 PM
To: ntsys...@googlegroups.com
Subject: RE: [ntsysadmin] Windows Update wants to install over and over

 

EXTERNAL EMAIL: This email originated from outside of Pitt County Government. Do not click any links or open any attachments unless you trust the sender and know the content is safe.

To view this discussion on the web visit https://groups.google.com/d/msgid/ntsysadmin/BN6PR13MB29132BE70F5068CFE017B2B9C5DEA%40BN6PR13MB2913.namprd13.prod.outlook.com.

[Mayo, Bill]

The reboot happens quickly enough that I know that the second part isn’t working. It is definitely restarting, but there is no progress shown for installing updates at that point. This is not something I originally caught, at least partly because it is happening at the same time as a bunch of other servers. IIRC from when I first started investigating, the event log was showing that the update completed. The update itself shows as installed in the history (but not uninstall), even though it will then pop back up again saying it is needed.

 

The bit about permissions is interesting and I suspect is at play here. I say that because /AnalyzeComponentStore now says cleanup is needed, but when I run /StartComponentCleanup it quickly exits at 20% and says complete. However, re-running Analyze says cleanup is needed again. This is a new development and a sign that I have made things worse.

 

I am no MBS but I am decent at PowerShell, so I am working on what you suggest. The issue I have is that the text files I generated for comparison are over 800MB and my main method for doing diffs won’t even load the files. I am now trying to do with PS also, but it is taking a long time (unsurprisingly) and I am a little dubious I am going to get useful info from using that method.

 

I have not been able to find any specific articles about an in-place upgrade of Windows to the same version, so if someone can point me to an article like that, I would appreciate it.

 

I do feel like a rebuild is the best scenario, but, as I mentioned, this is part of a cluster and that adds a layer of difficulty for me. We are using VMWare and I had to do some interesting things to get the shared disks working and the main disks are linked to this VM. Throw in that this is clustered in the first place because there is no tolerance for downtime and I am really concerned about blowing the whole thing up.

 

From: ntsys...@googlegroups.com <ntsys...@googlegroups.com> On Behalf Of Robert ECEO Townley
Sent: Wednesday, October 25, 2023 6:16 PM
To: ntsys...@googlegroups.com
Subject: Re: [ntsysadmin] Windows Update wants to install over and over

 

EXTERNAL EMAIL: This email originated from outside of Pitt County Government. Do not click any links or open any attachments unless you trust the sender and know the content is safe.

To view this discussion on the web visit https://groups.google.com/d/msgid/ntsysadmin/CACE8Fwk-SqyDrqK_Y7FhkSaBER3t9UTUQjWuQUqqZbZ-Z%2B3wRQ%40mail.gmail.com.

[Robert ECEO Townley]

I drafted this forever ago and never sent.   There is more to add.   My history with this problem goes back the other nt Sys Admin list and suggestions by   Susan Bradley who runs PatchManagement.

[PatchManagement] Sept Patches continually want to be re-installed Win2012r2

https://groups.google.com/g/patchmanagement/c/F-AgGATxzLk/m/bb4AldgyAgAJ

` msconfig.exe ` and turn off anything unnecessary. 

Try updating as a LOCAL Administrator.

Do this on both cluster machines and check for differences:

`PS> $packages = $(dism.exe /online /get-packages /format:table )  ` 

`PS> $packages >> $ENV:COMPUTERNAME-packages.txt `

`PS> notepad++.exe $ENV:COMPUTERNAME-packages.txt `

`PS> $packages  | select-string -notmatch Installed # See Susan Bradley post to PatchManagement link above.`

`PS> $packages  | select-string -pattern STAGED   #Watch this one very carefully.  `

`PS> $pStagedSoUninstall = $packages  | select-string -pattern STAGED   # Watch for STAGED very carefully.  These should be uninstalled.  `

Here it is important to distinguish between ServicingStackUpdates aka SystemServicingUpdates or "SSU" vs  LatestCumulativeUpdates "LCU".

SSUs contain the latest version of "Windows Update" itself, but are NOT removeable.  But since they are packaged with the LCU, how does one remove the LCU?  It is a little more work and to delete the LCU, but not the SSU installed with it.  Open up the list of packages in notepad and google them is one way.

Remove them in reverse chronological order, newest STAGED package first.  The following remove-package switch will not work for SSUs, only for LCUs. 

`PS> dism.exe /online /remove-package /packagename:Package_for_KBabcXYZ_FULL_NAME `

[Mayo, Bill]

Thank you for this. I ultimately wound up doing an in-place upgrade but will keep this for future reference.

 

From: ntsys...@googlegroups.com <ntsys...@googlegroups.com> On Behalf Of Robert ECEO Townley
Sent: Thursday, February 1, 2024 4:01 PM
To: ntsys...@googlegroups.com
Subject: Re: [ntsysadmin] Windows Update wants to install over and over

 

EXTERNAL EMAIL: This email originated from outside of Pitt County Government. Do not click any links or open any attachments unless you trust the sender and know the content is safe.

To view this discussion on the web visit https://groups.google.com/d/msgid/ntsysadmin/CACE8Fwmd%2B7Q4oz0q4gNbvgw434WcnWGd9%3DvRN68Pqt0c%2BaUWOA%40mail.gmail.com.

[Robert ECEO Townley]

The in-place upgrade actually worked?   Mine did not.

To view this discussion on the web visit https://groups.google.com/d/msgid/ntsysadmin/517d8d47ca0042e384a26b490da0f20f%40pittcountync.gov.

[Mayo, Bill]

It did. I was very concerned about doing it, but it went well.

To view this discussion on the web visit https://groups.google.com/d/msgid/ntsysadmin/CACE8FwmSU85LV36kKybn8Tkwz7xSAPa-Mxsm_21Aa3bOEHqAjg%40mail.gmail.com.