The Microsoft Recovery Tool was updated 7/22/2024 as version 3.1. While fundamentally there are no functional changes to the tool, for the Recover from WinPE option, we have expanded the logging, reattempt logic, and error handling. For the Recover from safe mode for USB delivery, we’ve added in better user awareness cues for when to run the repair command.
7/23/2024: Microsoft notes that CrowdStrike has updated its Remediation and Guidance Hub: Falcon Content Updates for Windows Hosts.
As a follow-up to the CrowdStrike Falcon agent issue impacting Windows clients and servers, Microsoft has released an updated recovery tool with two repair options to help IT admins expedite the repair process. The signed Microsoft Recovery Tool can be found in the Microsoft Download Center: https://go.microsoft.com/fwlink/?linkid=2280386. In this post, we include detailed recovery steps for Windows client, servers, and OS's hosted on Hyper-V. The two repair options are as follows:
Determining which option to use
Recover from WinPE (recommended option)
This option
quickly and directly recovers systems and does not require local admin
privileges. However, you may need to manually enter the BitLocker
recovery key (if BitLocker is used on the device) and then repair
impacted systems. If you use a third-party disk encryption solution,
please refer to vendor guidance to determine options to recover the
drive so that the remediation script can be run from WinPE.
Recover from safe mode
This option may enable recovery on
BitLocker-enabled devices without requiring the entry of BitLocker
recovery keys. For this option, you must have access to an account with
local administrator rights on the device. Use this approach for devices
using TPM-only protectors, devices that are not encrypted, or situations
where the BitLocker recovery key is unknown. However, if utilizing
TPM+PIN BitLocker protectors, the user will either need to enter the PIN
if known, or the BitLocker recovery key must be used. If BitLocker is
not enabled, then the user will only need to sign in with an account
with local administrator rights. If third-party disk encryption
solutions are utilized, please work with those vendors to determine
options to recover the drive so the remediation script can be run.
Additional considerations
Although the USB option is
preferred, some devices may not support USB connections. In such cases,
we provide detailed steps below for using the Preboot Execution
Environment (PXE) option. If the device cannot connect to a PXE network
and USB is not an option, reimaging the device might be a solution.
As with any recovery option, test on multiple devices prior to using it broadly in your environment.
Prerequisites to create the boot media
Instructions to generate the WinPE recovery media
To create recovery media, follow these steps on the 64-bit Windows client mentioned in prerequisite #1:
Prerequisites for using the boot media
The BitLocker recovery key
for each BitLocker-enabled impacted device on which the recover media
is used may be required. If you are using TPM-only protectors and using
the safe boot option, then the recovery key will not be required. If you
are using TPM+PIN protectors, then you may need the recovery key if you
do not know the PIN for the device.
Using Recovery from WinPE media
Using Safe Boot media
To repair an impacted device without using the BitLocker recovery key and if you have access to the local administrator account:
Using recovery media on Hyper-V virtual machines
The
recovery media can be used to remediate impacted Hyper-V virtual
machines. To do so, select the option to generate an ISO when creating
the recovery media using the steps above. For non-Hyper-V virtual
machines, follow instructions provided by your hypervisor vendor to
utilize the recovery media.
Steps to Recover Hyper-V virtual machines
Screenshot for where to add the DVD Drive.
Screenshot of where to add the image file.
Screen shot of the original boot order.
Screenshot of the change to the boot order.
Using PXE for Recovery
For most customers, the
options listed above or following the steps in the KBs linked towards
the end of this post will help restore your devices. However, if devices
are unable to use the option to recover from USB, for example, because
of security policies or port availability, IT admins can use PXE to
remediate.
To use this solution, you can use the Windows Imaging Format
(WIM) that the Microsoft Recovery Tool creates in an existing PXE
environment as long as the impacted devices are on the same subnet as
the PXE server. Alternatively, you can either use the PXE server
approach outline below. This option works best when the PXE server can
be moved subnet to subnet easily for remediation purposes.
Prerequisites for PXE Recovery
Configuring the PXE server
Recover an impacted device
For more information on the issue impacting Windows clients and servers running the CrowdStrike Falcon agent, please see:
Thank you for your continued feedback through this post, support,
and other feedback channels, such as @IntuneSuppTeam on X. Please note
this tool does not use Microsoft Intune, but we're sharing as a Support
tip to help customers. We’ll continue to provide updates to this post as
needed.
Updates to this post
[7/21/2024] - Complete
update of the blog post to describe the two options for recovery now
using the updated signed Microsoft Recovery Tool. Many customers have
used the tool and provided feedback, which we have incorporated, thank
you. The new release includes a new option for recovery using safe boot,
the option to generate ISO or USB, a fix for ADK detection when the
Windows Driver Kit is installed, and a fix for the USB disk size check.
[7/21/2024] - PXE recovery option added.
[7/22/2024] - Signed Microsoft Recovery Tool updated (multiple changes summarized below). Also updated a Windows KB article link.
[7/23/2024] - Minor text updates.
Changes to Recover from WinPE for USB, PXE and ISO delivery:
Screenshot of remediation script to try another drive
Configuration screen share.
Change to Recover from safe mode for USB and ISO delivery
The zip and PS1 names have been updated to:
Script doesn't seem to work well for me. I've not had time to dig into the .ps1. First run it downloaded and installed the ADK components and ended up with these same errors. This screen here is my second run attempt.
Is there a way the batch file could loop through various drives to find where windows and crowdstrike would be installed, even if those drives are encrypted with bitlocker?
@aarony Have you confirmed the ADK is installed to that location and the BAT file is where it is expected to be?
@Rob York Looks like when I also have these installed, there are problems - maybe because they install to the Windows Kits\10 folder also? Anyways - moving on to a cleaner machine to try. I guess my two test machines are too dirty (and many tech's machines might be...)
@Rob York Confirmed a machine where NO kits are installed works. If you have the "Windows Driver Kit" installed, you get a false positive for the ADK in your .ps1 script. It's because the "Windows Driver Kit" installs the DISM components into the '\Assessment and Deployment Kit\Deployment Tools\AMD64' folder, which your .ps1 script is using to detect if you have ADK.
@BruceK825 Sorry, this is not planned.
@Rob York thanks, no worries. I tried to handle it in the code below, in case it helps anyone. For now, we're trying to detect Bitlocker encryption on or off based on the "manage-bde -status" command and parsing output looking for the "Lock Status:" If its "Locked" I then parse the "Volume " line of the output to get to the drive letter. If there's multiple drive letters this won't work well. Maybe there's a better way, since we found the manage-bde -status output to be limited in the Windows PE environment with encryption enabled.
If the target computer isn't using Bitlocker encryption, the batch file loops through drive letters A-Z and finds the one where CrowdStrike exists. That seems to work pretty consistently. This isn't fool-proof but hopefully someone with better .bat file skills could enhance it. Thanks, Bruce
7/22: I haven't done .bat files in decades a colleague noticed I needed a "setlocal" at the beginning to clear variables. Also, the parsing of "manage-bde /status" doesn't seem to work in the WinPE environment so this approach is flawed.
setlocal cls @echo off for /f "delims=" %%a in ('manage-bde -status') do ECHO.%%a | FIND /I "Lock Status: Locked" > Nul && GoTo BitLockerOn GoTo BitLockerOff :BitLockerOn for /f "delims=" %%a in ('manage-bde -status') do ECHO.%%a | FIND /I "Volume " > Nul && for /f "tokens=2" %%i in ("%%a") do set word2=%%i && set drive=%word2% echo Using drive %drive% echo If your device is BitLocker encrypted use your phone to log on to https://aka.ms/aadrecoverykey. Log on with your Email ID and domain account password to find the BitLocker recovery key associated with your device. echo. manage-bde -protectors %drive% -get -Type RecoveryPassword echo. set /p reckey="Enter recovery key for this drive if required: " IF NOT [%reckey%] == [] ( echo Unlocking drive %drive% manage-bde -unlock %drive% -recoverypassword %reckey% ) Goto DeleteLogic :BitLockerOff for %%D in (A B C D E F G H I J K L M N O P Q R S T U V W X Y Z) do if exist "%%D:\Windows\System32\drivers\CrowdStrike\C-00000291*.sys" ( set drive=%%D: echo Using drive %drive% echo. Goto DeleteLogic ) :DeleteLogic del %drive%\Windows\System32\drivers\CrowdStrike\C-00000291*.sys echo Done performing cleanup operation.
Does this handle admin rights as well? We get prompted for admin rights when we go to safe mode manually.
Just a thought/idea as I was thinking of ideas for a similar tool.
One
of the big issues for enterprises with thousands of systems, each with
individual BitLocker recovery keys, is the slow one-at-a-time acquiring
and entering of the key.
I wonder if we could somehow bulk download a
database of keys, each with their Identifier and the Recovery Key, or
the .BEK files, (from the Enterprise based storage [somehow]), and then
have the usb based tool search and find and use the key from that
database.
Obviously this is 'risky' in that you now have all the keys
published to the set of trusted technicians, but would speed up the
process.
Or failing that, the trusted technicians need a txt file or
paper list of all the keys ... [but will soon get fed up of entering 48
digit keys manually ...]
Re "Does this handle admin rights as well? We get prompted for admin rights when we go to safe mode manually."
My understanding is you will still need a local admin account and credentials, to log on after entering Safe Mode.
Update - that relates to the other methods that use Safe Mode.
This
USB method uses its own recovery OS, and should be able to access the
filesystem on the original main OS to do the necessary fix up
Side
note - That's why the use of BitLocker is recommended to prevent
intruders with physical access to your system from being able to boot
from USB or similar and being able to access your files or tamper with
your system!
@stephc_msftYou can grab the bitlocker keys using graph api
Could use list to loop through and grab all of the deviceIds, put them in a table, then loop through with a Get and save all of the keys by deviceId. From there you could export it to a csv for easier parsing from the tool. From a security standpoint, at the end of this being useful, there's a beta command to rotate keys on managed devices. I'm thinking bulk key rotation.
@stephc_msft My solution does exactly that. You provide a csv file with keys and boot to the USB and it does the rest. No manually entering BitLocker keys or passwords.
https://theitchronicles.wordpress.com/2024/07/21/crowdstrike-bsod-recovery/
The comparison at line 276 is not working as expected.
The /1GB division is causing the types to be differente (double vs. int64) and skipping over the error message.
The following will work properly:
$usbVolume = Get-Volume -DriveLetter $USBDrive[0] if (($usbVolume.Size) -gt 32GB) { Write-Host "ERROR: USB drives larger than 32GB are not supported. Please shrink the drive partitions and re-run the script." Exit }
I would also suggest putting this info in the instructions about the drive limit.
But you don't even need Bitlocker keys to bypass Bitlocker and go into safe mode to remove the file good old bcdedit safeboot
Working great here. One suggestion though - could we have the option of creating an ISO as well? I know it creates one during the process, but it would be great if we had the option rather than having to race to copy the existing one. We've used the ISO to add to our network share so our support staff can create USB without needing the ADK.
Rather than creating an ISO ourselves, can Microsoft provide the ISO directly so we may just load it and get the ball rolling. This way we can send out to the respective support teams and mount them to our Hyper-V hosts.
@KirbyONG it creates one for you during the script - if you copy it before the USB drive is finished you can just use that one (should be in appdata\local under a subfolder - and the ISO is called 'CSRecoveryImage.iso')
But yes the tool should have the option to either create an ISO or the USB drive to make it easier.
@KirbyONG - an ISO would prevent the injection of drivers into WinPE, which the script covers.
jespinoza_vt - incorrect - an ISO is created either way (either with no additional drivers or after driver injection). The ISO is what the script uses to write the files to the USB, so they would just need to add an option if you wanted just the ISO.
It doesn't seem to like drives bigger then 16gb it seems, get this error and the drive will not boot
We've had reported issues (0x80070057) when running this tool on newer systems - anyone else?
[Edit] For those seeing this, I resolved it by injecting our OEM's driver pack into the PE image.
Looks like the MS Tool is worse than the CS infection. I sure hope one day we all get to find out who the CS Engineer (was) that released this infection brought the internet to its knees. Just like every other MS Patch does. TEST YOUR **bleep**!!!!!! Before you release it. So much unbillable support to this problem caused by someone's error.
Hi all, I´ve attached the ISO to a VM in Nutanix, and get the following error message, any ideas?
EDIT: Got it fixed by injecting the Nutanix VirtIO Drivers.
Re not seeing the disk if using a non-standard (non-built in) HBA driver.
Similarly for VMWare VM with paravirtual HBA, may need to drvload the relevant pvscsi.inf
Th boot from USB method also assumes the systems are allowed to boot from USB and/or do not have a 'BIOS/firmware' password set.
Thank you again for all your feedback. Still catching up on some of the comments, so appreciate the community help here! We've released an updated recovery tool with two repair options (see above) and the blog has been significantly adjusted. Thank you!
care to share brother? @aollivierre305
Classic Microsoft Solutions
Odd -- went through this -- created the USB stick, then tried to boot into it and got "required nvstor.sys is missing". I tried to inject some Dell PE drivers and recreate -- didn't help. Any ideas?
Thank you @Intune_Support_Team thank you for sharing this script.
Can you please incorporate the suggested fix by these experts @BruceK825 & @jespinoza_vt ?
I am getting below errors.
Seems like the problem is that it is not able unlock the bitlocker drive. Any suggestion on how this can be fixed? The machine also not able to go into safe mode.
@PrasannaD25
That looks like it cant find the drive, probably because it has a
non-standard HBA and may need to acquire and 'drvload' the relevant
driver.
Does diskpart, list disk see the disk (presumably not)
What
disk controller does it use? May well see this in some exotic servers
eg with HP megaraid, or on VMWare VM's with paravirtual scsi
eg as per Windows in recovery mode: No fixed disks to show - [bl...@kernstock.net]$
Update - the .ps1 script that creates the usb/iso has an option to add a driver
For those who had problem with the script try to move the folder containing the script on the root of the c drive
I was going to suggest a PXE version, but you beat me to it!
The script may show an error with Oscdimg
ERROR: The
following path for firmware files was not found: "C:\Program Files
(x86)\Windows Kits\10\Assessment and Deployment Kit\Deployment
Tools\AMD64\Oscdimg\..\..\amd64\Oscdimg".
more info to follow (may need to load the extra 'deployment tools' in the adk ..)
Go to add remove programs, find Windows Assessment and Deployment Toolkit, Modify, add Deployment tools
Then can rerun the script successfully
Note that if SecureBoot is also in use, that will affect the
'Safe Mode' option as SecureBoot doesnt like the boot config changing.
Plus cant just disable SecureBoot as it will notice that as well, and so still ask for a BitLocker Recovery Key.
Many people were able to access C: drive through cmd so that a script could run there to solve it. I made my own and successful.
For our organization, we solved many user's CrowdStrike app-related issues for Windows10 OS,
Hold the power button for 10 seconds to turn off your device and then press the power button again to turn on your device.
2. On the Windows sign-in screen, press and hold the Shift key while you select Power > Restart.
3. After your device restarts to the Choose an option screen, select Troubleshoot. On the Troubleshoot screen, select Advanced options
4. Pick Startup Settings and hit "Restart" and for the next step keep the BitLocker key ready if the drive is encrypted with the BitLocker.
5. After entering the BitLocker keys pick option # 4 which is "Safe Mode"
6. Next login with the OS Windows administrator account and the password if the account is renamed or using the LAPS password login to Safe Mode.
7. Type is Run or Search CMD to open up the command prompt.
CD C:\Windows\System32\drivers\CrowdStrike and hit Enter.
8. Once in the CrowdStrike directory, locate the file matching “C-00000291*.sys”. To do this, type the following command and then press Enter:
9. Permanently delete the file(s) found. To do this, type the following command and then press Enter.
del C-00000291*.sys
13. Type Exit from CMD and Safe Mode and restart your system to sign in normally.
@PrasannaD25 so most likely the BIOS is setup with RAID On/Storage. You'll need to change the BIOS settings from RAID to ACHI, let the PC go into the WinRE and follow the steps on the thread by either booting into Safe Mode or deleting the file C-00000291*.sys in CMD. Its possible you'll need to change the SATA Configuration back to RAID once these steps have been done.
@BruceK825 -- re: trying to get the status from the manage-bde.exe -status command, did you try using the -protectionaserrorlevel switch? This will send the status to the ERRORLEVEL variable (0=encrypted)
manage-bde status | Microsoft Learn
This is from the script that I wrote for our techs:
Thanks so much, how did I miss that? Way better than parsing the output of -status. I'll ask my colleague if it works in the WinPE environment. Thanks again !
The PXE option is exceedingly slow to transfer the WIM file.
Seeing about 29-30KB/s throughput best case at the moment.
Works for me. Thanks.
Here's my screenshot.
Hi all,
Was wondering if anyone encountered the below error, not a whole lot of information is given.
Thanks,
Evin
Just for info, CrowdStrike have released a very clear video
showing the manual ways of getting into SafeMode and deleting the
necessary bad file
It even mentions BitLocker and the need for the Recovery Key
CrowdStrike Host Self-Remediation for Remote Users with Local Administrator Privileges (youtube.com)
For Bitlocker (TPM only) setting up to go in Safe Mode via F8 or Advanced Settings needs the recovery key.
Going
into safe mode by directly editing the BCD (as per the usb or iso
techniques in this techcommunity article) usually allows you to boot
into Safe Mode without needing the key.
If SecureBoot is enabled,
then must ensure the iso is removed and the boot order is back to how it
was originally, to allow it to boot in safe mode without asking for the
key (else SecureBoot sees the config is different and blocks you).
For
unlocking BitLocker at scale eg Enterprises with hundreds of systems,
see a link earlier in the comments here, for another similar (3rd party)
tool/technique idea that can utilize a CSV database of BitLocker
Recovery Keys (need to preprovision that database obviously and only if
the Enterprise admins permit that approach).
Seems like a rather complex sort of arrangement to me. BOOTING to BIOS, enabling PXE etc.
My focus is server technology not desktops so this may not be ideal for desktop systems.
why not use an older method that is still usable on a Win2022 system. Execute the following as Administrator in CMD.
bcdedit /set {bootmgr} displaybootmenu yes
bcdedit /set {bootmgr} timeout n
I used 5 for n
this will delay the boot for n seconds but provide the options for selecting safe mode in advance of the GUI boot process.
Appreciate all the updates to this tool!! This has successfully remediated 100's of computers for us.
I got the same error as everyone else while trying to create bootable media.
For anyone who has a problem with the script:
Setup a
clean Windows 11 install (on a spare computer or even VM) and then run
the script again, so that it downloads a fresh copy of ADK and WinPE.
It is possible that the existing copy on the hard drive has missing
components that causes the script to fail.
Also, make sure the SD card or USB stick is freshly formatted - single partition and nothing else.
Oscdimg related error: See earlier comment about needing to add Deployment tools to the adk