On Friday, July 19, 2024, the world experienced a significant outage across various industry sectors due to what initially appeared to be a minor software update to the CrowdStrike Falcon product on Microsoft system endpoints. This update introduced a corrupt system file, which led to a "blue screen of death" when booted by a Microsoft operating system, rendering hundreds of thousands of endpoints unusable and in need of direct human intervention for recovery.
As a partner to CrowdStrike, we share concerns for both our customers and our partner regarding this issue. We want to provide a guide to help mitigate and recover impacted devices in this situation.
Summary
- CrowdStrike is aware of reports of crashes on Windows hosts related to the Falcon Sensor. Full Statement is covered here
Details
- Symptoms include hosts experiencing a bugcheck\blue screen error related to the Falcon Sensor.
- This issue is not impacting Mac- or Linux-based hosts
- Channel file "C-00000291*.sys" with timestamp of 0527 UTC or later is the reverted (good) version.
Current Action
- CrowdStrike Engineering has identified a content deployment related to this issue and reverted those changes.
- If hosts are still crashing and unable to stay online to receive the Channel File Changes, the following steps can be used to workaround this issue:
Workaround Steps for individual hosts:
Reboot the host to give it an opportunity to download the reverted channel file. If the host crashes again, then:
- Boot Windows into Safe Mode or the Windows Recovery Environment
- Navigate to the C:\Windows\System32\drivers\CrowdStrike directory
- Locate the file matching “C-00000291*.sys”, and delete it.
- Boot the host normally.
Note: Bitlocker-encrypted hosts may require a recovery key.
Workaround Steps for public cloud or similar environment:
- Detach the operating system disk volume from the impacted virtual server
- Create a snapshot or backup of the disk volume before proceeding further as a precaution against unintended changes
- Attach/mount the volume to to a new virtual server
- Navigate to the C:\Windows\System32\drivers\CrowdStrike directory
- Locate the file matching “C-00000291*.sys”, and delete it.
- Detach the volume from the new virtual server
- Reattach the fixed volume to the impacted virtual server