The Day CrowdStrike Took Down the World — How We Responded

What happened

At 4 AM I got pinged in one of my IT Discord channels. Someone had posted about workstations going down with CrowdStrike in the BSOD stack trace. I pulled up BleepingComputer, confirmed what was happening, posted the article to our cybersecurity Slack channel so the team would see it first thing, and went back to sleep.

Woke up at 7:30 AM to a lot of messages. We had 4,000+ workstations across 200+ locations stuck in a BSOD loop, plus servers that internal IT had to handle directly. The affected systems included front desk workstations, employee devices, and other club workstations. CrowdStrike had pushed a bad channel file, C-00000291*.sys, that caused a kernel crash before Windows could finish loading. The fix was documented: boot outside of Windows, go to C:\Windows\System32\drivers\CrowdStrike\, delete the file, reboot. Getting that done at our scale was the actual problem.

Friday morning: triage and SOP

At 8 AM we had an IT meeting to figure out priorities. Servers came first. After that we focused on front desk workstations at each location, then employee devices and other club workstations. Internal IT worked through servers directly.

For the 200+ gym locations the plan was helpdesk calling each site and walking staff through the fix. Before that could happen helpdesk needed a clear process, so I put together an SOP covering the full steps including how to retrieve the BitLocker recovery key from Active Directory. Some workstations were encrypted and staff would hit a BitLocker prompt before getting anywhere near the CrowdStrike folder. Having that documented upfront saved a lot of time.

Lunch: building the WinPE drive

The manual process worked but it was slow and inconsistent over the phone. Around noon I put together a bootable WinPE drive with a script embedded that automated the whole thing. Once a workstation boots from the USB, Remove-CrowdStrikeFile.ps1 runs automatically with no input needed from whoever is standing there. It finds the OS drive on its own, deletes any file matching C-00000291*, writes a timestamped log to C:\Temp\FileDeletionLog.txt, and reboots. About 5 minutes per workstation.

How to build the USB drive (Rufus SOP)

Download the ISO and Rufus portable first:

Verify the ISO hash before writing anything:

Get-FileHash .\CrowdStrike_File_Removal_Fix.iso -Algorithm SHA256
# Expected: 01D94B0A4610F45461233BCDF5BA959BFD2E2F94043EF5195B9533EB8670D381

Then open Rufus and create the bootable drive:

1. Plug in a USB drive (8 GB or larger) and open rufusp.exe
2. Under Device, select your USB drive from the dropdown
3. Under Boot selection, click Select and choose CrowdStrike_File_Removal_Fix.iso
4. Leave all other settings as default
5. Click Start, then select Write in ISO Image mode (Recommended) when prompted
6. Click OK on the warning that the drive will be wiped, then wait for it to complete
7. Click Close when done. The drive is ready.

Booting from the USB on an affected workstation

Plug the USB into the affected workstation and power it on. Hit the boot menu key as soon as the manufacturer logo appears. You have about 2 seconds. Select the USB drive from the list.

Once it boots from the USB the script runs on its own. No input is needed. It finds the OS drive, deletes the bad file, and reboots the workstation automatically. The whole thing is about 5 minutes.

Friday afternoon: ordering drives

We didn't have enough USB drives on hand to cover 200+ locations, so Friday afternoon we ordered a batch through Amazon. Next day delivery for what we could get, standard for the rest.

Monday: imaging and shipping

Monday morning we used Rufus to image all the drives that had arrived over the weekend and shipped them out to remaining locations. Helpdesk kept calling sites ahead of the drives arriving so staff knew what was coming and what to do when it got there.

Third party techs with site visits already scheduled got the ISO from us and burned their own drives on site with Rufus before heading out. That took care of a solid chunk of locations without us needing to ship anything.

Recovery

Servers and priority front desk workstations were back up within the first couple days. Everything else came in over the rest of the week as drives arrived and site visits happened. Full recovery took about 1 week. Each workstation left a log at C:\Temp\FileDeletionLog.txt that went into the incident report.