CrowdStrike IT Outage Explained by a Windows Developer

CrowdStrike IT Outage Explained by a Windows Developer

Introduction

Hey, I'm Dave. Welcome to my shop. I'm Dave, a retired plumber and software engineer from Microsoft, going back to the MS DOS and Windows 95 days. Thanks to my time as a Windows developer, today I'm going to explain what the CrowdStrike issue actually is, the key difference in kernel mode, why these machines are blue screening, and how to fix it if you come across one.

Background

I have a lot of experience working up to blue screens and having them set the tempo of my day. This Friday was a little different. I'm retired now, so I don't debug a lot of blue screens and I was traveling in New York City, which left me temporarily stranded as the airlines sorted out the digital carnage. That downtime gave me plenty of time to pull out the old MacBook and figure out what was happening to all the Windows machines around the world.

The Cause of Blue Screens

As far as we know, the CrowdStrike blue screens that we’ve been seeing around the world for the last several days are the result of a bad update to the CrowdStrike software.

Why CrowdStrike Software is on Machines

CrowdStrike Falcon is a security product that is essential for modern IT infrastructures. Its job is not just to look for antivirus definitions, but to analyze a wide range of application behaviors to proactively detect new attacks.

What Happens When a Kernel Driver Fails

The key difference in kernel mode involves tasks such as talking to the hardware, managing memory, and scheduling threads. In short, the kernel is the cornerstone of the operating system. When a kernel driver like CrowdStrike fails, it often results in what's called a blue screen of death (BSOD).

Debugging the Issue

To debug the problem, systems developers used to run tests under stress to replicate the crashes. After identifying the problematic areas, we could use tools like Telnet to connect to the target machine and debug it. Debugging was mostly done in assembly language, giving us just enough information to sort most crashes.

How Kernel Mode Works

User-mode and kernel-mode are bifurcated in their operations. Application code runs in user-mode while the operating system code runs in kernel-mode. If a kernel-mode crashes, it takes the whole system down with it, leading to the infamous blue screen.

CrowdStrike and Kernel Mode

CrowdStrike had to execute in kernel mode to effectively monitor application behaviors, making it susceptible to kernel failures. The recent update to CrowdStrike's Falcon sensor seems to be the culprit for these blue screens.

WHQL Certification

Microsoft offers WHQL certification for third-party kernel mode drivers to ensure their robustness and compatibility with Windows. However, CrowdStrike chose to take a shortcut by using dynamic definitions as PE (Portable Executable) code files that their driver could execute. These definitions aren't re-verified by WHQL, leading to potential instability.

Postmortem Debugging

We examined a typical crash dump from Twitter and found that the cause is related to invalid memory access. This happened because CrowdStrike’s dynamic definition file was all zeros, leading the driver to attempt operations on null pointers.

Fixing the Issue

To fix this issue, you need to boot the system into safe mode, navigate to the directory \Windows\System32\drivers\CrowdStrike, and delete the problematic update file. The system should then boot normally.

Conclusion

In summary, the CrowdStrike issue is a result of a bad update file that lacked adequate error checking and parameter validation, leading to a blue screen. Windows does have mechanisms like safe mode to mitigate these issues, but CrowdStrike's decision to employ dynamic updates without re-certification caused the instability.

Keywords

• CrowdStrike Falcon
• Kernel Mode
• Blue Screen of Death (BSOD)
• WHQL Certification
• Dynamic Definitions
• Windows Debugging

FAQ

Q: What is CrowdStrike Falcon?

A: CrowdStrike Falcon is a security product designed to detect malware and other threats proactively by analyzing application behaviors.

Q: Why do blue screens occur in this context?

A: Blue screens occur due to a fault in kernel mode, where CrowdStrike's current update had a bug that caused memory access violations.

Q: How can I fix my computer if it’s affected by this issue?

A: Boot your computer into safe mode, navigate to the directory \Windows\System32\drivers\CrowdStrike, and delete the problematic update file to resolve the issue.

Q: What is WHQL Certification?

A: WHQL stands for Windows Hardware Quality Labs, a certification that ensures third-party drivers are robust and compatible with Windows.

Q: Why doesn’t Windows boot without problematic drivers like CrowdStrike?

A: CrowdStrike's driver is marked as a boot driver, making it essential for starting the Windows operating system.

If you found today's article informative or entertaining, please consider subscribing to my channel and leaving a like. Also, check out my new book on Amazon: "The Non-Visible Part of the Autism Spectrum." Thank you!