CrowdStrike file update bricks Windows machines around the world

[kitekrazy1]

https://www.theregister.com/2024/07/19/crowdstrike_falcon_sensor_bsod_incident/?utm_source=daily&utm_medium=newsletter&utm_content=top-article

Love the comments.

Let's test patches before we roll them out....nah, testing is for pussies.

[craigb]

ClownStrike.

[bitman]

Automatic "security" corruptdates.

 

[bitflipper]

IMO the two biggest lessons learned from this incident aren't being talked about.

First of all, third-party applications should not run in ring 0 (the kernel). Traditionally, Microsoft didn't allow it. Windows, like all modern operating systems, is designed specifically to prevent this kind of tragedy from even being possible. But the EU forced them to make access to the kernel accessible to third parties. Politicians are not knowledgeable enough to be making decrees regarding tech stuff they don't understand.

A crucial piece of the Falcon software is implemented as a pseudo-driver, which gives it Ring 0 permission, and it was flagged as a must-load-on-boot component. Windows treats it as a trusted hardware driver. These types of files (.sys) normally undergo extensive testing before Microsoft will allow them, but CrowdStrike figured out a way around that annoying formality by having the MS-approved .sys file read and execute code from an external script, giving them unfettered kernel access for anything they wanted. It was a corrupt script that led to the null pointer. A frickin' text file. It wasn't a bug in the traditional sense, but a bad design that wouldn't have revealed itself during testing. All enabled by a government ruling.

Second, this is what happens when any piece of critical software becomes ubiquitous - it becomes a potential single point of failure for large numbers of systems.  We've seen several recent incidents where malware rode on the back of legitimate low-level software that was forcibly installed. I can think of at least two such incidents that also involved security tools, just this year. Remember the SolarWinds incident from a few years back? It was similar in many ways to last week's CrowdStrike situation.

As a footnote, I had to smile when I read that while every other airline was paralyzed by the CrowdStrike incident, Southwest Airlines was unaffected. The reason: they're still using Windows 95 and Windows 3.1. Remember that when somebody accuses you of not keeping up with the times. Especially when their rationale is "don't you think about security?"

[mettelus]

51 minutes ago, bitflipper said:

by having the MS-approved .sys file read and execute code from an external script

+1, this particular tactic has been employed for years. Macros, batch files, etc., that use a "trusted" program to execute can cause serious issues. As they get discovered/quelled there always seems to be another exploit in the pipeline behind it.

[craigb]

Can you get a more suspicious name than "Trusted Installer?" ?

[Grem]

5 hours ago, bitflipper said:

Politicians are not knowledgeable enough to be making decrees regarding _ stuff they don't understand.

Fixed.

Agreed. But that's never stopped them from doing exactly that.

[Notes_Norton]

14 hours ago, bitflipper said:

Politicians are not knowledgeable enough to be making decrees regarding tech stuff they don't understand.

There was a lot in your post that I didn't understand, but this statement should have written in the constitution.

 

Notes ♫

[orcmid]

17 hours ago, bitflipper said:

As a footnote, I had to smile when I read that while every other airline was paralyzed by the CrowdStrike incident, Southwest Airlines was unaffected. The reason: they're still using Windows 95 and Windows 3.1. 

I find that to be incredible.  It is plausible that they don't use CrowdStrike and I enjoyed your account until this point.  Where can I find confirmation of that statement?

[bitflipper]

1 hour ago, orcmid said:

Where can I find confirmation of that statement?

Here's an article from the always-reliable Tom's Hardware site: https://www.tomshardware.com/software/windows/windows-31-saves-the-day-during-crowdstrike-outage

[craigb]

I'm going to watch this one later today...

 

[bitflipper]

btw, there is quite a lot of misinformation out on the interwebs about this incident, written by people who wouldn't actually understand it if it was explained to them. Or worse, by people with an axe to grind against Microsoft. (Sorry, Linux and Apple fans, but this vulnerability exists in those ecosystems as well.)

Here are two explanations from reliable sources. The first is by Dave Plummer, a retired Microsoft engineer who knows Windows literally from the inside out. The other is a bit more geeky but still explained in simplified terms anyone can follow.

[EDIT] Craig beat me to this while I was typing...

 

[bitflipper]

The scariest part of this whole episode is how much worse it could have been had the problem been introduced by malicious disruptors, rather than a simple mistake that was diagnosed and fixed within 90 minutes (although it'll still be days before the fix will have been applied to all the affected computers). Imagine if this had been a targeted attack on, say, the electrical grid. Or worse, on DAWs (insert iLok joke here).

[Promidi]

17 hours ago, bitflipper said:

 Or worse, on DAWs (insert iLok joke here).

Actually, there were some versions of Sonivox plugins that included a version of iLok that installed as a boot driver that was absolutely not compatible with Windows 10, yet merrily installed it on Windows 10 anyway.   It installed even if there was a later version of iLok already installed.

Problem was, that version of the iLok driver bricked your Windows 10 PC.  End result, BSOD - even in safe mode. 

The only solution, in that case, was to side boot from a recovery CD to set the “TKPD.sys” driver in your Windows 10 partition to “disabled” rather than “Boot”. 

If you didn't know how to do that, then it was a complete fresh Windows reinstall (upgrade install would not have been enough)

Fortunately when this happened to me, I was able to recover using the side boot from a recovery CD method to get back up and running.

[craigb]

3 hours ago, Promidi said:

Actually, there were some versions of Sonivox plugins that included a version of iLok that installed as a boot driver that was absolutely not compatible with Windows 10, yet merrily installed it on Windows 10 anyway.   It installed even if there was a later version of iLok already installed.

Problem was, that version of the iLok driver bricked your Windows 10 PC.  End result, BSOD - even in safe mode. 

The only solution, in that case, was to side boot from a recovery CD to set the “TKPD.sys” driver in your Windows 10 partition to “disabled” rather than “Boot”. 

If you didn't know how to do that, then it was a complete fresh Windows reinstall (upgrade install would not have been enough)

Fortunately when this happened to me, I was able to recover using the side boot from a recovery CD method to get back up and running.

 

?

 

 

[craigb]

Time to go back to this maybe? ?

 

https://www.windows93.net/

[bitflipper]

And that is why I have no Pace-protected plugins on my machine. 

[bitflipper]

Dave Plummer posted an update this morning, with a thought-provoking comparison to the Tylenol-poisoning incident that I'm sure most of us here are old enough to remember. At the time, I believed the Tylenol brand was dead for good. Who would buy a product that had a history of killing people? Clearly, I was wrong.

Crowdstrike will take a temporary hit in the stock market, but it will not die. Expect a major redesign of their flagship product, however. 

 

[craigb]

I'm still SMH over this!  Any data that can be used as an input (especially user-entered data) needs to be checked!  A .sys file completely filled with zeros should have been an easy test... 

[Old Joad]