Page 1 of 1

Global IT outage caused by Cloudstrike

Posted: 2024-07-19 05:18am
by bilateralrope
Global IT outage: What caused the CrowdStrike incident?
David Swan
July 19, 2024, • 08:14pm


A software outage on Friday afternoon crippled the airlines, banks and supermarkets in Australia, leaving flights grounded and newsreaders scrambling to fill air time.

The outage appears global in nature, impacting millions of systems in Australia and across the globe, and seems related to US cybersecurity provider CrowdStrike, and its software called the Falcon Sensor.

What is CrowdStrike Falcon?

CrowdStrike is one of the world’s largest cybersecurity vendors, providing thousands of businesses across the world with software to defend against viruses and cyberattacks.

The company is headquartered in Austin, Texas and has nearly 10,000 employees.

CrowdStrike Falcon is the company’s software that sits in the background on many corporate systems, detecting any viruses and cyber threats.

How did the outage happen?

The outage caused Microsoft laptops and PCs to show a ‘blue screen of death’, meaning workers and users were unable to access their systems.

The outage seemed to begin at around 2pm on Friday NZT, and affected users in the US before hitting Australia and New Zealand.

“We’re aware of a widespread issue causing BSOD errors on Windows machines across various sensor versions,” a CrowdStrike representative said in a forum post.

Why was Microsoft affected so much?

While Microsoft was not the source of the attack, CrowdStrike’s Falcon software is used on Microsoft Windows systems, rather than Apple Macs for example.

Microsoft said in a statement on X that it was investigating the incident.

What has the Australian government said?

The Australian government said there is no evidence to suggest the mass outage is a hack or cybersecurity incident.

“I am aware of a large-scale technical outage affecting a number of companies and services across Australia this afternoon,” National Cyber Security Coordinator Michelle McGuinness said in a statement on X.

“Our current information is this outage relates to a technical issue with a third-party software platform employed by affected companies.

“There is no information to suggest it is a cyber security incident. We continue to engage across key stakeholders.”

How often has this happened before?

Outages are relatively common these days, with telcos, banks and supermarkets often suffering technical glitches that typically last a few hours.

Such an outage may have never been seen on this scale, however, highlighting how reliant our modern economy is on technology, and increasingly how interconnected it is.

Is there a fix?

CrowdStrike has issued advice about a temporary workaround.

Here’s what the tech company says you should do:

- Boot Windows into Safe Mode or the Windows Recovery Environment (you can do that by holding down the F8 key before the Windows logo flashes on screen)
- Navigate to the C:\Windows\System32\drivers\Crowdstrike directory
- Locate the file matching “C-00000291*.sys” file, right click and rename it to “C-00000291*.renamed”
- Boot the host normally.


- Sydney Morning Herald
We probably won't know the full scale of this mess until more countries wake up and see how many of their computers have been hit.

Then comes the fun of figuring out how this happened. How an update this broken slipped past Cloudstrike's testing.

Re: Global IT outage caused by Cloudstrike

Posted: 2024-07-19 03:48pm
by Mr Bean
Hey guess who this was his entire day? :banghead:
My day started at 2am when the Great Crowd strike of 2024 began and it just ended a little bit ago 14 hours later. So many computers affected by this bug. And best still since it's Crowdstrike it crashes the system before our remote tools load so I spent all day going of station to station manually deleting the bad files and restarting machines.

Re: Global IT outage caused by Cloudstrike

Posted: 2024-07-19 08:54pm
by EnterpriseSovereign
I was fortunate that my system has no directory named "Crowdstrike", though I've a feeling they're going to lose a crapton of customers as a result of this particular fuckup.

Re: Global IT outage caused by Cloudstrike

Posted: 2024-07-20 12:20am
by AniThyng
https://www.crowdstrike.com/blog/statem ... ows-hosts/

Next to see how this mind boggling QA fail occurred.

Re: Global IT outage caused by Cloudstrike

Posted: 2024-07-20 01:30am
by bilateralrope
Here's a fun little detail:
Too funny: In 2010 McAffe caused a global IT meltdown due to a faulty update. CTO at this time was George Kurtz. Now he is CEO of #crowdstrike

https://www.zdnet.com/article/defective ... of-xp-pcs/
Are you guessing accident, negligence or sabotage ?

Mr Bean wrote: 2024-07-19 03:48pm Hey guess who this was his entire day? :banghead:
My day started at 2am when the Great Crowd strike of 2024 began and it just ended a little bit ago 14 hours later. So many computers affected by this bug. And best still since it's Crowdstrike it crashes the system before our remote tools load so I spent all day going of station to station manually deleting the bad files and restarting machines.
The IT department where I work were there until midnight friday fixing things. Just the things that needed to be fixed urgently. When they left, most of the security camera computers were still down with a promise to fix them sometime today.

Re: Global IT outage caused by Cloudstrike

Posted: 2024-07-20 09:29am
by Eternal_Freedom
Ironically, the only people seriously affected in my organisation was the IT Helpdesk. They didn't have working phones, or live chat, or even email. SO they're all losing their shit thinking how crippling this must be to the entire court service...and we barely even noticed, apart from a few transient VPN glitches.

Re: Global IT outage caused by Cloudstrike

Posted: 2024-07-20 05:28pm
by Mr Bean
Eternal_Freedom wrote: 2024-07-20 09:29am Ironically, the only people seriously affected in my organisation was the IT Helpdesk. They didn't have working phones, or live chat, or even email. SO they're all losing their shit thinking how crippling this must be to the entire court service...and we barely even noticed, apart from a few transient VPN glitches.
Our after action report is still being written but best guess is we had roughly 40% of all users were down for this and about 85% ish of all critical systems were down. The rule of thumb was if it was turned on during the evening updates it was down. So people who's laptops were off or in sleep mode during Thursday night/Friday morning escaped things just fine. But all the big important things like servers which run 24/7 got killed by Crowdstrike. My best guess is I manually myself restored about 4-5 servers and 60 users back into operation during the 10 hours I was onsite. And because of bitlocker we could not automate anything. Every single machine it was restart to safe mode oops time for Bitlocker keys, log in with the stupid random admin password okay time to go to system32/drivers/crowdstrike/ Murder file 291 restart and it worked fine. Thankfully for a lot of users since I had been on there machines before I could log in with my admin profile and skip the bitlocker step. But we had to pull in everyone who had admin credentials be they with networking or with systems show them how to fix the thing then let them loose as we had over 400 machines affected.

Re: Global IT outage caused by Cloudstrike

Posted: 2024-07-20 09:07pm
by Solauren
From Wikipedia;
This was caused by a faulty driver Channel File 291, installed in the Windows kernel that contained only null bytes. In a postmortem analysis by CrowdStrike it was determined that the null bytes did not cause machines to crash, but a different logic error related to the file.
.

Yeah, I can see that. If you don't account for null values, they can wreck havoc.

I've had to explain the difference between '0', Blank, Null (and Black = False on yn Fields in databases) on multiple occasions to people.

Re: Global IT outage caused by Cloudstrike

Posted: 2024-07-25 08:05am
by bilateralrope
CrowdStrike offers $10 Uber Eats gift card to partners as apology for global IT outage
That'll make it right

July 24, 2024 By Sebastian Moss


CrowdStrike has offered some partners a $10 Uber Eats gift card as an apology for pushing out an update that brought down millions of Windows systems.

The update caused a blue screen of death, and led to flights being delayed, hospital appointments being canceled, and retailers losing out on sales around the world.

“And for that, we send our heartfelt thanks and apologies for the inconvenience,” an email to partners first reported by TechCrunch states. “To express our gratitude, your next cup of coffee or late-night snack is on us!”

When TechCrunch checked the voucher, the Uber Eats page provided an error message that said the gift card “has been canceled by the issuing party and is no longer valid.”

The update is believed to have impacted around 8.5 million Windows-based systems, and is expected to lead to billions of dollars in insurance claims over the coming weeks and months.

It is not clear if CrowdStrike plans to make any restitution to its customers.
That's an impressive fuckup of an apology.

Re: Global IT outage caused by Cloudstrike

Posted: 2024-07-25 01:36pm
by Solauren
I'd be looking to move to a new platform if I was a Cloudstrike customer.

Re: Global IT outage caused by Cloudstrike

Posted: 2024-07-26 06:40am
by His Divine Shadow
Well better luck next time, we almost got it.