A software outage on Friday afternoon crippled the airlines, banks and supermarkets in Australia, leaving flights grounded and newsreaders scrambling to fill air time.
The outage appears global in nature, impacting millions of systems in Australia and across the globe, and seems related to US cybersecurity provider CrowdStrike, and its software called the Falcon Sensor.
What is CrowdStrike Falcon?
CrowdStrike is one of the world’s largest cybersecurity vendors, providing thousands of businesses across the world with software to defend against viruses and cyberattacks.
The company is headquartered in Austin, Texas and has nearly 10,000 employees.
CrowdStrike Falcon is the company’s software that sits in the background on many corporate systems, detecting any viruses and cyber threats.
How did the outage happen?
The outage caused Microsoft laptops and PCs to show a ‘blue screen of death’, meaning workers and users were unable to access their systems.
The outage seemed to begin at around 2pm on Friday NZT, and affected users in the US before hitting Australia and New Zealand.
“We’re aware of a widespread issue causing BSOD errors on Windows machines across various sensor versions,” a CrowdStrike representative said in a forum post.
Why was Microsoft affected so much?
While Microsoft was not the source of the attack, CrowdStrike’s Falcon software is used on Microsoft Windows systems, rather than Apple Macs for example.
Microsoft said in a statement on X that it was investigating the incident.
What has the Australian government said?
The Australian government said there is no evidence to suggest the mass outage is a hack or cybersecurity incident.
“I am aware of a large-scale technical outage affecting a number of companies and services across Australia this afternoon,” National Cyber Security Coordinator Michelle McGuinness said in a statement on X.
“Our current information is this outage relates to a technical issue with a third-party software platform employed by affected companies.
“There is no information to suggest it is a cyber security incident. We continue to engage across key stakeholders.”
How often has this happened before?
Outages are relatively common these days, with telcos, banks and supermarkets often suffering technical glitches that typically last a few hours.
Such an outage may have never been seen on this scale, however, highlighting how reliant our modern economy is on technology, and increasingly how interconnected it is.
Is there a fix?
CrowdStrike has issued advice about a temporary workaround.
Here’s what the tech company says you should do:
- Boot Windows into Safe Mode or the Windows Recovery Environment (you can do that by holding down the F8 key before the Windows logo flashes on screen)
- Navigate to the C:\Windows\System32\drivers\Crowdstrike directory
- Locate the file matching “C-00000291*.sys” file, right click and rename it to “C-00000291*.renamed”
- Boot the host normally.
- Sydney Morning Herald
We probably won't know the full scale of this mess until more countries wake up and see how many of their computers have been hit.
Then comes the fun of figuring out how this happened. How an update this broken slipped past Cloudstrike's testing.
Hey guess who this was his entire day?
My day started at 2am when the Great Crowd strike of 2024 began and it just ended a little bit ago 14 hours later. So many computers affected by this bug. And best still since it's Crowdstrike it crashes the system before our remote tools load so I spent all day going of station to station manually deleting the bad files and restarting machines.
"A cult is a religion with no political power." -Tom Wolfe Pardon me for sounding like a dick, but I'm playing the tiniest violin in the world right now-Dalton
I was fortunate that my system has no directory named "Crowdstrike", though I've a feeling they're going to lose a crapton of customers as a result of this particular fuckup.
Are you guessing accident, negligence or sabotage ?
Mr Bean wrote: ↑2024-07-19 03:48pm
Hey guess who this was his entire day?
My day started at 2am when the Great Crowd strike of 2024 began and it just ended a little bit ago 14 hours later. So many computers affected by this bug. And best still since it's Crowdstrike it crashes the system before our remote tools load so I spent all day going of station to station manually deleting the bad files and restarting machines.
The IT department where I work were there until midnight friday fixing things. Just the things that needed to be fixed urgently. When they left, most of the security camera computers were still down with a promise to fix them sometime today.
Ironically, the only people seriously affected in my organisation was the IT Helpdesk. They didn't have working phones, or live chat, or even email. SO they're all losing their shit thinking how crippling this must be to the entire court service...and we barely even noticed, apart from a few transient VPN glitches.
Baltar: "I don't want to miss a moment of the last Battlestar's destruction!"
Centurion: "Sir, I really think you should look at the other Battlestar."
Baltar: "What are you babbling about other...it's impossible!"
Centurion: "No. It is a Battlestar."
Corrax Entry 7:17: So you walk eternally through the shadow realms, standing against evil where all others falter. May your thirst for retribution never quench, may the blood on your sword never dry, and may we never need you again.
Eternal_Freedom wrote: ↑2024-07-20 09:29am
Ironically, the only people seriously affected in my organisation was the IT Helpdesk. They didn't have working phones, or live chat, or even email. SO they're all losing their shit thinking how crippling this must be to the entire court service...and we barely even noticed, apart from a few transient VPN glitches.
Our after action report is still being written but best guess is we had roughly 40% of all users were down for this and about 85% ish of all critical systems were down. The rule of thumb was if it was turned on during the evening updates it was down. So people who's laptops were off or in sleep mode during Thursday night/Friday morning escaped things just fine. But all the big important things like servers which run 24/7 got killed by Crowdstrike. My best guess is I manually myself restored about 4-5 servers and 60 users back into operation during the 10 hours I was onsite. And because of bitlocker we could not automate anything. Every single machine it was restart to safe mode oops time for Bitlocker keys, log in with the stupid random admin password okay time to go to system32/drivers/crowdstrike/ Murder file 291 restart and it worked fine. Thankfully for a lot of users since I had been on there machines before I could log in with my admin profile and skip the bitlocker step. But we had to pull in everyone who had admin credentials be they with networking or with systems show them how to fix the thing then let them loose as we had over 400 machines affected.
"A cult is a religion with no political power." -Tom Wolfe Pardon me for sounding like a dick, but I'm playing the tiniest violin in the world right now-Dalton
This was caused by a faulty driver Channel File 291, installed in the Windows kernel that contained only null bytes. In a postmortem analysis by CrowdStrike it was determined that the null bytes did not cause machines to crash, but a different logic error related to the file.
.
Yeah, I can see that. If you don't account for null values, they can wreck havoc.
I've had to explain the difference between '0', Blank, Null (and Black = False on yn Fields in databases) on multiple occasions to people.
I've been asked why I still follow a few of the people I know on Facebook with 'interesting political habits and view points'.
It's so when they comment on or approve of something, I know what pages to block/what not to vote for.
CrowdStrike has offered some partners a $10 Uber Eats gift card as an apology for pushing out an update that brought down millions of Windows systems.
The update caused a blue screen of death, and led to flights being delayed, hospital appointments being canceled, and retailers losing out on sales around the world.
“And for that, we send our heartfelt thanks and apologies for the inconvenience,” an email to partners first reported by TechCrunch states. “To express our gratitude, your next cup of coffee or late-night snack is on us!”
When TechCrunch checked the voucher, the Uber Eats page provided an error message that said the gift card “has been canceled by the issuing party and is no longer valid.”
The update is believed to have impacted around 8.5 million Windows-based systems, and is expected to lead to billions of dollars in insurance claims over the coming weeks and months.
It is not clear if CrowdStrike plans to make any restitution to its customers.
