A global IT outage has caused chaos at airports, banks, railways andbusinesses around the world as a wide range of services were taken offline and millions of people were affected.
In one of the most widespread IT crashes ever to hit companies and institutions globally, air transport ground to a halt, hospitals were affected and large numbers of workers were unable to access their computers. In the UK Sky News was taken off air temporarily and the NHS GP booking system was down.
Microsoft’s Windows service was at the centre of the outage, with experts linking the problem to a software update from cybersecurity firm Crowdstrike that has affected computer systems around the world. Experts said the outage could take days from which to recover because every PC may have to be fixed manually.
Overnight, Microsoft confirmed it was investigating an issue with its services and apps, with the organisation’s service health website warning of “service degradation” that meant users may not be able to access many of the company’s most popular services, used by millions of business and people around the world.
Among the affected firms are Ryanair, Europe’s largest airline, which said on its website: “Potential disruptions across the network (Fri 19 July) due to a global third party system outage … We advise passengers to arrive at the airport three hours in advance of their flight to avoid any disruptions.”
Agreed, but again these updates were done by the Crowdstrike software. Nothing to do with Microsoft or Windows.
In this case it was an update to the security component which is specifically designed to protect against exploits on the endpoint. You'd want your security system to be up to date to protect as much as possible against new exploits. So updating this every day is a normal thing. In a corporate environment you do not want you end users to be able to block or postpone security updates.
With Microsoft updates they get rolled out to different so called rings, which get bigger and bigger with each ring. This means every update is already in use by a smaller population, which reduces the chances of an update destroying the world like this greatly.
I absolutely expect vendors to push out new patterns automatically and as fast as possible.
But in this case, a new system driver was rolled out. And when updating system software, I absolutely expect security vendors to use a staged rollout like everyone else.
100% agreed, Crowdstrike fucked up with this one. I'm very interested to hear what went wrong. I assume they test their device drivers before deploying them to millions of customers, so something must have gone wrong between testing and deployment.
Something like this simply cannot happen and this will cost them customers. Your reputation is everything in the security business, you trust you security provider to protect your systems. If the trust is gone, they are gone.
We'll probably never know. Given the impact of this fuck up, the most that crowdstrike will probably publish is a lawyer-corpo-talk how they did an oopsie doopsie, how complicated, unforseen, and absolutely unavoidable this issue has been, and how they are absolutely not responsible for it, but because they are such a great company and such good guys, they will implement measures that this absolutely, never ever again will happen.
If they admit any smallest wrongdoing whatsoever they will be piledrived by more lawyers than even they'd be able to handle. That's a lot of CEO yachts in compensations if they will be held responsible.