Cybersecurity firm CrowdStrike said a faulty update that caused global technology outages was checked before being sent out last Friday, but a bug in the validation tool caused it to miss the underlying issue.
In a post-mortem released on Wednesday, the company said it uses a “Content Validator” to check updates before they are distributed to customers’ systems. The faulty update “passed validation despite containing problematic content data,” according to CrowdStrike.
Officials at CrowdStrike said they trusted the content validator because previous checks it had done as recently as March 5 had no issues.
The “problematic content” sent out on July 19 resulted in a Windows operating system crash that impacted thousands of critical systems across the world, including airlines, hospitals and banks.
CrowdStrike pledged to provide a more detailed breakdown of the faulty update and also listed out several changes designed to prevent a similar situation from ever happening again. The cybersecurity giant plans to institute more local testing procedures and validation checks ahead of any future release.
CrowdStrike will also implement a staggered deployment strategy for updates going forward, gradually deploying them to large portions of their customer base. Customers will also be given more control over how updates are delivered.
Microsoft said on Saturday that its estimates showed about 8.5 million Windows devices were taken offline by the faulty update. The figure represents less than one percent of all Windows machines, according to Microsoft, but CrowdStrike products are used by some of the world’s most critical organizations — including federal agencies, emergency services and more.
Both Microsoft and CrowdStrike have released troves of guidance and videos to help IT administrators with the herculean task of restoring thousands of devices — a process which has to be done manually and can take up to 30 minutes.
CrowdStrike also faced significant backlash on Wednesday when TechCrunch revealed that the company offered $10 UberEats gift cards to partners trying to remediate the issues.
The crisis drew criticism from the White House last week, and on Monday the House Committee on Homeland Security demanded that officials from CrowdStrike testify before Congress and provide details about what happened.
“This incident must serve as a broader warning about the national security risks associated with network dependency. Malicious cyber actors backed by nation-states, such as China and Russia, are watching our response to this incident closely,” leading members of the committee said in a letter to CrowdStrike CEO George Kurtz.
“In fact, as CrowdStrike relayed in a recent blog post, malicious actors presumably targeting your Latin American customers have already seized the moment and sought to exploit the vulnerability. Protecting our critical infrastructure requires us to learn from this incident and ensure that it does not happen again.”
Recorded Future
Intelligence Cloud.
No previous article
No new articles
Jonathan Greig
is a Breaking News Reporter at Recorded Future News. Jonathan has worked across the globe as a journalist since 2014. Before moving back to New York City, he worked for news outlets in South Africa, Jordan and Cambodia. He previously covered cybersecurity at ZDNet and TechRepublic.
