On Wednesday, CrowdStrike released a report outlining the initial results of its investigation into the incident, which involved a file that helps CrowdStrike’s security platform look for signs of malicious hacking on customer devices.
The company routinely tests its software updates before pushing them out to customers, CrowdStrike said in the report. But on July 19, a bug in CrowdStrike’s cloud-based testing system — specifically, the part that runs validation checks on new updates prior to release — ended up allowing the software to be pushed out “despite containing problematic content data.”
The bad release was published just after midnight Eastern time on July 19, and rolled back an hour and a half later, at 1:27 a.m. Eastern, CrowdStrike said. But by then millions of computers had already automatically downloaded the faulty update. The issue affected only Windows devices, not Mac or Linux machines, and only those that were switched on and able to receive updates during those early morning hours.
Thanks to the timing of the incident, organizations in Europe and Asia “had more of their work day affected by the outage, unlike the Americas,” Fitch wrote in its blog post.
Impact on Windows Devices
When Windows devices using CrowdStrike’s cybersecurity tools tried to access the flawed file, it caused an “out-of-bounds memory read” that “could not be gracefully handled, resulting in a Windows operating system crash,” CrowdStrike said.
That’s the Blue Screen of Death that many people reported seeing on their machines, and that only a manual intervention to delete the bad file could fix — a slow, painstaking process when you consider that as many as 8.5 million individual devices will need to be reset this way.
That figure is small as a percentage of the wider Windows ecosystem, said Microsoft — a company that played no direct role in the outage. Still, Microsoft said in a blog post, it “demonstrates the interconnected nature of our broad ecosystem.”
Future Precautions
CrowdStrike said that the testing and validation system that approved the bad software update had appeared to function normally for other releases made earlier in the year. But it pledged Wednesday to keep software glitches like last week’s from happening again, and to publicly release a more detailed analysis when it becomes available.
The company added that it is developing a new check for its validation system “to guard against this type of problematic content from being deployed in the future.”
And CrowdStrike said it also plans to move to a staggered approach to releasing content updates so that not everyone receives the same update at once, and to give customers more fine-grained control over when the updates are installed.
CNN’s Sean Lyngaas contributed to this report.