CrowdStrike, the cybersecurity company responsible for the recent global IT outages, has attributed the chaos to a bug in their quality control software. This bug allowed faulty data to be sent to millions of computers running Microsoft Windows.
On July 19, approximately 8.5 million Windows machines worldwide crashed as a result of this issue. The fallout from these crashes was significant, with airports grounding flights, TV broadcasts going off the air, and disruptions occurring in banks, hospitals, and even the London Stock Exchange. Some businesses affected by these outages are still struggling to recover.
CrowdStrike regularly sends out updates for its Falcon Sensor product—a software suite designed to monitor and protect users’ computers from threats and attacks. These updates are delivered in two ways: “Sensor Content” directly updates CrowdStrike’s Falcon Sensor at a high level of system resource access; while “Rapid Response Content” updates how the sensor detects malware for quick response to evolving threats.
However, on the morning of July 19th, a Rapid Response Content update containing a broken file slipped through CrowdStrike’s quality control software due to a bug in their Content Validator. In their post-incident review published on July 24th, CrowdStrike acknowledged that one of the problematic updates passed validation despite containing faulty content data.
The assumption that this update wouldn’t cause any issues led to an out-of-bounds memory read error within Falcon Sensor when it loaded the problematic update. This error occurs when a program attempts to read data from memory outside its allowed bounds and triggered an exception that resulted in crashes within Windows operating systems.
Following this incident which caused CrowdStrike’s stock value to drop by one-fifth, they have pledged reforms regarding critical content updates. They plan on implementing a “staggered deployment strategy,” known as “canary deployment,” where future updates will be initially sent only to select machines before being rolled out globally.
Additionally, CrowdStrike intends to enhance error handling within their Content Interpreter component and introduce extra validation checks into their content validator. They will also provide customers with more control over when and where these updates are deployed.
George Kurtz, founder and CEO of CrowdStrike expressed his commitment towards transparency and preventing similar incidents from happening again: “Nothing is more important than the trust our customers have placed in us… We will provide full transparency on how this occurred.”
It remains crucial for companies like CrowdStrike not only address such incidents but also take proactive measures towards preventing future disruptions caused by bugs or faulty software releases.
