Post-Incident Review on the Atlassian April 2022 outage - Atlassian Engineering

A 14-day outage for a major SaaS provider is exceedingly rare, and incredibly painful. This one is even more so as it was an accident.

The script that was executed followed our standard peer-review process that focused on which endpoint was being called and how. It did not cross-check the provided cloud site IDs to validate whether they referred to the Insight App or to the entire site, and the problem was that the script contained the ID for a customer's entire site. The result was an immediate deletion of 883 sites (representing 775 customers) between 07:38 UTC and 08:01 UTC on Tuesday, April 5th, 2022.

This had to be an incredibly difficult incident report to write-up, but Atlassian deserves credit for being clear with their customers on what happened.