Brief thoughts on the recent Cloudflare outage – Surfing Complexity

This is a great overview of the Cloudflare issue from a couple weeks ago and the writeup. Hochstein brings some additional color that is really good. And I love this statement because it is completely true.

We humans are excellent at recognizing patterns based on our experience, and that generally serves us well during incidents. Someone who is really good at operations can frequently diagnose the problem very quickly just by, say, the shape of a particular graph on a dashboard, or by seeing a specific symptom and recalling similar failures that happened recently.

When you run a platform, particularly a large and complex one, you get a sense for it’s behavior and what is normal. A single graph shape can be all you need to know if something is amiss.