Roblox Return to Service 10/28-10/31 2021 - Roblox Blog

Roblox had a tremendous 73 hour outage right before Halloween and they have now published a public incident report on it. I was very curious to read this. I remember thinking at the time that whatever was causing the problem must be horribly complicated, or was it a security issue? This writeup is very well done, and is a good read for anyone that is seeking to understand the incredible complexity of running very large scale services. Reading this made my palms sweaty in places because of situations I've been in the past, or just thinking what options the Roblox team had at various points.