How AWS S3 serves 1 petabyte per second on top of slow HDDs
bigdata.2minutestreaming.com/p/how-aws-s3-scales-with-tens-of-millions-of-hard-drives
What Jamie wrote
The scale of S3 is really hard to understand even for technologists who have worked on very scaled systems.
AWS S3 is a massively multi-tenant storage service. It's a gigantic distributed system consisting of many individually slow nodes that on aggregate allow you to access data faster than any single node can provide. S3 achieves this through:
- massive parallelization across the end-to-end path (user, client, server)
- neat load-balancing tricks like the power of two random
- spreading out data via erasure coding
- lowering tail latency via hedge requests
- the economies of multi-tenancy at world scale
This article is a very approachable overview on the key methods they use to hit this performance.