How important is a rate limiter for a system like S3? ⚡
S3 stores data on commodity magnetic hard disks - a classic performance vs cost tradeoff. Given that magnetic disks are prone to wear and tear and mechanical failures, it is essential for S3 to employ a sophisticated rate limiting
- per access key,
- per object,
- per bucket, and
- per AWS account
In the absence of a properly configured rate limiter, the underlying hardware would throttle and will undergo wear and tear requiring replacement. Second order effect of this will be hot disk partitions leading to failures, and worse data loss.
Keep the above pointers in mind when you dive deep into building blob storage or build a similar system at your workplace.
⚡ The above discussion happened in my bi-weekly doubt-solving session for my System Design Beginner’s course, where I have covered System Design from absolute scratch.
arpitbhayani.me/sys-design
I keep writing and sharing these engineering nuggets, so if you are keen on learning them, follow along.