When building a highly available system, you generally focus on one of two strategies
- reduce mean time to failure, or
- reduce mean time to repair
Depending on what you are building and what complexity it involves, you will pick one over the other. Let’s understand them better with concrete examples.
Reducing MTTF is all about ensuring fewer failures; some common practices to minimize this are.
- load-balanced active-active replicas
- multi-AZ or multi-region deployments
- data replicated to ensure no data or availability loss
- prevent outages with circuit breakers and rate limiters
While reducing Mean Time To Recovery (MTTR) is about accepting that things will fail, the time it takes to recover will be significantly smaller; some common practices for this are
- containerized workloads with VM auto-scaled
- putting features behind feature flags
- staggered rollouts with instant rollbacks
- read replicas with alerting to do a quick switch
Like for everything else in software engineering, there is no one right way to do anything. It depends on the context, the engineering complexity, the money in the bank, and the tolerance level.
Hope this helps.
btw, enrollments open for sys design June cohort for sde2+, ~13 seats left - arpitbhayani.me/course