Ever wondered how the Replication happens, how a new Replica is set up? how the Replication Lag comes down to 0? Here’s everything you need to know
✨ General steps to set up a new Replica are
- Take a point-in-time snapshot of the Master data.
- Spin up a Replica node with this snapshot.
- Configure the Replica to follow the Master and start the Replication.
- Replication begins and the Replica eventually catches up.
✨ How does Replica keep track of the sync with the Master?
Sequence Number. Every update on the Master is associated with a monotonically increasing sequence number. Replica persists the sequence number of the last operation of the Master it executed on its data.
When the point-in-time snapshot is taken of the Master, this sequence number is also part of the snapshot. So when Replica starts with this snapshot it knows till which operation it has the data already updated.
✨ Does this mean, we can skip creating Replica from a snapshot?
Yes. In this case, the Replica will start pulling and applying operations since the first-ever query that happened on the Master; and might take a long time to catch up and bring down Replication Lag to 0.
✨ How does a Replica catch up with Master?
Replica, through a Replication thread, pulls operations from the Master and applies them on its own data. The Replication proceeds only when it gets scheduled on the CPU. The more CPU Cycles it gets, the faster the data replicates.
Hence Replica context switches between multiple threads, a few to drive Replication, while others to serve live read traffic. This way even while serving the read traffic, the replica continues to fetch updates from the Master.
This also means if the Replication thread gets fewer CPU Cycles due to
- a large number of CPU-bound queries,
- a large number of updates from Master, or
- a very high live Read traffic
it may never catch up with Master.
✨ If one of your Replica is having a high Replication lag,
- stop any live read traffic going to it, and
- let it utilize all the CPU Cycles for Replication
This will quickly bring down the Replication lag.
✨ Long-form of this snippet dropping soon on my newsletter.
To date, I have written ~60 articles on Distributed Systems, System Design, Advanced Algorithms, and Python Internals. Right now I am running a series on Distributed Systems and System Design, which is shared through my newsletter.
1900+ people have subscribed to my newsletter. Join them at arpitbhayani.me/newsletter.
✨ If you want to master System Design and learn it the right way, you can enroll in my upcoming cohort starting September 4th.
We will together build some of the most amazing systems and dissect them to understand the intricate details.
You can find the week-by-week curriculum and topics, benefits, testimonials, and other details 👉 https://lnkd.in/dtBk7eE.