Tag: replication lag

What is Replication and Why is it Important?

Sean Mayer3 Comments

Lets firstly breakdown the goals of replication

  • High availability
    • Maintaining that a system is running at an agreed level of operation.
    • Even when one machine or several machines or an entire data centre goes down!
  • Disconnected Operations – Clients with offline operations
    • Enabling the application to continue working when there is network latency
  • Latency
    • Placing data geographically close to users so that the users can interact with it faster
  • Scalability
    • The ability to handle a higher volume of reads, that a single machine can handle by performing these reads on replicas

The humbling reality…🙏

Despite being a simplistic objective to keep copies on several machines, replication is remarkably a tricky problem! It requires careful attention to:

  • 📝Concurrency 👈
  • 📝Things that can go wrong 👈
  • 📝Dealing with the consequences of the faults 👈

At a minimum, we will generally require dealing with the following:

  • 📝Unavailable nodes 🔴❌
  • 📝Network interruptions 📶 ❌

And that is not even considering the more insidious faults 🧟‍♂️, such as:

  • 📝Silent data corruption due to software bugs 🪲

What approaches we can take with replication?

  • Single leader replication
    • Clients send all writes to a single node (leader)
      • Streams of data change events are sent between followers
    • Reads can be performed by any replica
      • But followers may return stale reads
  • Multi-leader replication
    • Where clients send each write to one of several leader nodes
      • Any of which can accept writes
    • Streams of data change events are sent between leaders and to any follower nodes
    • Related, on choosing the best multi-leader topology: The Multi-Leader Replication Topologies
  • Leaderless replication
    • Clients send each write to several nodes
    • There is the ability to read from several nodes in parallel ⬇️
      • In order to correct and detect nodes with stale data

Advantages and disadvantage of replication

Single leader replication is the most popular because:

  • Easier to understand ✅ ☺️
  • No conflict resolution to worry about ✅

Multi-leader replication and Leaderless replication can be more robust in handling:

  • Faulty nodes ✅
  • Network interruptions ✅
  • Latency spikes ✅

At the cost of being:

  • Harder to reason about ❌
  • Providing only very weak consistency guarantees to end users ❌

Asynchronous and synchronous replicaton

This can have a profound affect on the system behaviour when there is a fault.

Asynchronous replication:

  • Can be faster when the system is running as expected ✅
  • It is important to figure out what happen when replication lag increases or servers failures 👈📝
  • If a leader fails and you promote an asynchronously updated follower to be the new leader 🤔
    • There is a risk… ⚠️
      • the recently committed data maybe lost ❌

Replication Lag

Consistency models can be utilised to combat replication lag by giving the replicas a set of instructions on how to behave when this occurs.

The consistency models:

  • Read after write consistency
    • Users should always see data that they have submitted themselves
  • Monotonic reads
    • After the users have seen data at one point in time
      • They should not see data from an earlier point in time
  • Consistent prefix reads
    • Users should see the data in a state that makes casual sense
      • For example, seeing a question and it’s reply in the correct order

Concurrency issues:

These are inherent in multi leader and leaderless replication… 🤷‍♂️

Final author recommendations

This concludes the end of the blog series on replication in data intensive systems. To note, most of this information can be found in “Designing Data Intensive Applications” by Martin Kleppmann, which is in my opinion a highly recommend and extensive book on this subject area.

What Are Consistent Prefix Reads?

Sean Mayer2 Comments

First of all the problem case…

Replication lag… imagine two people talking between each other. 🗣🗣

  • Person A 🗣 : “How far can you see into the future Person B?”
  • Person B 🗣 : “About 10 seconds”

🤔 There can be times where this may appear in a different different order from an observer’s point of view, or too fast to make much sense…

Consistent Prefix Reads to the rescue?

This guarantee of Consistent Prefix Reads can resolve this problem case.

  • If a sequence of writes happen in a certain order, anyone reading those writes can see them appear in the same order…
  • 👀 Look out for this problem in partitioned (sharded) databases
  • If database always applies writes in a certain order, reads can always see a consistent prefix… so this particular anomaly cannot happen

How about distributed databases with different partitions?

In this situation there is no global ordering of writes…

  • 👎 When a user reads from a database they may see older part of a database and some in a newer state
    • 👉 One solution is to make sure any writes that are causality related to each other are written to the same partition
  • 👎 Although in some applications, this cannot be done efficiently
    • 👉 There are also algorithms that explicitly keep track of causal dependencies

Eventual Consistency vs Strong Consistency

Sean Mayer1 Comment

Eventual Consistency

What is it?

  • Theoretical guarantee
  • If no new updates to an entity are made
    • All reads of the entity will eventually return the last updated value

Example of an eventual consistency model?

  • Internet Domain Name System (DNS)!
  • DNS servers are cached and replicated across directories over the internet
    • Time is needed to replicated modified changes to all DNS clients and servers
  • DNS is the foundations of the internet so it been a very successful system
  • It is has high availability and scalable
    • This has enabled name lookups for any device connected to the internet

Warning

  • Whilst availability is high on reads… replicas may be inconsistent with the latest write on the originating node

Strong Consistency

What is it?

  • Traditional relational databases have been designed based on this concept of strong consitency
  • Also called immediate consistency
  • Means that data is viewed immediately after an update
    • Therefore, all observers of the entity will be consistent

Trade offs

  • To have strong consistency… scalability and performance are compromised!
  • Data would required locking during the period of updates or replication process (replication lag)
    • To ensure that no other processes are updating on the same data

How to Read Your Own Writes?

Sean Mayer2 Comments

Reading after write consistency… aka (read your writes consistency)

What does this mean?

  • Guarantees when a user reloads a page they can see any updates they done themselves
    • Although makes no promises about other users
  • Other user updates maybe available at some other later time
  • This is achieved reassuring the user that there input has been inserted in correctly

How do we implement read after write consistency?

There are numerous number of techniques, here are a few:

Example 1

  • When a user is reading something, it may have modified and read it from the leader
  • Otherwise they have read it from a follower…
    • This way you know if something may have been modified without querying it 🤔

Example, social media website…

  • 👉 The user of the profile is only able to edit there profile
  • 👉 Thus a simple rule, the user should always read there profile from the leader and any other user profiles from a follower

⚠️ If most things are editable by the users this is not an affective approach! 👎

  • As most things are read by the leader node, therefore negating the benefits of read scaling 👎

Example 2

In the case above an alternative maybe used to decided a read from a leader.

  • For example, you can track the time of the last update and for one minute after the last update make all reads from the leader
  • You can also monitor the Replication Lag on the followers and prevent queries on any follower that is any more than one minute behind the leader ✅

Example 3

Another technique is to have the client remember the timestamp of the most recent write.

  • Then the system can ensure the replica serving any reads for that user reflects updates at least until that time stamp ✅
  • If a replicas is not sufficiently up to date then either read can be handled by another replica, or query can wait until replica is up to date
  • Timestamps can be logical timestamp, something that indicates ordering of writee such as the log sequence number or the actual system clock which means clock synchronisation becomes critical! 🕗

Multi data centre considerations

Keep in mind your replicas are distributed across multiple data centres for geographical proximity to users, or for availability… there is additional complexity! ⚠️

Another complication can occur, when a user is accessing the same service from multiple devices! ⚠️

  • For example, a desktop and a mobile
    • If a user enters information on one device, the user should be able to see this on another device 🤷‍♂️
      • In this case there some additional issues to consider
        • ❌ Approaches that require remembering the timestamp of the last users update become more difficult (example 3) ⚠️
        • 👉 Because the code on one device does not know what happened on the other device…
        • 👉 This metadata will need to be centralised
  • If replicas are distributed across different data centres
    • This guarantee the connection will be routed to the same data centre
      • Example: a user uses the home broadband connection
        • The other device uses cellular data network, and the device network routes maybe completely different! 🤪

👉If your approach requires reading from the leader you will first need to route all the user devices requests to the same data centre.

What is Replication Lag?

Sean Mayer4 Comments

Replication lag.. why bother with replication?

Reasons why we use this:

  • Fault tolerance, from node failures
  • Scalability of nodes based on requests
  • Latency, placing nodes geographically closer to users

How can replication lag occur with a read scaling architecture?

Let walk through a common replication pattern with leader based replication:

  • Requirements
    • Writes go through a single node 🔵
    • Read only queries go to any replica 🟢 🟢 🟢

When does this work?

In the example above, it would require a higher perentage of reads and a lesser amount of writes for this pattern to be attractive, as changing replica conditions due to writes are minimal, meaning delivery of read transactions are more consistent. ✅

Asynchronous vs Synchronous replication:

  • Asynchronous replication only really works when wanting to add more follower nodes ✅
  • Synchronous replication with a node outage can cause an entire system to lag or create down time ❌

Followers can fall behind although temporary state as they will catch up eventually… This is called eventual consistency.

Eventual consistency

Eventually is deliberately vague as there is no limit to how far a node can fall behind:

  • Maybe a fraction of second (unnoticeable) 🤷‍♂️
  • If there is lag in the entire system this can easily become several seconds to several minutes 🕙
  • When lags are so large, it is not just theoretically an issue, this can cause real problems for applications ❌

Problem of replication lag?

Reading your own writes…

  • Many applications let you submit data and let other users view it
    • This maybe a record in a customer database, comment in a forum etc
  • Asynchronous replication will mean some nodes maybe not up to date 🤔
    • So if a user is submitting changes to a leader
      • They may not see this on the follower they viewing from… this can cause distress to a user ❌
      • 👉 Especially if it involves depositing your own money to another local account and not seeing the changes immediately being transferred 💸

How to handle replication lag?

When working with an eventually consistent system, it is worth considering the application behaviour if there are replication lags of several minutes or hours:

  • If the answer is no problem… then great!
  • But if the result is a poor experience for users 👎
    • It is important to provide a stronger guarantee 💪
      • Like read after write (in the next blog post I will talk about this deeper)

❌ Pretending 🎭 an application is synchronous, when it is asynchronous is a concoction of problems later down the line. 🪲🦟

Potential solution

There are ways to provide a stronger guarantee that the underlying database…

  • By performing certain types of reads on a leader 🤔
    • However this is complex to do do on the application layer 👎

It would be better for some developer not needing to worry about replication issues, instead they can just just the database is doing the right thing. 🤷‍♂️

  • This is why transactions exist
    • They are a way for a database to provide strong guarantees so that the application can be simpler ✅
  • Single node transactions have existed for a long time
    • However, the move to distributed, replicated and partitioned databases many system have abandoned them 🤷‍♂️
      • Claiming that transactions are too expensive ❌
      • 👉 And asserting eventual consistency is inevitable in a scalable system
        • There are some truths on that statement, but this is overly simplistic, there are many more nuances out there