-
Continue reading →: Eventual Consistency vs Strong ConsistencyEventual Consistency What is it? Theoretical guarantee If no new updates to an entity are made All reads of the entity will eventually return the last updated value Example of an eventual consistency model? Internet Domain Name System (DNS)! DNS servers are cached and replicated across directories over the internet…
-
Continue reading →: How to Read Your Own Writes?Reading after write consistency… aka (read your writes consistency) What does this mean? How do we implement read after write consistency? There are numerous number of techniques, here are a few: Example 1 Example, social media website… ⚠️ If most things are editable by the users this is not an…
-
Continue reading →: What is Replication Lag?Replication lag.. why bother with replication? Reasons why we use this: How can replication lag occur with a read scaling architecture? Let walk through a common replication pattern with leader based replication: When does this work? In the example above, it would require a higher perentage of reads and a…
-
Continue reading →: What is Trigger-Based Replication?The replication process typically is implemented by the database system. No application code required. ✅ (Many cases that is what we want) Although.. in some circumstances replication can been needed to perform: You may ay require moving replication to the application layer such as toolset: Alternative, features that are available…
-
Continue reading →: What is Logical Log Replication?Logical replication consists of a process of replicating data objects and their changes. Key characteristics as followed: “Logical replication sends row-by-row changes, physical replication sends disk block changes. Logical replication is better for some tasks, physical replication for others.” https://stackoverflow.com/questions/33621906/difference-between-stream-replication-and-logical-replication How does logical replication work? Logical replication uses a publish…
-
Continue reading →: What is a Write-ahead Logging (WAL)?Key-value store store is a fundamental component, which is gaining exponential demand in a multitude of horizontally scaling environments, including: Example of features of key value storage engine: Write-ahead Logging (WAL) is used in storage engines to provide transactions with: Storage engines and WAL Log statements are compacted appended and…
-
Continue reading →: Replication Logs – What is Statement-Based Replication?In this blog post I will be covering one approach to implementing replication logs… statement-based replication. How does leader based replication work under the hood? 🤔 Several different replication methods are used in practice (due to the multitude of edge cases!) Final note However, as there are so many edge…
-
Continue reading →: What are Version Vectors? – AlgorithmExpanding upon the previous blog post regarding Capturing the happens before relationship, the scenario that is described only uses a single replica… The question is, how does the algorithm change when there are multiple replicas but no leader? (leaderless replication) A collection of version numbers from all the replicas is…
-
Continue reading →: Understanding the “Happens-Before” Relationship in Distributed SystemsIn this post, we will delve into an algorithm that can tell whether two operations are concurrent or whether one happened before another. Scenario Let’s begin with a database with only one replica. (so we can simplify this and thereafter generalise the approach to a leaderless database with multiple replicas)…
-
Continue reading →: How to Define a Concurrent Operation?Let’s begin with understanding, on how do we decide whether two operations are concurrent or not? Consider this scenario: The two writes are not concurrent… On the other hand: How to define concurrency? Whether one operation happens before another operation is the key to defining what concurrency means… In fact…
-
Continue reading →: What to consider with Replication and Multi-Data Centre Operations?Leaderless replication is suitable for multi-data centre operations, due its ability to tolerate conflicts, such as: For instance both Cassandra and Voldemort implement there multi data centre support within the normal leaderless model. Detecting concurrent writes Dynamo styled databases allow for several clients to concurrently write to the same key.…
-
Continue reading →: What are Sloppy Quorums and Hinted Handoffs?Why optimised quorums useful? Databases with appropriately configured quorums can tolerate the failure of individual nodes, this can allow systems to: These characteristics make leaderless replication appealing for use cases: “Achilles heel” of Quorums! Quorums can provide some reliable outcomes for end users, they are not fault tolerant… Network interruptions…
-
Continue reading →: How to Monitor StalenessWhat is Staleness? Describing staleness in an operational and monitoring perspective, this means understanding whether your databases are returning up to date results. The how to Manage Staleness? Even if your application can tolerate stale reads, you need to: Leader based replication The database typically exposes metrics for the replication…
-
Continue reading →: What are the Limitations of Quorum Consistency?Quorum Variables (before diving into this subject) (W + R) > N If you have N replicas and you choose W and R such as (W+R) > N you will generally expect every read to return the most recent value written. Pros Cons (W + R) < N But Quorums…
-
Continue reading →: How do Reading and Writing Quorums work?Illustrate the Quorum Variables Summary of Quorum Reads and writes that obey this R 📖 and W ✍️ values are called Quorum reads and writes. In a Dynamo style databases: The Quorum Condition (W+R) > N The (W+R) > N allows us to tolerate theses nodes with these types of…
-
Continue reading →: Leaderless Replication – Writing to a database when a node is downIf you need a refresher on leaderless replication please read my previous blog post on: What is Leaderless replication? Or read Martin Kleppman book on ‘Designing Data-Intensive Applications’. Scenario of a node going down Conceptualise a situation where you have a database with 3 replicas and 1 of the replicas…
-
Continue reading →: What is Leaderless Replication?Quickie on multi leader and single leader replication Typically a single or multi leader replication approach is adopted, this is based on the concept: Although.. some data storage systems can take another approach… The leaderless replication set up As the name entails this means abandoning the concept of a leader.…
-
Continue reading →: The Multi-Leader Replication TopologiesWhat is a replication topology? This outlines the communication paths, where writes are propagated to one node or the other. You only have two leaders? There is only one suitable topology… With more that two leaders various topologies are possible The most general topology is all to all, every leader…
-
Continue reading →: What is a Conflict in Replication?The simple question on what is a conflict? Pronounced types of conflicts There are some types conflicts that are obvious. For instance, the wiki example: Understated types of conflicts Whilst there are clear conflict we can understand, there are types of conflicts that we may not consider. For example, a…
-
Continue reading →: How to Resolve Conflicts with Automatic Conflict ResolutionConflict resolution rules can become complicated and custom code can be prone to errors. Effects of conflict resolution handler 🛒 Example: Amazon has had some interesting effects when utilising a conflict resolution handler. It has been documented that for some time the conflict resolution logic on the shopping cart preserves…
Hello,
I’m Sean
Welcome to the Scalable Human blog. Just a software engineer writing about algo trading, AI, and books. I learn in public, use AI tools extensively, and share what works. Educational purposes only – not financial advice.
Let’s connect
Join the mailing list
Stay updated with latest posts.
Scalable Human Blog 