-
Continue reading →: What is Partitioning and Why is it Important?In my previous blogs published, I have covered the different ways of which how partitioning is managed with large datasets and thereby reduced to smaller subsets. All this content was derived from the author Martin Kleppman in “Designing Data Intensive Applications“. In this post I will summarise all the topics…
-
Continue reading →: Parallel Query Execution – What is it?To begin as Martin Kleppman from Designing Data Intensive Applications outlines, there are simple queries that read and write a single key. Plus there are scatter gather queries, in the case of document partition secondary indexes. Massive Parallel Processing (MPP) This is about the level of access supported by most…
-
Continue reading →: How does Partitioning work when Requests are being Routed?Let depict a situation where we have now partitioned our dataset running across multiple nodes and multiple machine… There is a question here… When a client wants to make a request, how does it know which node to connect to? 🤔 As partitions are rebalanced, the assignment of partitions to…
-
Continue reading →: How to decide between Automatic and Manual Rebalancing in Operations? (Partitions)There is a critical question to be raised on deciding whether we choose automatic or manual rebalancing… “There is a gradient between fully automatic rebalancing, in which the system decides automatically when to move partitions to one node to another without any administrator interaction, and fully manual rebalancing, in which…
-
Continue reading →: Understanding Partitioning Proportional to NodesWith dynamic partitioning the number of partitions is proportional to the size of the dataset. Since splitting and merging processes, this keeps the size of each partition between some fixed number of partitions But there is another option A third option is used by Cassandra… What happens when you increase…
-
Continue reading →: What is Dynamic Partitioning?For database that use key range partitioning, a fixed number of partitions with fixed boundaries would be very inconvenient! For that reason… Key range partition databases such as HBase and RethinkDB create partitions dynamically! 👍 Advantage of Dynamic Partitioning Is that the number of partitions adapts to the total data…
-
Continue reading →: What is Fixed Partitioning?There are few ways to assigning partitions to nodes, in this post we will discuss fixed partitioning. How not to rebalance partitions! When partitioning by the hash of a key… it is best to divide the possible hashes into ranges and assign each range to a partition… Why don’t we…
-
Continue reading →: Rebalancing Partitions – What is it?Overtime things change in a database… For example ↘️ All of these changes call for data and requests to be moved from one node to another… 🟢 ←→ 🟢 ←→ 🟢 ←→ 🟢 ←→ 🟢 ←→ 🟢 The process of moving load from one node in a cluster to another…
-
Continue reading →: Partitioning Secondary Indexes by Term – What is it?Rather than each partition having its own secondary index (local index). We can construct a global index that covers data in all partitions. Global Index and term partitioning? A global index must also be partitioned. But it can be partitioned differently from the primary key index. Here is how this…
-
Continue reading →: How to Partition with Secondary Indexes by DocumentConceptualise a situation where you are operating a website for used cars. 🚗 🏎 🚕 🚓 🚘 🚖 🚔 🚙 The need for the secondary index… You want to let users search for cars, allowing them to filter by colour and by make. Secondary index relationships with partitions In the…
-
Continue reading →: What to Consider with Secondary Indexes and PartitioningThe partitioning schemes that have been covered in the previous blogs, rely on a key value data model. If records are only ever accessed via a primary key… What are secondary indexes? The situation becomes more complicated when secondary indexes are involved… 🤯 “Secondary indexes are the bread and butter…
-
Continue reading →: How to Relieve Hotspots with Skewed WorkloadsAs discussed in the previous post What is Consistent Hashing?, hashing a key to determine its partition can aid in reducing hotspots… However they cannot be avoided entirely… In some extreme cases where all reads and writes offer the same key This scenario of workload is potentially unusual, although this is…
-
Continue reading →: What is Consistent Hashing?As defined by the author of Consistent Hashing and Random Trees, Distributed hashing protocols for Relieving Hotspots on the World Wide Web, consistent hashing is a way of evenly distributing load across an internet wide system of caches. Example of this? 👉 Content delivery network (CDN) “A content delivery network…
-
Continue reading →: What is Partitioning by Hash of Key?As discussed in the previous blogs: These posts identify the risk of skew and hotspots. ⚠️ Many distributed data stores use a hash function to determine the partition for a given key. What does it take for a hash function? For partitioning purposes the hash functions need not to be…
-
Continue reading →: What is Partitioning By Key Range?One method of partitioning is to assign a continuous range of keys, for some minimum to some maximum to each partition. How are keys arranged? The arrangement of keys are not necessary evenly spaced… How to distribute the data evenly? The partition boundaries need to adapt to the data OR……
-
Continue reading →: What is Partitioning of Key-Value Data?Say you have a large amount of data, and you want to partition it… How do you decide which records to store on which nodes? 🤔 Unfair partitioning (Skew) 👎 If the partitioning is unfair… So that more partitions have more data than others.. Avoiding hot spots The simplest approach…
-
Continue reading →: What is Partitioning?In the previous blogs I have been discussing replication, where we have multiple copies of the same data on different nodes… Although, for very large datasets, or very high query throughput this is not sufficient! 👎 We require something that can divide the data up into partitions… aka sharding (previously…
-
Continue reading →: What is Replication and Why is it Important?Lets firstly breakdown the goals of replication The humbling reality…🙏 Despite being a simplistic objective to keep copies on several machines, replication is remarkably a tricky problem! It requires careful attention to: At a minimum, we will generally require dealing with the following: And that is not even considering the…
-
Continue reading →: What Are Consistent Prefix Reads?First of all the problem case… Replication lag… imagine two people talking between each other. 🗣🗣 🤔 There can be times where this may appear in a different different order from an observer’s point of view, or too fast to make much sense… Consistent Prefix Reads to the rescue? This…
-
Continue reading →: What are Monotonic Reads?First of all the problem case… The problem: anomalies can occur when reading asynchronous followers! Monatonic Reads to the rescue? Well this method is a weaker guarantee than strong consistency, but a stronger guarantee than eventual consistency. For more readings on strong consistency vs eventual consistency please read my previous…
Hello,
I’m Sean
Welcome to the Scalable Human blog. Just a software engineer writing about algo trading, AI, and books. I learn in public, use AI tools extensively, and share what works. Educational purposes only – not financial advice.
Let’s connect
Join the mailing list
Stay updated with latest posts.
Scalable Human Blog 