-
Continue reading →: Replication vs Partitioning vs Clustering vs Sharding (1 minute read)In the complex world of database management, terms like replication, partitioning, clustering, and sharding are often interchanged, leading to confusion. As someone who has delved into these topics for over a year, I believe it’s crucial to demystify these concepts. Replication: The Art of Duplication Replication involves duplicating tables or…
-
Continue reading →: What is Partitioning and Why is it Important?In my previous blogs published, I have covered the different ways of which how partitioning is managed with large datasets and thereby reduced to smaller subsets. All this content was derived from the author Martin Kleppman in “Designing Data Intensive Applications“. In this post I will summarise all the topics…
-
Continue reading →: Parallel Query Execution – What is it?To begin as Martin Kleppman from Designing Data Intensive Applications outlines, there are simple queries that read and write a single key. Plus there are scatter gather queries, in the case of document partition secondary indexes. Massive Parallel Processing (MPP) This is about the level of access supported by most…
-
Continue reading →: How does Partitioning work when Requests are being Routed?Let depict a situation where we have now partitioned our dataset running across multiple nodes and multiple machine… There is a question here… When a client wants to make a request, how does it know which node to connect to? 🤔 As partitions are rebalanced, the assignment of partitions to…
-
Continue reading →: How to decide between Automatic and Manual Rebalancing in Operations? (Partitions)There is a critical question to be raised on deciding whether we choose automatic or manual rebalancing… “There is a gradient between fully automatic rebalancing, in which the system decides automatically when to move partitions to one node to another without any administrator interaction, and fully manual rebalancing, in which…
-
Continue reading →: Understanding Partitioning Proportional to NodesWith dynamic partitioning the number of partitions is proportional to the size of the dataset. Since splitting and merging processes, this keeps the size of each partition between some fixed number of partitions The size of each partition is proportional to the size of the dataset ✅ For both of…
-
Continue reading →: What is Dynamic Partitioning?For database that use key range partitioning, a fixed number of partitions with fixed boundaries would be very inconvenient! You could end up with all the data in one partition and all the other partitions empty 👎 Reconfiguring the partition boundaries manually would be very tedious👎 For that reason… Key…
-
Continue reading →: What is Fixed Partitioning?There are few ways to assigning partitions to nodes, in this post we will discuss fixed partitioning. How not to rebalance partitions! When partitioning by the hash of a key… it is best to divide the possible hashes into ranges and assign each range to a partition… Why don’t we…
-
Continue reading →: Rebalancing Partitions – What is it?Overtime things change in a database… For example ↘️ So you want to add more CPU to handle the load ⚡️ So you want to add more discs and RAM to store it ⚡️ Other machine needs to take over the failed machine responsibilities ⚡️ All of these changes call…
-
Continue reading →: Partitioning Secondary Indexes by Term – What is it?Rather than each partition having its own secondary index (local index). We can construct a global index that covers data in all partitions. 👉 However, we cannot just store that index on just one node! 👉 Since it will likely bottleneck and defeat the purpose of partitioning! Global Index and…
-
Continue reading →: How to Partition with Secondary Indexes by DocumentConceptualise a situation where you are operating a website for used cars. 🚗 🏎 🚕 🚓 🚘 🚖 🚔 🚙 Each listing has a unique ID (call it the document ID) You partition the database by document ID IDs 0-499 = partition 0 IDs 500-999 = partition 1 And so…
-
Continue reading →: What to Consider with Secondary Indexes and PartitioningThe partitioning schemes that have been covered in the previous blogs, rely on a key value data model. If records are only ever accessed via a primary key… We can determine the partition from that key ✅ And… use to route read/write requests to partition responsible for that key 👍…
-
Continue reading →: How to Relieve Hotspots with Skewed WorkloadsAs discussed in the previous post What is Consistent Hashing?, hashing a key to determine its partition can aid in reducing hotspots… However they cannot be avoided entirely… In some extreme cases where all reads and writes offer the same key ⚠️ You still end up with all requests being routed…
-
Continue reading →: What is Consistent Hashing?As defined by the author of Consistent Hashing and Random Trees, Distributed hashing protocols for Relieving Hotspots on the World Wide Web, consistent hashing is a way of evenly distributing load across an internet wide system of caches. Example of this? 👉 Content delivery network (CDN) “A content delivery network…
-
Continue reading →: What is Partitioning by Hash of Key?As discussed in the previous blogs: What is Partitioning By Key Range? What is Partitioning of Key-Value Data? These posts identify the risk of skew and hotspots. ⚠️ Many distributed data stores use a hash function to determine the partition for a given key. Say you have a 32 bit hash function…
-
Continue reading →: What is Partitioning By Key Range?One method of partitioning is to assign a continuous range of keys, for some minimum to some maximum to each partition. You can easily determine which partition contains a given key Then you can make your request to the appropriate node Or in the case of the encyclopaedia Pick the…
-
Continue reading →: What is Partitioning of Key-Value Data?Say you have a large amount of data, and you want to partition it… How do you decide which records to store on which nodes? 🤔 In theory 10 nodes should be able to 10x as much data the load (read write throughput) is currently handling on one node (ignoring…
-
Continue reading →: What is Partitioning?In the previous blogs I have been discussing replication, where we have multiple copies of the same data on different nodes… Although, for very large datasets, or very high query throughput this is not sufficient! 👎 We require something that can divide the data up into partitions… aka sharding (previously…
-
Continue reading →: What is Replication and Why is it Important?Lets firstly breakdown the goals of replication Maintaining that a system is running at an agreed level of operation. Even when one machine or several machines or an entire data centre goes down! Enabling the application to continue working when there is network latency Placing data geographically close to users…
-
Continue reading →: What Are Consistent Prefix Reads?First of all the problem case… Replication lag… imagine two people talking between each other. 🗣🗣 Person A 🗣 : “How far can you see into the future Person B?” Person B 🗣 : “About 10 seconds” 🤔 There can be times where this may appear in a different different…
Hello,
I’m Sean
Welcome to the Scalable Human blog. I’m a software engineer who likes building things, from software and AI tools to robots, Raspberry Pi projects, smart-home systems and energy monitoring. I learn in public, experiment with technology and share what I discover along the way.
Let’s connect
Join the mailing list
Stay updated with latest posts.
