Scalable Human Blog

There is a critical question to be raised on deciding whether we choose automatic or manual rebalancing…

“There is a gradient between fully automatic rebalancing, in which the system decides automatically when to move partitions to one node to another without any administrator interaction, and fully manual rebalancing, in which the assignment of partitions to nodes are explicitly configured by an administrator, and only changes when a administrator explicitly reconfigures it.”

Martin Kleppmann from Designing Data Intensive Applications

For example, CouchBase, Riak, Voldermort:

• 👉 Generate a suggested partition assignment automatically
• 👉 But require an administrator to commit it before it takes affect!

Fully automated partitioning can be convenient…

• Because there is less operational work to do for maintenance 👍
• However, it can be unpredictable! 👎

Rebalancing is an expensive operation…

This is due to rerouting requests and moving a large amount of data from one node to another.

If this is not done carefully:

• This process can overload the network or the nodes
• And harm the performance of other requests while the rebalancing is in progress 😔

Automation can be dangerous!

Such automation can be dangerous with combination of automatic failure detection 😵‍💫 🤖

For instance…

1. Say one node is overloaded 🔴❗️
2. Temporary slow to respond to requests ➡️ ➡️ 🔴 ❌
3. The other nodes conclude that the other node is dead 🤷‍♂️
4. And automatically rebalancing the cluster to move load away from it 🤷‍♂️
5. This puts additional load on the overloaded node 🤯
6. Other nodes and the network making the situation worse and potentially causing a cascading failure 🤯
7. For that reason it can be a good thing to have a human in the loop for rebalancing! 👨‍💻
   • Its slower than a fully automatic process 🐢
   • But can help operation surprises 🤗

📚 Further Reading & Related Topics

If you’re exploring how to decide between automatic and manual rebalancing in operations and partitions, these related articles will provide deeper insights:

• Understanding Partitioning: Proportional to Nodes – Learn how partitioning strategies impact rebalancing decisions and how proportional partitioning helps in managing load distribution automatically.

• How Does Partitioning Work When Requests Are Being Routed? – Explore the intricacies of partitioning and request routing in distributed systems, and how the decision between automatic and manual rebalancing can affect system performance.