The Two Sides of Data Partitioning

Boltu set his teacup on the table and said, turning serious, "See, Montu, nothing in this world is free. All that effort you'll put into chopping the database into pieces, it has some magical upsides, sure, but it comes with its share of pain too. Come on, let's look at the good side first."

Boltu started counting on his fingers:

• Rocket-like speed (Query Performance): "Since your giant tables get small after partitioning, you no longer have to dig through the whole database to find any piece of data. Isn't finding a book on a small shelf easier than searching the entire library? So query performance shoots up like a rocket."
• Load balancing and scaling: "Earlier, one server took the full brunt of the storm. Now, with the data spread across separate servers, the load gets balanced. And as users grow, you can buy a new server and drop a new shard on it whenever you want. Scaling becomes a piece of cake!"
• Parallel processing: "Say you want to run a Map function over all your data for analytics. Now you can run the process in parallel across all the shards at once. The work finishes in the blink of an eye."
• Limiting the damage (Blast Radius & Recovery): "The biggest upside is, all your eggs aren't in one basket! If a server crashes for some reason, only the users on that server hit a temporary problem, the whole of BiralTube doesn't go down. And since the data size is small, backing up or recovering takes less time too."

Montu, beaming, said, "Boltu, that sounds perfect! I'm off to buy some servers right now."

— "Hey, hold on! Don't get ahead of yourself. There's a dark side too, hear it out first and then get to work."

Montu sat back down quietly. Now Boltu opened the ledger of downsides:

• Operational Complexity: "Managing one database and managing 10 pieces are not the same thing. Which data is where, whether backups are happening right, re-balancing data when you add a new server, all of it is a ton of hassle."
• Cross-partition Queries: "The biggest headache you'll hit is with Joins. Say a query needs to combine data from both 'Shard 1' and 'Shard 3'. Pulling data across the network from separate servers and joining it cranks up the latency so much that performance goes straight down the drain."
• Data Skew or Hotspots (Data Skew): "Partition with the wrong logic and you'll find one shard buried under huge load while another twiddles its thumbs. Say some cute cat video goes viral. Whichever server that video's data lives on, the whole world piles onto it. That's the hotspot problem."
• Migration pain: "If you ever want to change your database technology later, migrating this distributed, chopped-up database is an absolute nightmare."

— "Bottom line is," Boltu finished, "no matter how much hassle it is, if you want to survive at scale, you've got to put up with this grind. Your system's load is climbing, so don't wait, get started fast."

Montu, having understood it all, headed home happy. He got to work. For days on end, staying up late and running on coffee, he put in some back-breaking effort and partitioned the database beautifully. BiralTube's database is no longer a single server, it's a powerful distributed cluster of many servers!