Why can't you just split a huge library into smaller ones?

Image: Software: Red Hat, Inc. and Podman communityScreenshot: VulcanSphere, Apache License 2.0, via Wikimedia Commons

database sharding does: splits data across machines by a partition key

Why can't you just split a huge library into smaller ones?

Imagine you have a huge library with thousands of books. Finding a specific book takes forever because the library is too big and unwieldy.

Splitting the library into smaller sections, each with a few books, makes it easier to find what you're looking for. This is called "sharding."

Example

If the library has 10,000 books, you could split it into 10 sections of 1,000 books each.

Remember this

Sharding helps manage and access data more efficiently by dividing it into smaller, more manageable parts.

Text adapted from Wikipedia, licensed under CC BY-SA 4.0.

Related concepts

consistent hashing solves: minimizes key redistribution when servers are added/removed

Consistent hashing minimizes key redistribution when servers are added/removed

consistent hashing does: minimizes remapping when nodes join/leave

How can we efficiently share resources without constant reorganization?

Load balancing (computing)

Load balancing distributes tasks efficiently across resources

B-trees optimize: disk-based sorted data with O(log n) reads per query

B-trees optimize disk-based sorted data with O(log n) reads per query

fused kernels do

Why wait for a computer to finish one task before starting another?

load balancing loss is needed in MoE

Can one expert handle all tasks perfectly?

Swipe through 100 ML concepts daily