Why can't you just split a huge library into smaller ones?

Image: Software: Red Hat, Inc. and Podman communityScreenshot: VulcanSphere, Apache License 2.0, via Wikimedia Commons

database sharding does: splits data across machines by a partition key

Why can't you just split a huge library into smaller ones?

Imagine you have a huge library with thousands of books. Finding a specific book takes forever because the library is too big and unwieldy.

Splitting the library into smaller sections, each with a few books, makes it easier to find what you're looking for. This is called "sharding."

Example

If the library has 10,000 books, you could split it into 10 sections of 1,000 books each.

Remember this

Sharding helps manage and access data more efficiently by dividing it into smaller, more manageable parts.

Text adapted from Wikipedia, licensed under CC BY-SA 4.0.

Related concepts

consistent hashing solves: minimizes key redistribution when servers are added/removed

Ever wonder why adding or removing servers doesn't mess up your favorite streaming service?

consistent hashing does: minimizes remapping when nodes join/leave

How can we efficiently share resources without constant reorganization?

Tensor network

Ever wondered how scientists manage massive data without endless storage?

Load balancing (computing)

Load balancing distributes tasks efficiently across resources

BPE tokenization does: iteratively merges the most frequent byte pairs

How can we shrink text files without losing meaning?

fused kernels do

Why wait for a computer to finish one task before starting another?

Swipe through 100 ML concepts daily