Ever wondered how a simple doll can teach us about nested complexities?
Image: Vassily Kandinsky by Adolf Elnain Photo credits : Georges Meguerditchian - Centre Pompidou, MNAM-CCI /Dist. RMN-GP Imag, Public domain, via Wikimedia Commons
Matryoshka embeddings are: trained to be useful at multiple truncated dimensions
Ever wondered how a simple doll can teach us about nested complexities?
Imagine you're trying to pack a Russian doll set for a trip, but you want to fit them all neatly in a suitcase without leaving too much empty space.
Think of each doll as a layer in a mathematical concept called a nested sequence, where each layer contains a smaller version of the previous one.
Example
If the largest doll is 10 inches tall and each subsequent doll is 90% the size of the one before, the third doll would be 7.29 inches tall (10 × 0.9 × 0.9).
Remember this
The concept of Matryoshka embeddings shows how systems can be designed to function effectively at multiple levels of complexity.
Text adapted from Wikipedia, licensed under CC BY-SA 4.0.
Related concepts
cosine similarity is preferred over dot product for normalized embeddings
Why do we need a special way to measure similarity in high-dimensional spaces?
weight tying does in language models: shares embedding and output projection matrices
Ever wonder how machines understand the sequence of words in a sentence?
random projection to O(log n/ε²) dimensions preserves pairwise distances within 1±ε
Can you shrink a 3D object into 2D without losing its shape?
ALiBi allows length extrapolation better than learned position embeddings
ALiBi uses relative positional encoding, avoiding fixed-size embeddings, enabling better handling of variable-length sequences
Ordinary least squares
OLS minimizes squared differences
Vector quantization
How can you store a huge library of books in a tiny closet?
Swipe through 100 ML concepts daily
Open Pocket Polymath