cosine similarity is preferred over dot product for normalized embeddings

Why do we need a special way to measure similarity in high-dimensional spaces?

Imagine you're trying to find the best match for a friend on a dating app. You want to compare people's interests and personality traits, which are represented as vectors in a high-dimensional space.

In a high-dimensional space, straightforward distance measures can be misleading. Cosine similarity helps us focus on the direction of the vectors, ignoring their magnitude, which is more meaningful for normalized embeddings.

Example

If two people have interests represented as vectors (3, 4) and (6, 8), the dot product suggests they're very similar. However, if we normalize these vectors, cosine similarity shows they're actually less similar because the direction of their interests is different.

Remember this

Cosine similarity measures orientation, not magnitude, making it better for comparing normalized vectors in high-dimensional spaces.

Text adapted from Wikipedia, licensed under CC BY-SA 4.0.

Related concepts

List of algorithms

Cosine similarity measures the angle between vectors, not their magnitude

cosine similarity works better than Euclidean distance in high dimensions

Cosine similarity measures orientation, not magnitude, making it more robust to irrelevant dimensions in high-dimensional spaces

mean pooling often outperforms [CLS] for sentence similarity tasks

Mean pooling captures overall sentence meaning better than [CLS] token embedding

Matryoshka embeddings are: trained to be useful at multiple truncated dimensions

Ever wondered how a simple doll can teach us about nested complexities?

ALiBi allows length extrapolation better than learned position embeddings

ALiBi uses relative positional encoding, avoiding fixed-size embeddings, enabling better handling of variable-length sequences

[CLS] pooling does: uses the first token's embedding as the sentence representation

CLS pooling: uses the first token's embedding as the sentence representation

Swipe through 100 ML concepts daily

Open Pocket Polymath