List of algorithms

Cosine similarity measures the angle between vectors, not their magnitude

Cosine similarity is a metric used to measure the similarity between two vectors by comparing the direction of the vectors, not their magnitude. This is particularly useful in high-dimensional spaces where the magnitude of vectors can be very large and less informative.

Example

Consider two vectors A = [1, 2, 3] and B = [2, 4, 6]. The cosine similarity between A and B is calculated as the dot product of A and B divided by the product of their magnitudes. In this case, the dot product is 1*2 + 2*4 + 3*6 = 28, and the magnitudes are sqrt(1^2 + 2^2 + 3^2) = sqrt(14) and sqrt(2^2 + 4^2 + 6^2) = sqrt(56). The cosine similarity is 28 / (sqrt(14) * sqrt(56)) = 0.5.

Remember this

Cosine similarity is important because it focuses on the direction of vectors, making it useful for comparing the similarity of high-dimensional data, such as text embeddings or image features, where the magnitude of vectors can vary significantly.

Text adapted from Wikipedia, licensed under CC BY-SA 4.0.

Related concepts

cosine similarity works better than Euclidean distance in high dimensions

Cosine similarity measures orientation, not magnitude, making it more robust to irrelevant dimensions in high-dimensional spaces

cosine similarity is preferred over dot product for normalized embeddings

Why do we need a special way to measure similarity in high-dimensional spaces?

Cosine similarity

Cosine similarity formula: cos(θ) = (A · B) / (||A|| ||B||)

the dot product measures alignment: it equals |a||b|cos(θ)

Why do vectors sometimes "agree" with each other?

ALiBi allows length extrapolation better than learned position embeddings

ALiBi uses relative positional encoding, avoiding fixed-size embeddings, enabling better handling of variable-length sequences

Euclidean geometry

Euclidean distance measures absolute position in space

Swipe through 100 ML concepts daily

Open Pocket Polymath