ALiBi allows length extrapolation better than learned position embeddings

cosine similarity is preferred over dot product for normalized embeddings

Why do we need a special way to measure similarity in high-dimensional spaces?

rotary position embeddings (RoPE) do

RoPE encodes relative position by applying rotation matrices to input features

weight tying does in language models: shares embedding and output projection matrices

Ever wonder how machines understand the sequence of words in a sentence?

768-dim BERT embeddings capture: bidirectional context from masked language modeling

768-dim BERT embeddings capture bidirectional context from masked language modeling

List of algorithms

Cosine similarity measures the angle between vectors, not their magnitude

batch size affects generalization: larger batches find sharper minima

Larger batch sizes lead to sharper minima, enhancing generalization by providing more accurate gradient estimates