rotary position embeddings (RoPE) do

RoPE encodes position: multiply Q,K by rotation matrix R(θ_i) at each position

How does a robot arm rotate smoothly?

RoPE's advantage is: supports length extrapolation beyond training context length

RoPE (Relative Position Encoding) advantage: supports length extrapolation beyond training context length

ALiBi allows length extrapolation better than learned position embeddings

ALiBi uses relative positional encoding, avoiding fixed-size embeddings, enabling better handling of variable-length sequences

sinusoidal position encoding works: each dimension has a different frequency

Sinusoidal position encoding assigns unique frequencies to each dimension, enabling the model to distinguish positions effectively

List of algorithms

Cosine similarity measures the angle between vectors, not their magnitude

weight tying does in language models: shares embedding and output projection matrices

Ever wonder how machines understand the sequence of words in a sentence?