RoPE's advantage is: supports length extrapolation beyond training context length

rotary position embeddings (RoPE) do

RoPE encodes relative position by applying rotation matrices to input features

ALiBi allows length extrapolation better than learned position embeddings

ALiBi uses relative positional encoding, avoiding fixed-size embeddings, enabling better handling of variable-length sequences

RoPE encodes position: multiply Q,K by rotation matrix R(θ_i) at each position

How does a robot arm rotate smoothly?

Vanishing gradient problem

Residual connections help by allowing gradient flow through the skip connection

sinusoidal position encoding works: each dimension has a different frequency

Sinusoidal position encoding assigns unique frequencies to each dimension, enabling the model to distinguish positions effectively

soft targets carry more information than hard labels: they encode class similarities

Why do some learning methods need to explore more than others?