RoPE's advantage is: supports length extrapolation beyond training context length

rotary position embeddings (RoPE) do

RoPE encodes relative position by applying rotation matrices to input features

ALiBi allows length extrapolation better than learned position embeddings

ALiBi uses relative positional encoding, avoiding fixed-size embeddings, enabling better handling of variable-length sequences

loop unrolling does: trades code size for reduced loop overhead

Loop unrolling optimizes execution speed by reducing loop control instructions

RoPE encodes position: multiply Q,K by rotation matrix R(θ_i) at each position

How does a robot arm rotate smoothly?

Vanishing gradient problem

Residual connections help by allowing gradient flow through the skip connection

sinusoidal position encoding works: each dimension has a different frequency

Sinusoidal position encoding assigns unique frequencies to each dimension, enabling the model to distinguish positions effectively