RoPE encodes position: multiply Q,K by rotation matrix R(θ_i) at each position

How does a robot arm rotate smoothly?

Imagine you're programming a robot arm to paint a circular mural. The arm needs to rotate smoothly to paint the circle perfectly.

Think of the arm's movement as a point on a piece of paper. To rotate this point smoothly, we use a special math trick called a rotation matrix. This matrix helps us rotate the point by a certain angle, making the arm move in a circle.

Example

If the arm's starting position is (1, 0) and it needs to rotate 90 degrees, the rotation matrix R(90°) = [ [0, -1], [1, 0] ] will transform the position to (0, 1).

Remember this

The rotation matrix R(θ_i) rotates the robot arm's position smoothly, ensuring it paints a perfect circle.

Text adapted from Wikipedia, licensed under CC BY-SA 4.0.

Related concepts

rotary position embeddings (RoPE) do

RoPE encodes relative position by applying rotation matrices to input features

sinusoidal position encoding works: each dimension has a different frequency

Sinusoidal position encoding assigns unique frequencies to each dimension, enabling the model to distinguish positions effectively

RoPE's advantage is: supports length extrapolation beyond training context length

RoPE (Relative Position Encoding) advantage: supports length extrapolation beyond training context length

the L1 norm is not differentiable at zero

Absolute value's kink makes it non-differentiable at zero

Shor's algorithm

Shor's algorithm factors integers in polynomial time on a quantum computer

Matrix multiplication algorithm

Ever wondered how computers speed up multiplying huge numbers?

Swipe through 100 ML concepts daily

Open Pocket Polymath