Rotary embeddings encode relative position exactly

ProofAdvanced⏱ 30 minDeep Learningroperelative positionrotationextrapolation

Theory: The Mathematics of Transformers →

Problem

Prove that $\langle R_pq, R_{p'}k\rangle = \langle q, R_{p'-p}k\rangle$ for the block-diagonal rotation $R_p$ of RoPE, and contrast with additive sinusoidal encodings, where relative position is only linearly recoverable.

Write a complete argument. Every inequality you use should be one you could state precisely.

Solution

Opening this before you have a real attempt on paper is the fastest way to feel like you understand something you cannot reproduce.

Related problems

All problems

Problem	Field	Type	Difficulty	Time
When does attention dominate the MLP?	Deep Learning	Numerical	Beginner	30 min
Attention is permutation equivariant	Deep Learning	Proof	Intermediate	30 min
Implement multi-head attention from scratch run	Deep Learning	Coding	Intermediate	3 hours
The softmax Jacobian as a variance, and saturation	Deep Learning	Proof	Intermediate	30 min
Why attention divides by the square root of d_k	Deep Learning	Proof	Intermediate	30 min