Bounding how far the adapted model moves

ProofIntermediate⏱ 30 minFine-Tuninglipschitzdeviationweight decay

Problem

Prove $\|\frac{\alpha}{r}BAx\|_2 \le \frac{\alpha}{r}\|B\|_2\|A\|_2\|x\|_2$ and state when it is tight.
Show that weight decay with coefficient $\mu$ on both factors bounds $\|B\|_2\|A\|_2$ in terms of the achieved training objective.
Explain why this does not bound the change in the network's output.

Write a complete argument. Every inequality you use should be one you could state precisely.

Opening this before you have a real attempt on paper is the fastest way to feel like you understand something you cannot reproduce.

Problem	Field	Type	Difficulty	Time
Lipschitz constants depend on the norm	Foundations	Numerical	Beginner	15 min
The supremum is 1-Lipschitz in the uniform norm	Foundations	Proof	Intermediate	15 min
Why AdamW differs from Adam with L2	SGD	Proof	Intermediate	30 min
Derive the EWC objective	Fine-Tuning	Proof	Advanced	1 hour
Normalisation makes weights scale-invariant	Deep Learning	Proof	Advanced	30 min