Logistic regression diverges on separable data

ProofIntermediate⏱ 30 minClassical MLseparabilitydivergenceimplicit biasmax margin

Problem

Show that if the data is linearly separable the logistic likelihood has no maximiser and $\|w_t\|\to\infty$ . Then state (without proving) what is known about the direction of divergence, and explain why that is a rigorous instance of implicit bias.

Write a complete argument. Every inequality you use should be one you could state precisely.

Solution

Opening this before you have a real attempt on paper is the fastest way to feel like you understand something you cannot reproduce.

Related problems

All problems

Problem	Field	Type	Difficulty	Time
XOR is not linearly separable	Deep Learning	Proof	Foundation	15 min
Convexity of the logistic loss	Classical ML	Proof	Beginner	15 min
The exact step-size threshold on a quadratic	Optimization	Proof	Beginner	15 min
The proximal operator of the l1 norm	Advanced Optimization	Proof	Beginner	15 min
Gradient descent finds the minimum-norm interpolator	Learning Theory	Proof	Intermediate	30 min