IRLS is a weighted least-squares solve
ProofIntermediate⏱ 30 minClassical MLirlsnewtonweighted least squares
Problem
Show that the Newton step for logistic regression can be written as the solution of a weighted least-squares problem, and identify the working response. Explain why this gives fast convergence and when it fails.
Write a complete argument. Every inequality you use should be one you could state precisely.
Solution
Opening this before you have a real attempt on paper is the fastest way to feel like you understand something you cannot reproduce.
Related problems
All problems| Solved | Problem | Field | Type | Difficulty | Time |
|---|---|---|---|---|---|
| Convexity of the logistic loss | Classical ML | Proof | 15 min | ||
| In-sample calibration of logistic regression | Classical ML | Proof | 30 min | ||
| Logistic regression diverges on separable data | Classical ML | Proof | 30 min | ||
| Newton's method is affine invariant | Advanced Optimization | Proof | 30 min | ||
| Local quadratic convergence of Newton's method | Advanced Optimization | Proof | 1 hour |