Classical Machine Learning
Linear and logistic regression, nearest neighbours, kernels and RKHS, support vector machines, trees and ensembles.
Level 1 — Core
- Linear, Ridge and Lasso Regression12 min
The normal equations as orthogonal projection, the exact bias and variance of ridge, and why the unbiased estimator is rarely the best one.
Normal equations and the projection interpretationGauss–Markov theoremRidge regression in closed form and in the SVD basis - Logistic Regression6 min
The log-likelihood, its convexity, the Hessian as a weighted Gram matrix, and what happens when the data is separable.
The logistic negative log-likelihood and its derivativesThe logistic loss is convex, and strictly convex iff the design has full rankThe maximum likelihood estimate does not exist under separability
Level 2 — Advanced
- Nearest Neighbours and the Curse of Dimensionality8 min
Consistency of k-NN, the exact rate under smoothness, and the concentration argument that explains why it fails in high dimension.
Cover–Hart inequality for 1-NNUniversal consistency of k-NNNonparametric rate for k-NN regression - Trees, Bagging and Boosting14 min
Impurity and greedy splitting, the exact variance algebra behind bagging, and boosting derived as functional gradient descent.
Any concave impurity gives non-negative information gainBagging variance algebraAdaBoost is coordinate descent on the exponential loss
Level 3 — Graduate
- Kernels, RKHS and the Representer Theorem13 min
Positive definite kernels, the reproducing property, a complete proof of the representer theorem, and kernel ridge regression.
Positive definite kernelA kernel is positive definite iff it is an inner product of featuresConstruction of the reproducing kernel Hilbert space - Support Vector Machines — Complete Derivation23 min
Hard and soft margin, the Lagrangian, every KKT condition interpreted, both duals derived in full, the kernel trick, and the margin generalisation bound.
The geometric marginThe hard-margin primal is a well-posed convex programThe hard-margin SVM dual
Problems (32)
Open in the browser| Solved | Problem | Type | Difficulty | Time |
|---|---|---|---|---|
| Least squares as orthogonal projection | Proof | 15 min | ||
| The geometric margin | Proof | 15 min | ||
| Convexity of the logistic loss | Proof | 15 min | ||
| Simulate distance contrast run | Coding | 30 min | ||
| Solve a two-point SVM by hand | Numerical | 30 min | ||
| Two different curses of dimensionality | Numerical | 15 min | ||
| Concave impurity gives non-negative information gain | Proof | 30 min | ||
| Distance contrast and the failure of nearest neighbours | Proof | 30 min | ||
| Gradient boosting fits the negative functional gradient | Proof | 30 min | ||
| Implement kernel ridge regression run | Coding | 1 hour | ||
| Implement ridge and find the optimal lambda run | Coding | 1 hour | ||
| In-sample calibration of logistic regression | Proof | 30 min |