Gradient boosting fits the negative functional gradient

ProofIntermediate⏱ 30 minClassical MLgradient boostingfunctional gradientresiduals

Problem

Show that gradient boosting's per-round target is $r_i = -\partial L(y_i,F_i)/\partial F_i$ , and compute $r_i$ for the squared, logistic, exponential and absolute losses. Explain the sense in which the base-learner fit is a projection.

Write a complete argument. Every inequality you use should be one you could state precisely.

Solution

Opening this before you have a real attempt on paper is the fastest way to feel like you understand something you cannot reproduce.

Related problems

All problems

Problem	Field	Type	Difficulty	Time
Concave impurity gives non-negative information gain	Classical ML	Proof	Intermediate	30 min
The bagging variance formula	Classical ML	Proof	Intermediate	30 min
AdaBoost as exact line search on the exponential loss	Classical ML	Proof	Advanced	30 min
AdaBoost drives training error to zero exponentially	Classical ML	Proof	Advanced	1 hour