Language Modelling, Cross Entropy and Perplexity

For any distribution $p$ on sequences $x = (x_1,\dots,x_T)$ over a finite vocabulary,

where $x_{<t} = (x_1,\dots,x_{t-1})$ and the $t=1$ factor is the marginal $p(x_1)$.

Let $\hat p$ be the empirical distribution of the data. Then

where $H(\hat p) = -\sum_x\hat p(x)\log\hat p(x)$ does not depend on $\theta$. Hence minimising cross entropy is exactly minimising $\mathrm{KL}(\hat p\|q_\theta)$.

Suppose the data is drawn from a true distribution $p$. Then for any model $q$,

with equality if and only if $q = p$ almost everywhere.

Let a corpus have $C$ characters and be tokenised two ways, into $T_1$ and $T_2$ tokens. Two models assigning the same probability to the corpus have per-token losses

so their perplexities differ by the factor $\exp\big(-\log q\cdot(1/T_1-1/T_2)\big)$. Only the per-character (or per-byte) loss $-\log q(\text{corpus})/C$ is tokeniser-independent.

Let $z\in\R^{|\Vcal|}$ be the logits, $q = \operatorname{softmax}(z)$, and the loss $\ell = -\log q_{v^\star}$ for the true token $v^\star$. Then

and consequently $\dfrac{\partial\ell}{\partial W_U} = (q-e_{v^\star})h^\top$ and $\dfrac{\partial\ell}{\partial h} = W_U^\top(q-e_{v^\star})$.

Let $q^{(\tau)} = \operatorname{softmax}(z/\tau)$. Then $H(q^{(\tau)})$ is non-decreasing in $\tau>0$, with $q^{(\tau)}\to$ uniform as $\tau\to\infty$ and $q^{(\tau)}\to$ the argmax one-hot as $\tau\to0^+$ (assuming a unique maximum).