Perplexity is not comparable across tokenisers

NumericalBeginner⏱ 15 minLLMsperplexitytokenisationevaluationbits per byte

Theory: Language Modelling, Cross Entropy and Perplexity →

Problem

Two models assign the same total probability to a corpus of $C = 10^6$ characters, but tokenise it into $T_1 = 2.5\times10^5$ and $T_2 = 5\times10^5$ tokens. Given $-\log q(\text{corpus}) = 1.6\times10^6$ nats, compute both perplexities and the bits-per-character for each, and state which comparison is meaningful.

Compute the requested quantities and say what they mean. A number without an interpretation is not an answer.

Solution

Opening this before you have a real attempt on paper is the fastest way to feel like you understand something you cannot reproduce.

Related problems

All problems

Problem	Field	Type	Difficulty	Time
Maximum likelihood is KL minimisation	LLMs	Proof	Beginner	15 min
The cross-entropy gradient is the prediction error	LLMs	Proof	Beginner	15 min
The irreducible loss is the conditional entropy	LLMs	Proof	Intermediate	30 min
Build and instrument a minimal RAG pipeline run	RAG	Coding	Advanced	1 day
Temperature monotonically controls entropy	LLMs	Proof	Advanced	30 min