leaderboard
Sorted by validation perplexity, the run's main metric. Perplexity is only comparable between runs that share a tokenizer and vocabulary, so the byte-pair row is marked separately rather than ranked against the char runs — see what this shows.
| variant | what changed | tokenizer | vocab | params | val loss | val perplexity |
|---|
validation loss curves
Same 2000 training steps, evaluated every 500. Click a name in the legend to hide or show that run.
what this shows
about this run
Every model here is a 4-layer, 128-dim transformer trained from scratch
(attention, positional embeddings and the byte-pair tokenizer are hand
written, not torch.nn.Transformer) on the tiny Shakespeare
corpus, 2000 steps, batch size 32, block size 64. The only thing that
differs between runs is the one change named in the table above. Numbers
on this page come straight from the training logs saved at the end of
each run — nothing here is estimated or rounded by hand.