{"slug":"research-tokenizer-not-the-language","name":"Is the Language Hard to Model, or Its Tokenizer?","rules":{"start":"open","maxDepth":14,"allowRewind":true,"allowReplay":true},"nodes":{"open":{"short":"the note","pose":{"base":"centre"},"beat":{"layout":"center","marker":"pre-registration · interactive","title":["Is the Language Hard","to Model, or Its Tokenizer"],"glass":1,"lede":"A pre-registered test of whether cross-lingual efficiency gaps come from languages or from how we represent them. Sanskrit encodes 5.07 characters per token under an aligned tokenizer, 1.13 under GPT-4's. Five languages, three scripts, four tokenizers.","caption":"14 beats · 18 min to read in full"},"prompt":"Where do you want to start?","choices":[{"label":"From the top","to":"premise"},{"label":"how the experiment","to":"how-the-experiment-is-built"},{"label":"Straight to the end","to":"close"}]},"premise":{"short":"the premise","pose":{"base":"flooded"},"beat":{"layout":"left","marker":"01 · pre-registration","title":["This series keeps","making one move"],"lede":"This series keeps making one move. The substrate a worker runs on decides more than the worker. We showed it for coding agents, where the environment moved the cost and the model already had the skill.","caption":"Condensed from the note."},"prompt":"Where next?","choices":[{"label":"why some languages","to":"why-some-languages-only-look-har"},{"label":"Wrap it up","to":"close"}]},"why-some-languages-only-look-har":{"short":"why some languages","pose":{"base":"drained"},"beat":{"layout":"right","marker":"02 · why some languages","title":["Why some languages","only look hard to model"],"lede":"Ask why a model is worse at Amharic than at English and the ready answer is that Amharic is harder. The script is unfamiliar, the morphology is dense, there is less data. The answer feels complete, and it puts the cause inside the language.","caption":"Condensed from the note."},"prompt":"Keep going?","choices":[{"label":"the two questions","to":"the-two-questions-the-experiment"},{"label":"Wrap it up","to":"close"}]},"the-two-questions-the-experiment":{"short":"the two questions","pose":{"base":"centre"},"beat":{"layout":"left","marker":"03 · the two questions","title":["The two questions","the experiment answers"],"lede":"The whole experiment reduces to two, and the second is the cleaner one.","caption":"Condensed from the note."},"prompt":"What next?","choices":[{"label":"three findings","to":"three-findings-the-literature-al"},{"label":"Wrap it up","to":"close"}]},"three-findings-the-literature-al":{"short":"three findings","pose":{"base":"flooded"},"beat":{"layout":"right","marker":"04 · three findings","title":["Three findings the literature","already supports"],"lede":"We are not starting from zero. Three findings are solid enough to build on, and we cite them as established rather than assumed.","caption":"Condensed from the note."},"prompt":"Where next?","choices":[{"label":"why sanskrit","to":"why-sanskrit-is-the-sharpest-tes"},{"label":"Wrap it up","to":"close"}]},"why-sanskrit-is-the-sharpest-tes":{"short":"why sanskrit","pose":{"base":"drained"},"beat":{"layout":"left","marker":"05 · why sanskrit","title":["Why Sanskrit","is the sharpest test"],"lede":"Every hypothesis wants an extreme case, and Sanskrit is it. Formalized by Panini in the Aṣṭādhyāyī over two millennia ago, it builds words from verbal roots, affixes, and deterministic phonetic fusions called sandhi.","caption":"Condensed from the note."},"prompt":"Keep going?","choices":[{"label":"four ways","to":"four-ways-a-cross-lingual-compar"},{"label":"Wrap it up","to":"close"}]},"four-ways-a-cross-lingual-compar":{"short":"four ways","pose":{"base":"centre"},"beat":{"layout":"right","marker":"06 · four ways","title":["Four ways a cross-lingual","comparison goes wrong"],"lede":"Most cross-lingual comparisons are quietly broken, and the breaks all inflate or deflate the wrong variable.","rows":[{"title":"Perplexity is not comparable across tokenizers. ","note":"It is computed per token, so a tokenizer that fragments more makes each next-token prediction easier and reports a lower perplexity while compressing the sequence worse."},{"title":"Same steps is not same compute. ","note":"Byte sequences run several times longer than subword sequences, so training both for the same number of gradient steps hands the subword model far more content per step."},{"title":"Translationese fakes fluency. ","note":"Benchmarks translated out of English carry English syntax, so a model can score well on them by exploiting English-aligned structure rather than by handling the target language naturally."},{"title":"Fragmentation warps the geometry of context. ","note":"Position-based attention treats tokens that are far apart as weakly related."}],"caption":"Condensed from the note."},"prompt":"What next?","choices":[{"label":"how the experiment","to":"how-the-experiment-is-built"},{"label":"Wrap it up","to":"close"}]},"how-the-experiment-is-built":{"short":"how the experiment","pose":{"base":"flooded"},"beat":{"layout":"left","marker":"07 · how the experiment","title":["How the experiment","is built"],"lede":"The design is a factorial grid, built so the same content passes through every combination we want to compare. Every cell is one measurement on identical meaning.","rows":[{"title":"Compression. ","note":"Sliding-window bits-per-byte, with stride adaptation so only the non-overlapping trailing segment counts, which equalizes the metric across tokenization granularities and removes short-context bias."},{"title":"Efficiency. ","note":"Floating-point operations to reach a target loss, plus sequence length and throughput per step, which is where the Token Tax shows up as money."},{"title":"Memorization. ","note":"Discoverable memorization and extraction rate, measured per language."},{"title":"Representation quality. ","note":"Cross-lingual probing alignment, and the native-versus-transliterated activation comparison using sparse autoencoders, following Verma et al."}],"caption":"Condensed from the note."},"prompt":"Where next?","choices":[{"label":"how we will answer","to":"how-we-will-answer-the-two-quest"},{"label":"Wrap it up","to":"close"}]},"how-we-will-answer-the-two-quest":{"short":"how we will answer","pose":{"base":"drained"},"beat":{"layout":"right","marker":"08 · how we will answer","title":["How we will answer","the two questions"],"lede":"For the variance-decomposition question, the factorial grid feeds a straightforward analysis.","caption":"Condensed from the note."},"prompt":"Keep going?","choices":[{"label":"how strongly","to":"how-strongly-the-evidence-backs"},{"label":"Wrap it up","to":"close"}]},"how-strongly-the-evidence-backs":{"short":"how strongly","pose":{"base":"recede"},"beat":{"layout":"left","marker":"09 · how strongly","title":["How strongly the evidence","backs each claim"],"lede":"Keeping the honesty explicit, here is how strongly the literature backs each link, from most causal to most confounded.","panelRows":[{"title":"Tokenization is the primary structural lever on efficiency","note":"Strength Strong · Basis Fertility predicts accuracy and cost; fragmentation inflates sequence length geometrically (Lundin 2026, Petrov 2023)"},{"title":"The compute budget must be normalized, not the step count","note":"Strength Strong · Basis Iso-FLOP simulation shows the subword win is throughput, not representation (Gigant 2026)"},{"title":"Script drives internal representation more than structure does","note":"Strength Strong · Basis Native and romanized text activate disjoint neurons (Verma 2026); dedicated tokenizers help most for under-represented languages (Rust 2021)"},{"title":"Transliteration recovers efficiency for low-resource languages","note":"Strength Strong · Basis Unified-script inputs lift NER F1 substantially (Moosa 2022)"}],"caption":"Condensed from the note, which carries 7 rows."},"prompt":"What next?","choices":[{"label":"scope and limits","to":"scope-and-limits"},{"label":"Wrap it up","to":"close"}]},"scope-and-limits":{"short":"scope and limits","pose":{"base":"flooded"},"beat":{"layout":"right","marker":"10 · scope and limits","title":["Scope","and limits"],"lede":"The caveats shape the claim, so they sit in the body.","rows":[{"title":"No first-party results yet. ","note":"This is a pre-registration. Every number here is from the published literature, verified against source, and every claim about our own outcomes is a plan, not a finding."},{"title":"A from-scratch iso-FLOP sweep is a real bill. ","note":"Training small decoder models across this grid is far heavier than our coding-agent experiments, which ran off-the-shelf agents."},{"title":"Sanskrit is a stress test, not a proof about Sanskrit. ","note":"Its density is measured on a small, classical, poetic corpus. That is what makes it a clean probe of tokenizer alignment, and it is also why we do not generalize from the Gita to Sanskrit as a…"},{"title":"Memorization is the soft outcome. ","note":"It is the hardest of the four to measure without confounds at small scale, and we will treat a memorization result as suggestive rather than settled."}],"caption":"Condensed from the note, which carries 5 points."},"prompt":"Where next?","choices":[{"label":"what a result","to":"what-a-result-would-change"},{"label":"Wrap it up","to":"close"}]},"what-a-result-would-change":{"short":"what a result","pose":{"base":"drained"},"beat":{"layout":"left","marker":"11 · what a result","title":["What a result","would change"],"lede":"The coding experiments in this series kept landing on the same sentence: the capability was already there, and the substrate around it moved the cost. This is that sentence again, one layer down.","caption":"Condensed from the note."},"prompt":"Keep going?","choices":[{"label":"references","to":"references"},{"label":"Wrap it up","to":"close"}]},"references":{"short":"references","pose":{"base":"centre"},"beat":{"layout":"right","marker":"12 · references","title":["References","pre-registration"],"lede":"Cited works, verified against source. Years and venues follow the primary record.","rows":[{"title":"Asgari, E., El Kheir, Y., and Sadraei Javaheri, M. ","note":"(2025). MorphBPE: A Morpho-Aware Tokenizer. arXiv:2502.00894."},{"title":"Conneau, A., et al. ","note":"(2020). Unsupervised Cross-lingual Representation Learning at Scale (XLM-R). ACL 2020. arXiv:1911.02116."},{"title":"Gigant, T., Peng, B., and Quesnelle, J. ","note":"(2026). Decoupling the Benefits of Subword Tokenization for Language Model Training via Byte-level Simulation. arXiv:2604.27263."},{"title":"Kallini, J., et al. (2025). MrT5","note":"Dynamic Token Merging for Efficient Byte-level Language Models. ICLR 2025. arXiv:2410.20771."}],"caption":"Condensed from the note, which carries 14 points."},"prompt":"What next?","choices":[{"label":"Close the note","to":"close"}]},"close":{"short":"the end","pose":{"base":"finale"},"beat":{"layout":"center","title":["That was one path.","The note has the rest."],"glass":1,"lede":"This walk is one route through Is the Language Hard to Model, or Its Tokenizer?. The full note carries every chapter, the tables, the code, and the source it was adapted from.","links":[{"href":"/research/tokenizer-not-the-language","label":"Read the full note"},{"href":"/research","label":"All research","tone":"ghost"}]}}}}