Token Inequality: Why a Model Learns Far More From Some Text Than Others

In 2023, Microsoft researchers trained a small language model called phi-1 — just 1.3 billion parameters — on roughly six billion tokens of carefully filtered, "textbook-quality" code and synthetic exercises. It was a fraction of the size of the frontier models and trained on a fraction of the data. It also outperformed much larger models trained on far more text at the coding tasks it was built for. The paper's title said the quiet part out loud: Textbooks Are All You Need. The lesson was not that scale is worthless. It was that the tokens were not equal — that six billion well-chosen tokens taught the model more than many times that number of ordinary ones, because the informational value packed into text varies enormously, and a training process that treats all tokens the same is leaving most of its leverage on the floor.

This is token inequality: the fact that the tokens a model learns from are radically unequal in what they teach, even though the training process consumes them as if they were interchangeable. From the outside, training looks uniform — trillions of tokens go in, the model improves. Inside, some of those tokens are doing almost all the work, and others are teaching the model close to nothing at all.

Why tokens teach unequal amounts

The intuition that all training data contributes roughly equally is wrong for a reason that becomes obvious once stated: learning happens at the surprises, not the repetitions. A token that is densely patterned, structurally rich, and not already predictable teaches the model a great deal, because predicting it forces the parameters to encode something they did not already contain. A token that is formulaic — boilerplate, filler, the thousandth near-copy of a common phrase — teaches almost nothing, because the model could already predict it, and correctly predicting what you already know produces no learning. The tokens are physically identical units of training; their marginal contribution to the model's capability spans orders of magnitude. This is why data curation has turned out to matter as much as data quantity: filtering for high-information text is not tidying, it is concentrating the tokens that actually teach.

The research has made the inequality precise. Deduplicating training data — simply removing the repeated tokens that teach nothing new — produces better models that memorize less and train faster, which only makes sense if the duplicates were contributing near-zero learning. And the 2024 work pointedly titled Not All Tokens Are What You Need (the Rho-1 models) went further: by scoring tokens for usefulness and training selectively on the high-value ones, it achieved large accuracy gains while ignoring much of the corpus, direct evidence that a big fraction of ordinary training tokens can be skipped with no loss. The model does not need all the tokens. It needs the ones that teach.

The second face: tokens are unequal in cost, too

Token inequality has a second dimension that compounds the first, and it falls along the fault lines of language. Because tokenizers are built mostly on English and Latin-script text, the same meaning costs wildly different numbers of tokens in different languages: one 2023 analysis found the leading tokenizer produced sequence-length disparities up to 15×, with Burmese requiring about 11.7× more tokens than English and Vietnamese around 2.5×. This is not a curiosity — it is a tax. Because commercial models charge per token and reason within a fixed token budget, speakers of non-Latin, low-resource languages pay more money for the same query, wait longer, and get a smaller effective context window in which the model can think. Tokens are unequal in what they teach and in what they cost, and both inequalities disadvantage the same populations: the languages underrepresented in training are also the ones that are most expensive to use and the ones the model learned the least from. The inequality of learning and the inequality of access are two faces of the same skew in the data.

Why the inequality matters

Token inequality reframes what a model actually is. It is not an even distillation of all the text it saw; it is disproportionately shaped by the small fraction of high-information tokens, which means the character of a model is set by the quality and composition of its best data far more than by the raw size of its corpus. That has consequences that ripple outward. It means adding more low-information data — including the flood of formulaic, machine-generated text now filling the internet — does little to improve a model and may actively degrade it, the mechanism behind the model collapse the series discussed in the Misinformation Bootstrap (#40): training on your own average output feeds the model tokens that teach nothing and reinforce everything. It means the competitive frontier in AI is shifting from who has the most data to who has the most informative data, which is a very different and much scarcer resource. And it means the linguistic skew is not a rounding error to be fixed later but a structural inequality baked in at the level of the token, disadvantaging most of the world's languages in both capability and cost.

The counterpoint: inequality is not always injustice

Honesty requires separating the two senses of "inequality" the concept trades on, because they are not equally troubling. That some tokens teach more than others is not unfair — it is simply true, and exploiting it through curation is good engineering, not discrimination; there is nothing wrong with a model learning more from a proof than from filler. The genuinely troubling inequality is the second one: that the skew systematically disadvantages speakers of underrepresented languages, in both what the model can do for them and what it charges them to do it. Conflating the two would be an error — the first is a fact about information to be used, the second is a harm to be corrected. The reason to hold them in one concept is that they share a root: both come from the fact that tokens, treated as interchangeable, are nothing of the kind, and a field that measures its progress in raw token counts is measuring with a ruler that hides exactly the differences that matter most.

What follows

Token inequality is a correction to the reflex that made "more data" the answer to every question. The tokens are not equal; the learning is concentrated; the costs are skewed; and the model that results is the shadow of its most informative fraction, cast unevenly across the languages it was fed. The practical implications run in two directions at once. For building models, the leverage is in curation and composition — finding, weighting, and generating the tokens that teach, rather than accumulating the ones that do not. For using them fairly, the work is in flattening the cost-and-capability skew that makes the same intelligence cheaper and better for some languages than others. Both start from the same refusal of the comfortable assumption the whole field ran on: that a token is a token is a token. It is not, and almost everything interesting about what a model becomes lives in the difference.

This is article #78 in The IUBIRE Framework series. Token Inequality was articulated by IUBIRE V3 in artifact #3944 — "The Token Inequality Problem: Why AI Systems Learn Unevenly." Real-world data: Microsoft's phi-1 / "Textbooks Are All You Need" (2023), where ~6B tokens of curated code beat far larger corpora; "Deduplicating Training Data Makes Language Models Better" (Lee et al., 2021/ACL 2022); "Not All Tokens Are What You Need" / Rho-1 selective language modeling (2024), gaining accuracy by training only on high-value tokens; and Petrov et al. (2023) on tokenizer unfairness between languages (up to ~15× sequence-length disparity; Burmese ≈11.7×, Vietnamese ≈2.5× vs English), which translates directly into higher cost and smaller effective context for non-Latin languages.

Next in series: Brand Fortress (#79)