The Objective Wars
How seven decades of research across information theory, psychology, linguistics, and deep learning shaped the way AI models are trained today
Next-token prediction — predicting which token comes next given the current context — has become the standard training paradigm for modern, generative AI models. It is so thoroughly the default that it barely registers as a choice any more. But it was one, and for a while it even looked like the wrong one.
This essay traces the history of how we got here, in three eras:
Prehistory (1951–2017) runs from Shannon’s early experiments in the 1950s to Google’s word2vec and beyond. Both objectives modern LLMs still run on — NTP (next-token prediction) and MLM (masked language modeling) — were specified here, decades before anyone had a real use for them.
The Objective Wars (2018–2020) is the short and turbulent two-year era in which the learning objective itself became the research contribution: NTP, MLM, NSP, RTD, and span corruption, all arguing over which pretraining task extracts the most useful information from a token of text for downstream tasks.
The Modern Era (2020+) is what happened after that argument settled — NTP for generation, MLM for embeddings — and where the momentum moved towards building around these objectives, resulting in innovations such as FIM and MTP. It also spawned diffusion as a serious competitor to NTP.
Almost everything here is older than it looks. The two objectives at the heart of modern AI were both specified in the 1950s, by people who weren't trying to build AI at all — a field that did not even exist yet.
Let's start our journey in 1951, inside Shannon's lab.
Prehistory (1951-2017)
Shannon’s entropy experiments (1951)
In “Prediction and Entropy of Printed English”, Claude Shannon, working at Bell Labs, describes what is essentially the first NTP experiment — except at the character level instead of the token level, and with humans instead of machine-learned models. And crucially, nothing was being trained: her used prediction here as a measuring tool, not a learning objective.
The setup was simple. He sat humans down to predict the next character in a passage given the preceding ~100 characters, and recorded how many attempts it took them to guess correctly. From that data he derived upper and lower bounds on the entropy of English language relative to a human predictor: roughly 0.6–1.3 bits per character, corresponding to a perplexity (2 to the power of entropy) of about 1.5–2.5. In other words, given a hundred characters of context, a competent reader is about as uncertain as a coin flip about the next one.
Compare that to the ceiling of log2(27) = 4.75 bits — what you would get from a model that knew nothing but the size of the alphabet. Shannon’s own progression using n-grams fills in the middle: letter frequencies (unigrams) get you to ~4.1 bits, bigrams ~3.6 bits, and trigrams ~3.3 bits. Each row is a better model of English, and the entropy falls monotonically.
In a sense, every subsequent breakthrough in language modeling and AI can be viewed as yet another rung on Shannon’s entropy ladder.
Taylor’s cloze test (1953)
While Shannon wanted to measure the entropy of a language holistically, Wilson Taylor, a communications grad student working under Psychology professor Charles Osgood at the University of Illinois, wanted to measure the reading difficulty of text passages. In order to do so, he invented the “cloze procedure”, a technique that would eventually become MLM, masked language modeling, and give rise to BERT and modern embedding models — all decades before Google even existed.
Here’s how it worked: Taylor replaced every 5th word in a passage with a blank “___” and asked human participants to guess the missing word. He then measured what percentage of blanks people filled in correctly, which became the cloze score. The higher the cloze score, the easier the lower the reading difficult.
Note that Taylor’s deletion is mechanical rather than hand-picked, and that is the whole point: choosing which words to delete would confound the test-writer’s judgment into the measurement, which is precisely what conventional sentence-completion tests did (and what Taylor chose not to do).
He validated his scores against the readability formulas of the day, Flesch and Dale–Chall, which score a passage from surface features alone such as syllable counts and sentence length. On ordinary prose, cloze broadly agreed with them. The interesting result is where it didn’t. Taylor deliberately selected passages from Joyce and Stein — writing with easy vocabulary but high conceptual load — and the formulas rated them as very readable while cloze ranked them much harder. His conclusion was that the formulas can be fooled by an author’s style, and that cloze was the more valid measure.
Bengio’s neural language model (2003)
Half a century after the work from Shannon and Taylor, Yoshua Bengio — one of the three “godfathers” of AI alongside LeCun and Hinton — introduced his Neural Probabilistic Language Model, the first neural network trained using the NTP objective that was capable of beating the existing n-gram baselines (which descended from Shannon’s work) on real corpora.
Bengio’s model architecture was embarrassingly simple by modern standards: embedding table lookup for each of the previous n−1 words, concatenated, fed through a tanh hidden layer, and then a softmax over the full vocabulary. No attention, no recurrence, no depth, and a hard context window of just five words (this is nine years before AlexNet and long before CUDA). It also predates subword tokenization, so the vocabulary consists of entire words pruned by frequency. Even with these crude simplifications, it took three weeks to train the model on 40 CPUs.
But the results were a real breakthrough. In addition to beating the best n-gram model by 10–20% perplexity on a text corpus, Bengio also showed that the neural model kept improving all the way to a context size of n=5 while the n-gram models saturated at n=3. This is the first demonstration that a neural model could actually convert extra context into better predictions, unlike n-gram models.
The reason neural language models did not take over at the time was economics. Three weeks on 40 CPUs for a ~20% perplexity gain over a trigram you could build in minutes wasn’t a trade anyone shipping a speech recognizer would take. Bengio was at least a decade too early to benefit from CUDA-programmable GPUs, which would have made his matrix operations much cheaper.
Mikolov’s word2vec (2013)
While Bengio presented the first neural language model, Google’s Tomáš Mikolov, working a decade later, was the first to make its byproduct shippable. In the paper “Efficient Estimation of Word Representations in Vector Space”, he trained a heavily pruned version of Bengio’s neural language model — dropping the hidden layer, using embedding pooling instead of concatenation, and using hierarchical instead of full softmax — with one of two training objectives:
CBOW (continuous bag of words) — predict the middle word from the four words to its left and the four to its right, making it a variant of Taylor’s cloze task with word order thrown away.
skip-gram — predict the surrounding words from the middle word, essentially an “inverted” cloze task.
Most importantly, word2vec — the name he gave the resulting library — was fast: CBOW trained on the Google News dataset on a single machine in about a day. For comparison, it took Bengio’s team three weeks on 40 CPUs to train their model on a dataset that was 50x smaller.
Mikolov’s was also the first work in which the word embedding itself was the product, not a byproduct. Indeed the model itself was throwaway: downloading word2vec means, in practice, downloading an embedding table.
You can read word2vec as Taylor’s cloze task, implemented on Bengio’s architecture, with Google’s engineering rigor applied until it was cheap enough for practical applications. And it worked — embeddings became infrastructure almost overnight.
But the compression that made it fast also capped what it could do. Averaging the context throws away word order, and more fundamentally, every word gets exactly one vector: bank means the same thing on a river as in a portfolio, forever. A representation that cannot change with context can only take you so far — and fixing that is precisely what the next era is about.
The Objective Wars (2018-2020)
ELMo and BERT (2018)
Both ELMo (work done by the Allen Institute for AI and the University of Washington) and BERT (work done by Google) were released in 2018, in February and October respectively. Both had the goal of generating general-purpose contextual text embeddings, but ELMo went the NTP route while BERT went the MLM route.
ELMo used plain NTP to train two language models independently, a forward LM (left-to-right) and a backward LM (right-to-left), which were then stapled together into a single, quasi-bidirectional LM. Under the hood the backbone is a biLSTM, predating the field’s switch to Transformers. The limitation is that neither direction ever conditions on the other: bidirectionality happens only at the final concatenation, so every layer of representation-building still runs on half the context.
BERT went with MLM on a bidirectional (i.e. no causal masking) Transformer architecture, making it a direct descendant of Taylor’s work — which the paper cites explicitly. The authors chose a 15% masking ratio — curiously but coincidentally close to Taylor’s 20% — with an 80/10/10 split, using a special [MASK] token 80% of the time, a random token 10%, and the original unchanged token 10% of the time. The reason for the last 20% is to fix train/serve skew: [MASK] only exists during pretraining and never appears downstream, and leaving some scored positions untouched means the model cannot tell from the input which positions it will be graded on, so it has to represent every token as if it might be asked. (A side effect is that the model also learns corrupted-token detection along the way — a task the later ELECTRA model would make its entire objective.)
BERT also used a second training objective called NSP (next sentence prediction): given two segments, predict whether B follows A, with negatives drawn from a different document. The motivation was to help with sentence-pair tasks such as QA and NLI, and the authors’ own ablation showed that NSP helped. This claim was later shown to be false by the authors of SpanBERT, RoBERTa, and ALBERT. The ALBERT paper explains why: because negatives came from a different document, NSP conflated topic prediction with coherence prediction — and topic is trivially easy, something MLM had already learned. ALBERT introduced a better second objective, SOP (sentence order prediction): take two consecutive segments from the same document and predict whether they are in the correct order, a task that cannot be solved by topic detection alone.
In effect, BERT was a bet: sacrifice 85% of the gradient signal — unmasked tokens produce no loss — in exchange for deep bidirectionality by design. (The 15% itself was a judgment call rather than a derived optimum; T5 later found the rate barely matters.) And the bet paid off: BERT beat GPT-1 on GLUE and set a new SOTA on SQuAD, both standard benchmarks at the time. This made MLM look like the superior objective over NTP — at least by the benchmarks of the day, which were primarily classification tasks, not generative ones.
The rise of GPT (2018-2020)
The origins of OpenAI’s GPT line of models can be traced back to four pivotal publications:
GPT-1 — June 2018, “Improving Language Understanding by Generative Pre-Training”
GPT-2 — Feb 2019, “Language Models are Unsupervised Multitask Learners”
Scaling laws — Jan 2020, “Scaling Laws for Neural Language Models”
GPT-3 — May 2020, “Language Models are Few-Shot Learners”
From the point of view we are taking here — training objectives — nothing in this work is fundamentally new, as all of these models are trained with the NTP objective we already traced back to Shannon and Bengio. GPT-1’s main contribution was the claim that NTP alone is sufficient for general-purpose pretraining.
Concretely, GPT-1 still followed the pretrain/fine-tune paradigm, setting new SOTA on 9 out of 12 tasks tested in the paper. However, it was eclipsed on exactly those fine-tuning-style tasks just months later by BERT and its bidirectionality/MLM bet.
GPT-2 and GPT-3 were subsequent scaled-up versions of the same decoder-style Transformer architecture (more parameters, more data), which allowed for transfer to downstream tasks without fine-tuning, using prompting alone. The scaling laws paper, published between them, provided the theoretical backbone: test loss falls as a smooth power law in model size, data, and compute, and architectural details barely matter next to those three. This finding is ultimately what redirected the field’s momentum from “which training objective” to “how many parameters and tokens”.
Curiously, throughout all of this, the training objective in this line remained unchanged from Shannon in the 1950s and Bengio in the early 2000s — all while researchers in the bidirectional/MLM camp were heavily debating and iterating on theirs. The two camps had made opposite bets: one that the leverage was in the objective, the other that it was in scale.
RoBERTa, T5, and ELECTRA (2019-2020)
Three more works from this era are worth highlighting here, as they made pivotal contributions to the bidirectional/MLM camp: RoBERTa, T5, and ELECTRA.
RoBERTa (July 2019, Meta) was a carefully executed replication study that found BERT was significantly undertrained. Meta’s engineers removed the NSP objective (which their own ablations found not to help downstream performance), introduced dynamic masking (re-mask at each epoch instead of just once ahead of training, effectively increasing the amount of training signal), increased the batch size, replaced BERT’s WordPiece tokenizer with byte-level BPE, and trained for longer on far more data (160GB, versus BERT’s ~16GB). Importantly, Meta’s work showed that Google’s findings were partially confounded by training budget, which goes to show that scientific rigor is hard when so many moving pieces are involved.
T5 (October 2019, Google) replaced token-level MLM with span corruption, masking a contiguous sequence of tokens instead of one token at a time and training the model to reconstruct the missing sequences. For example:
# Original
Thank you for inviting me to your party last week
# Corrupted input
Thank you <X> me to your party <Y> week
# Target output
<X> for inviting <Y> last <Z>Note that each masked span collapses into a single sentinel token, so the model isn’t even told how many tokens are missing. This is better than vanilla MLM, the authors argue, because unmasking a single token can often be trivial just based on its immediate surroundings. Reconstructing an entire subsequence forces the model to actually do some form of reasoning.
Empirically, T5’s performance peaked with a mean span length of 3 tokens, while the corruption rate barely mattered at all — BERT’s 15% is a fine default. T5-11B, the largest variant, set new SOTA on GLUE, SuperGLUE, and SQuAD, comfortably ahead of BERT and RoBERTa (though using a ~30x larger model, hence confounding gains due to its new training objective with sheer model size).
ELECTRA (March 2020, Stanford/Google) directly attacked BERT’s waste of loss signal — a masking ratio of 15% means 85% of the positions in each batch produce no gradient at all — replacing MLM with RTD, or “replaced token detection”, which receives gradients from every position in the sequence.
Under the hood, ELECTRA uses a small MLM generator that corrupts random tokens, and trains a discriminator to classify every position in the sequence as either original or replaced. While this sounds a lot like a GAN (generative adversarial networks), it is not: in ELECTRA the gradients never cross between the two networks, and the generator is trained with ordinary maximum likelihood, whereas in a GAN the forger is specifically trained to maximize the discriminator’s loss. The generator here is just a corruption source that happens to be learned — which is the whole point, since a learned corrupter produces plausible yet wrong tokens rather than obviously absurd ones.
Ultimately, ELECTRA-Small beat a comparably-sized GPT model on GLUE at a fraction of the compute, while ELECTRA-Large matched RoBERTa at roughly a quarter of the pretraining compute, highlighting RTD’s superior signal extraction efficiency with respect to plain MLM.
The Modern Era (2020+)
FIM for code completion models (2022)
FIM (”fill in the middle”) was proposed in a July 2022 paper by OpenAI as a way to make NTP-trained models better at code completion. The motivation is that code is edited in place rather than written forward, with context both above and below, and a causal model throws away everything below. FIM’s trick is to simply rewrite training examples in a permuted order, moving the middle to the end:
# original
[prefix][middle][suffix]
# transformed
<PRE>[prefix]<SUF>[suffix]<MID>[middle]and then use plain NTP to have the model, quite literally, fill in the middle. Note that nothing else changes about the model itself: FIM is just a data preprocessing step.
OpenAI’s paper shows that applying this transform to a large fraction of the training data (they recommend 50–90%) buys infilling capability with no measurable degradation to left-to-right generation. FIM training is documented in open-source coding models such as StarCoder, Code Llama, DeepSeek-Coder, and Qwen-Coder; the frontier labs do not publish their pretraining recipes, so its use there is likely but unconfirmed.
MTP for massive inference speed-up (2024)
MTP (multi-token prediction) was first proposed in an April 2024 paper by Meta, and first productized in December of the same year in DeepSeek-V3. The idea is to predict the next n tokens instead of just the next one (using n heads on a shared trunk) and sum the resulting n softmax losses during training. This forces the model to look further into the future, which in theory should give it a better internal representation of where the sequence is going.
While the training gains are somewhat contested, what has become clear is that MTP brings sizable inference speed-ups, because the auxiliary heads can propose a draft of several tokens which the model then verifies in a single forward pass — a technique known as “speculative decoding”. The Meta paper reports ~3x speed-up in inference.
MTP is now standard across the open frontier: DeepSeek, Qwen3-Next, GLM, MiMo, and Nemotron all document it, and you can verify it directly in their published checkpoint configs on Hugging Face. The closed labs (Google, OpenAI, Anthropic) do not publish their pretraining recipes, so their use of it is unknown.
Diffusion: a new competitor to NTP (2021-)
Diffusion is an interesting recent development in the field because it brings Taylor’s branch of research back into the picture in the era of generative AI (which so far has been dominated by Shannon’s NTD branch). There is no clear first mover — diffusion-trained language models have been proposed in multiple works over the past few years including D3PM (Google, 2021), Diffusion-LM (Stanford, 2022), SEDD (Lou et al., 2024), and MDLM (Cornell, 2024).
The key idea is to start with a fully masked sequence and then train a model to iteratively unmask that sequence, conditioning at each step on everything currently visible. Each step reveals more tokens, and the model is trained to denoise at every noise level, for example:
[M][M][M][M][M][M] → step 1
[M][M] cat [M][M][M] → step 2
the [M] cat [M] the [M] → step 3
the big cat sat on the → doneArchitecturally, this is done using a BERT-style bidirectional Transformer instead of the unidirectional one typically used for generation.
Because many positions are unmasked at each step, the main advantage of this approach is speed: a sequence of length L takes far fewer than L forward passes. Gemini Diffusion, for example, generates at ~1.5K tokens/sec, roughly an order of magnitude faster than Gemini Flash, scoring comparatively on quality benchmarks. Other examples of diffusion language models are LLaDA-8B (Renmin University and Ant Group, 2025) and Mercury (Inception Labs, 2025).
Diffusion is the first serious challenge to NTP’s monopoly in the generative AI era. Its clearest advantage today is speed, but the more interesting one is that masking different positions at each pass lets it keep extracting new signal from a corpus that a fixed left-to-right factorization has already exhausted. Whether that matters depends on whether (and if so, when) we run out of text to train the next generation of models.
Coda
Every training objective used in production today was specified in the late 1940s and early 1950s. NTP is Shannon’s guessing game. MLM is Taylor’s cloze test. And diffusion is the natural extension of MLM to the complete range of masking rates — which turns out to be all it took to make it generative.
The people behind these inventions were not AI researchers at all. They were an engineer at Bell Labs measuring channel capacity, and a communications grad student measuring the reading difficulty of written passages. Neither could have imagined the effect their work would have seven decades later.
A common theme running through the entire history is that of the “byproduct becoming the main product”. Embeddings were a byproduct in Bengio's neural LM, but word2vec made them the main product a decade later, throwing away the model away and shipping the embedding table. BERT's random replacement was an adhoc fix for train/serve skew; ELECTRA scaled it up and made it the whole objective. And masking itself was just a trick to enable bidirectional learning — the masking rate a hyperparameter nobody thought much about (and that didn’t seem to matter too much either). Diffusion turned that hyperparameter into a schedule, and got a generative model out of it.
The objective wars were ultimately won by NTP, but only because the momentum moved away from classification towards generation — driven by GPT-3, which showed that with a sufficiently large, generative model you no longer need to fine-tune for each individual downstream task. That said, MLM never really died. It is still the workhorse of production systems where the task is representation learning — building high-quality embeddings — not generation. It is just not viewed as being at the frontier any more.