🎓 How LLMs Are Trained: Pretraining

How LLMs Are Trained: Pretraining: Pretraining a model is like a photography student who studies millions of photographs without ever being told explicitly "this composition is g

Pretraining a model is like a photography student who studies millions of photographs without ever being told explicitly "this composition is good, this one is bad" — the mechanism is exact: instead the student is given ONE recurring drill, "given the first 90% of a photo, guess what the missing corner looks like," and doing that billions of times across millions of photos forces the student to internalize composition, lighting and subject matter without anyone hand-labeling "composition" as a concept. Here is the question worth sitting with: if nobody explicitly taught the model grammar or Python syntax, how does it end up knowing both? Because predicting the next token accurately, at internet scale, is only possible if you have implicitly absorbed the regularities underneath — grammar, syntax, factual patterns — the prediction objective forces the competence in as a side effect, never as an explicit lesson. Java comparison: this is like a compiled .class file — it carries the DISTILLED behavior of the source, not the source itself; pretrained model "weights" are the compiled artifact of billions of next-token predictions, and you can no more read the original training text back out of the weights than you can recover Java source from bytecode. The QA stake: since the model's "knowledge" is frozen at the moment its training data was collected (the training cutoff), it will confidently use syntax that was correct then but deprecated now — this is the mechanical root of the "outdated library syntax" risk covered on the Claude AI page; knowing WHY it happens (frozen weights, not carelessness) is what leads a tester to always state their exact library version in a prompt.

The One Drill: Predict the Missing Piece

Pretraining feeds a model enormous slices of internet text — books, code, forums, documentation — and repeats one exercise: hide the next token, ask the model to guess it, measure how wrong the guess was (this gap is called "loss"), and nudge the model's billions of internal weights very slightly toward a better guess. Repeat this drill trillions of times.

Reasoning: why does this require GPUs, months, and enormous cost, when the "exercise" sounds so simple? Because "very slightly nudge billions of weights" must happen after EVERY single prediction, across trillions of predictions, and each nudge requires enormous matrix computation. Java comparison: it is like recompiling a massive shared codebase after every single line changes, except the "codebase" has billions of parameters and this happens trillions of times — the sheer volume of repeated computation, not conceptual difficulty, is what demands specialized hardware running for months.

Step by Step: From Raw Text to a Pretrained Model

Books, code, forums, docs — a massive, diverse slice of text is gathered.

The next token of a passage is hidden from the model — it is forced to guess it.

The model guesses, the guess is compared to the real token, and the gap is measured as "loss".

Billions of internal weights are nudged very slightly toward reducing that loss.

Steps 2-4 repeat trillions of times across the whole dataset, producing the pretrained "base model".

Arrange the pretraining loop in the correct order.

Collect a massive, diverse slice of text

Hide the next token of a passage

Let the model predict, compare to the real token, measure the loss