🎯 Fine-tuning & RLHF

Fine-tuning & RLHF: A pretrained base model fresh out of pretraining is like a brilliant new hire who has read every book in the company library but has never once been shown wha

A pretrained base model fresh out of pretraining is like a brilliant new hire who has read every book in the company library but has never once been shown what a GOOD answer to a customer actually looks like — the mechanism is exact: pretraining only teaches "predict plausible next text," not "be a helpful, on-topic assistant that admits uncertainty" — those are entirely separate skills taught AFTER pretraining, through fine-tuning. Here is the question worth sitting with: if the base model already knows grammar, facts and code from pretraining, why does it need MORE training just to become a chatbot? Because a raw pretrained model, given "How do I center a div in CSS?", is just as likely to continue with "...is a common question on Stack Overflow, asked 47,000 times" (a plausible CONTINUATION) as it is to actually answer the question — pretraining never taught it "the user wants a direct helpful answer," only "predict likely next text." Java comparison: this is like the difference between a compiler that only checks syntax validity (pretraining: is this a plausible sentence?) and a code reviewer who checks whether the code actually serves the ticket's intent (fine-tuning: is this actually a HELPFUL response to what the user wanted?) — two different, sequential quality gates. The QA stake: RLHF specifically trains the model to prefer confident, complete-sounding answers, since human raters in blind comparisons tend to rate a hedgy "I don't know" as less helpful than a confident wrong answer — this is the BEHAVIORAL root of hallucination covered on the Claude AI page: the model is not malfunctioning, it was optimized, in part, toward exactly the confident tone that makes an occasional wrong answer dangerous.

SFT: Showing, Not Just Predicting

Supervised Fine-Tuning (SFT) takes the pretrained base model and continues training it, but now on a much smaller, curated set of example conversations — a prompt and a genuinely good, human-written or human-approved response — teaching the model the FORMAT and TONE of being a helpful assistant, not new facts.

Reasoning: why is RLHF needed on top of SFT — why isn't "show good examples" enough? Because writing enough example responses to cover every possible way a user might ask something is combinatorially impossible. RLHF instead trains the model with a cheaper signal: humans just RANK two of the model's own candidate answers ("which is better?"), a reward model learns to predict that ranking, and the base model is nudged to produce answers the reward model scores highly — turning a small amount of human JUDGMENT into a scalable training signal, instead of needing infinite hand-written examples.

Alignment, in One Paragraph

"Alignment" is the umbrella term for this whole post-pretraining process (SFT + RLHF + other techniques) — the goal is making the model's behavior match human intent and values (helpful, harmless, honest), not just "predicts plausible text." It is an ongoing area of research, not a solved problem, which is exactly why the risks covered later on this page — and on the Claude AI page — still exist even in well-aligned, modern models.

Step by Step: From Base Model to Helpful Assistant

A raw next-token predictor — no notion of "helpful" yet.

Trained on curated example conversations showing the desired format and tone.

Generate candidate pairs

The model produces two different answers to the same prompt.

A human rates which answer is better; a reward model learns this signal.

The base model is nudged, at scale, toward answers the reward model scores highly.

Arrange the sequence turning a pretrained base model into a helpful assistant.