🎓 Can You "Train" an Agent? Prompt vs RAG vs Fine-tune

Can You "Train" an Agent? Prompt vs RAG: Deciding whether to "train" a model for a QA task is like deciding whether a new employee needs a full multi-month training program or ju

Deciding whether to "train" a model for a QA task is like deciding whether a new employee needs a full multi-month training program or just a well-written onboarding doc — the mechanism is exact: most of what feels like "the model needs to learn our way of doing things" is actually solved by GIVING it the right information at the right moment (a system prompt, a pasted document), exactly like a new hire who reads the team's style guide before their first pull request needs no separate training course, just the document at the right time. Here is the question worth sitting with: if fine-tuning a model on your company's exact bug-report format is possible, why is it usually the WRONG first move, not the smart one? Because fine-tuning solves a problem — the model doesn't consistently use your format — that a well-written system prompt with 2-3 examples usually solves for free and instantly, while fine-tuning costs real engineering time (curating a labeled dataset), real money, and produces a static artifact that must be redone every time your format changes. You would be building a multi-month onboarding program for something a one-page style guide already fixes. Java comparison: this is the same judgment call as deciding whether a repeated code pattern deserves a new abstraction — a shared utility class, i.e. fine-tuning, expensive to build and only pays off if reused constantly and stable — or is fine as an inline snippet with a comment, i.e. a prompt, cheap and flexible until proven otherwise; premature abstraction is a real cost in both code and in AI customization. The QA stake: a tester who defaults to "let's fine-tune a model for this" without first exhausting prompt and RAG options is doing the AI equivalent of introducing a design pattern before the second use case — the decision table below exists specifically to prevent that.

Level 1 — Prompt: Free, Instant, Solves 90%

A system instruction — the "role" ingredient from the Claude AI page's Prompt Engineering tab — combined with a few examples directly in the prompt covers the vast majority of "make the model behave our way" needs. No dataset, no cost beyond the API call itself, and it can be changed in seconds. This should always be the first thing tried.

Level 2 — RAG: An Open-Book Exam, Not Training

Retrieval-Augmented Generation means fetching relevant company documents — a style guide, past bug reports, API docs — at the moment of the request and pasting the relevant chunks into the context window. The model's weights never change; this is not "training" in any sense, it's giving an open-book exam instead of expecting the model to have memorized the book. Use this when the model needs to know something specific to your company that changes often, rather than a stable behavior or format.

Level 3 — Fine-Tuning: Teaching a Stable Behavior

Fine-tuning via OpenAI's fine-tuning API does change the model's weights — a smaller-scale version of the SFT process from the earlier tab — training it on a curated dataset of example inputs and desired outputs so a specific behavior or format becomes consistent without needing to restate it every prompt. The real work is preparing a labeled dataset, often hundreds of examples, not the training run itself.

A fine-tuning training file (JSONL format)

When Fine-Tuning is NOT the Right Move

Fine-tuning is usually the wrong first move when: you only need the model to know current facts or docs (that's RAG, not fine-tuning); your format or behavior can already be achieved with a good system prompt plus 2-3 examples; your requirements change frequently (a fine-tuned model must be retrained every time, a prompt is edited in seconds); you don't yet have at least dozens-to-hundreds of quality labeled examples; you need the model to know something project-specific that changes daily (fine-tuning bakes in a snapshot, exactly like pretraining's training-cutoff problem); or you are trying to fix a single occasional mistake rather than a systematic, consistent one, where a better prompt or a validation step is far cheaper.

When Fine-Tuning DOES Make Sense

Fine-tuning earns its cost when: you need a very specific, stable output format followed with extreme consistency across huge volume, and prompting alone still leaves inconsistency; you want to reduce prompt length and cost at massive scale by baking in behavior that would otherwise require a long, repeated system prompt; or you already have a genuinely large, high-quality labeled dataset prepared.

Level 4 — Training From Scratch: Not Your League

Training from scratch (pretraining) requires the same massive compute/cost you saw in the Pretraining tab — this is not a decision a tester or most companies get to make. If you find yourself considering it, you have almost certainly misdiagnosed a problem that Levels 1-3 would actually solve.