🧭Knowledge vs behaviour — the whole decision
Retrieval-augmented generation supplies relevant information at query time. The model is unchanged; you are handing it the right documents at the right moment. When the documents change, you update the documents.
Fine-tuning adjusts the model's weights on examples of the behaviour you want. The knowledge is not the point — you are teaching format, tone, structure, or a task pattern. When your knowledge changes, fine-tuning does not help, because you would have to retrain to update a fact.
That asymmetry is why the two are not really competitors. Almost every "should we use RAG or fine-tune?" question dissolves once you ask which of the two problems you actually have.
⚖️Trade-offs side by side
| Dimension | RAG | Fine-tuning |
|---|---|---|
| Solves | Model lacks information | Model responds in the wrong way |
| Updating knowledge | Edit a document — instant | Retrain — slow and costly |
| Citations possible | Yes — every claim traceable | No — knowledge is baked in |
| Setup effort | Ingestion + retrieval + evals | Dataset curation + training runs |
| Per-request cost | Higher — retrieved context adds tokens | Lower — shorter prompts |
| Latency | Retrieval adds a step | No retrieval hop |
| Ongoing maintenance | Keep the index fresh | Retrain when behaviour drifts |
| Auditability | Strong — show the source | Weak — cannot show why |
📚Choose RAG when…
The information changes
Policies, prices, product details, documentation. Anything you would update in a document rather than in a model.
Answers must be verifiable
Regulated environments need a citation a reviewer can click. Fine-tuning cannot show its source; retrieval can.
Access control matters
Retrieval can filter by the asking user's permissions at query time. A fine-tuned model cannot forget what it was trained on for one user and not another.
The corpus is large
You cannot fine-tune a model into reliably reciting fifty thousand documents. Retrieval scales with corpus size; memorisation does not.
🎛️Choose fine-tuning when…
You need a consistent output shape
A specific format, schema or house style enforced across thousands of calls, where prompting gets you most of the way but not reliably enough.
You need a specialised task pattern
Domain classification or extraction where the pattern is hard to express in a prompt but easy to demonstrate with examples.
Prompt length is the cost problem
If every request carries a long instruction block, tuning that behaviour in can shorten prompts and cut per-request cost at high volume.
Latency is critical and retrieval is the bottleneck
Removing the retrieval hop matters when you are optimising milliseconds — voice being the obvious case.
Before fine-tuning, exhaust prompting. A surprising share of fine-tuning projects are solving a problem that a better-structured prompt and a few examples would have solved for nothing.
🔗When you need both
The two compose cleanly, because they address different layers. A support assistant might be fine-tuned to always reply in your brand voice with a fixed structure, while retrieving the customer's actual order data at query time. The tuning handles how it speaks; retrieval handles what it knows.
Sequence matters, though. Build the RAG layer first and measure it. Retrieval failures and behaviour failures look identical from the outside — an answer that is wrong — and if you fine-tune before retrieval is solid, you will be tuning a model to compensate for documents it never received. That is expensive, and it produces a system nobody can debug.