RAG vs. fine-tuning: how to actually decide

'Should we use RAG or fine-tuning' is one of the most common questions we get asked, and it's usually the wrong question. The two solve different problems, and most production systems that work well end up using both. But if you have to pick a starting point, there is a real decision to make, and getting it backwards costs weeks.

RAG solves a knowledge problem: the model needs access to facts it wasn't trained on, those facts change over time, and you need to be able to point at a source for any given answer. Fine-tuning solves a behavior problem: the model needs to consistently produce a specific output format, apply a specific judgment pattern, or perform a narrow task faster and cheaper than a general-purpose model can. Confusing the two is the single most common mistake we see. Teams fine-tune a model to 'know' their product documentation, which decays the moment the docs change, when they needed retrieval; or they bolt RAG onto a task that's actually a stable classification problem, when a small fine-tuned model would have been cheaper, faster, and just as accurate.

The test we actually use has three questions. First: does the correct answer depend on information that changes on a timescale shorter than your retraining cadence? If yes, that's a RAG signal. A fine-tuned model's 'knowledge' is frozen at training time, and continuously retraining to keep pace with changing facts is almost always more expensive than a retrieval layer. Second: do you need to show your work, cite a source, produce an audit trail, let someone verify the answer against ground truth? That's also a RAG signal, structurally, because fine-tuning bakes knowledge into weights with no pointer back to a source document. Third: is the task actually about producing a consistent output shape or judgment against a stable rule set, where the 'knowledge' involved is really just pattern-matching to examples? That's a fine-tuning signal, and RAG will usually just add retrieval latency and irrelevant context to a problem that doesn't need external facts at all.

We ran into the RAG-shaped mistake with a fintech client's compliance screening system. The team had fine-tuned a model to flag transactions against a set of regulatory rules, and every time a regulation was updated, they had to recollect examples and retrain. Retraining lag meant new rules took an average of three weeks to actually take effect in production, during which the model was silently screening against outdated policy. We moved the rule definitions into a retrieval layer the model queried at inference time. The 'knowledge' was now a document, not a weight, and a rule update took effect the moment the document changed, with zero retraining. The classification logic that decided how to weigh conflicting signals stayed as a lightweight fine-tuned component, because that judgment pattern actually was stable.

We ran into the opposite mistake with a support-ticket triage system that had been built as a RAG pipeline retrieving 'similar past tickets' to help decide a ticket's category and priority. The category taxonomy hadn't changed in over a year, and the task was structurally just classification against a fixed label set. The retrieval step was adding 400ms of latency and, worse, was occasionally retrieving a superficially similar but wrongly-categorized past ticket and anchoring the model's answer to the wrong label. We replaced it with a small fine-tuned classifier trained on the labeled ticket history. Accuracy went up (no more anchoring on bad retrieval results) and p95 latency dropped from 1.4s to under 200ms.

In practice, the systems that work best combine both, each doing the job it's structurally suited for: a fine-tuned model handling the parts of the task with a stable, learnable pattern (tone, format, a fixed taxonomy), while retrieval supplies the facts that change or need to be cited. Our own LLMOps and fine-tuning and RAG work are frequently the same engagement, not two competing options a client has to choose between up front.

The generalizable rule: if the honest answer to 'why would this response be wrong' is 'the facts changed since training' or 'we can't verify where that came from,' you have a retrieval problem. If the honest answer is 'the pattern is right but the format or judgment is inconsistent,' you have a fine-tuning problem. Diagnose the failure mode before picking the architecture. The architecture should follow from what's actually broken, not from whichever approach got the most attention in a conference talk that quarter.