RAG vs fine-tuning: how to choose for a real business use case

What is the short answer?

RAG for knowledge, fine-tuning for behaviour. If the question is "how do we get the model to answer from our own data," the answer is almost always RAG. If the question is "how do we get the model to respond in exactly this way, every time, at lower cost," the answer is usually fine-tuning, after prompting has been pushed as far as it goes.

Most teams should start with RAG for practical reasons: it works with any model, including hosted ones from OpenAI, Anthropic and Google, updates the moment you re-index a document, produces citations, and fails visibly when retrieval misses. Fine-tuning needs curated data, an evaluation harness and re-training when requirements change, and still needs retrieval if the facts move.

What do RAG and fine-tuning actually do?

RAG: retrieve, then generate

RAG was described by Lewis et al. (2020) as combining a model's parametric memory with a non-parametric memory, a searchable index of documents. In a business system that means chunking your content, embedding it into a vector store such as pgvector or Qdrant, often alongside a keyword index, retrieving the most relevant chunks for each question, and placing them in the prompt with instructions to answer only from them and to cite. The model's weights never change. Quality depends on chunking, retrieval and reranking far more than on the model.

Fine-tuning: change the weights with examples

Fine-tuning continues training a pre-trained model on your own examples so that its default behaviour shifts. OpenAI's model optimization guide lists supervised fine-tuning ("provide examples of correct responses to prompts"), direct preference optimization (a correct and an incorrect response per prompt), reinforcement fine-tuning (grade responses and reinforce reasoning) and vision fine-tuning. For open-weight models such as Llama, Qwen or Mistral, parameter-efficient methods dominate: LoRA (Hu et al., 2021) freezes the base weights and trains small low-rank matrices, which the paper reports cut trainable parameters by 10,000 times and GPU memory by 3 times versus full fine-tuning of GPT-3 175B. QLoRA (Dettmers et al., 2023) adds 4-bit quantization of the base model and reports fine-tuning a 65B-parameter model on a single 48GB GPU.

How do you decide between RAG and fine-tuning?

Run through the criteria below. If most answers point one way, start there; if they split, you probably need both.

• Does the knowledge change often? Yes favours RAG; re-indexing takes minutes, re-training takes a cycle.
• Do answers need citations or an audit trail? Yes favours RAG; retrieved passages are the evidence.
• Is the problem style, format or a narrow repeated task? Yes favours fine-tuning; examples teach format better than instructions.
• Is prompt length or latency a cost problem at volume? Yes favours fine-tuning a smaller model, which can replace long few-shot prompts.
• Is the data private or regulated? RAG keeps documents outside the model and lets you enforce per-user access at retrieval time; fine-tuning bakes data into weights you must then protect.
• Do you have thousands of high-quality labelled examples? No favours RAG or prompting; fine-tuning on thin or noisy data makes things worse.
• Must the base model stay hosted (OpenAI, Anthropic, Google)? RAG works with all of them; fine-tuning availability depends on the provider and model.
• Does the model misunderstand your domain language? Yes favours fine-tuning, often on top of RAG.

What do RAG and fine-tuning cost, and how long do they take?

A first RAG prototype over a few hundred documents typically takes an experienced engineer days, because every component is off the shelf. The ongoing cost is retrieval infrastructure plus per-token inference, which rises with how much context goes into each prompt. The hidden cost is quality work: cleaning documents, tuning chunking, adding hybrid search, evaluating retrieval precision and handling permissions. That is where most RAG budgets go.

Fine-tuning cost splits into data, compute and evaluation. Data is the largest item: someone has to assemble and clean the examples. Compute has fallen sharply since LoRA and QLoRA; a mid-sized open model can now be adapted on one rented GPU in hours rather than on a cluster. Hosted providers charge per training token and then a premium per inference token for the tuned model. OpenAI's guidance is to build evals first and iterate on prompts, noting that "the prompt engineering process may be all you need in order to get great results for your use case." Fine-tuning also recurs: each change in requirements or base model version means a new training run and a fresh evaluation pass.

When do you need both?

Combine them when facts and behaviour are both problems. A support assistant that must answer from a knowledge base (RAG) in a strict, short, brand-compliant format with structured fields for the ticketing system (fine-tuning) is the classic case. The fine-tuned model is better at using retrieved context in the way you want, and the retrieval keeps the content current.

Two other combinations are common: fine-tuning the embedding model or reranker to lift retrieval on domain jargon without touching the LLM, and distilling a large model's behaviour into a smaller fine-tuned one to cut latency and cost while RAG supplies the knowledge. In every case, build the RAG pipeline and the evaluation set first, then fine-tune only the component the evals point at.

What are the common mistakes?

• Fine-tuning to teach facts. The model may learn phrasing and tone from your documents but will not reliably recall specifics, and it will drift as the documents change.
• Skipping evaluation. Without a labelled test set of real questions and expected answers, neither approach can be compared or declared good enough.
• Judging RAG by the model instead of the retriever. Most RAG failures are retrieval failures: wrong chunks, missing documents, poor ranking.
• Training on thin or synthetic-only data. A few hundred noisy examples usually make a model worse at everything except the training set.
• Ignoring access control. RAG must filter documents by the asking user's permissions at retrieval time; a fine-tuned model cannot forget what it was trained on.
• Not planning for re-training. Base models get deprecated; a fine-tune is tied to one version and must be redone when it goes away.

How does the choice play out in real use cases?

Customer support knowledge base

Start with RAG over help articles, past resolved tickets and product docs, with citations and a hand-off to a human when retrieval confidence is low. Add fine-tuning only if the output format or tone cannot be fixed by prompting, or if token cost at volume justifies a smaller tuned model.

Policy and compliance documents

RAG, almost without exception. Policies change, answers must quote the clause, and per-user access matters. Invest in document parsing, metadata filters by version and jurisdiction, and a reranker. Fine-tuning here is a liability because the model cannot be updated as fast as the policy.

Code assistant for an internal codebase

RAG over the repository, architecture docs and recent pull requests gives immediate value. Fine-tune only when house conventions or internal frameworks are unusual and the model keeps producing generic code, and refresh it as the code moves.

Brand voice and content generation

This is the strongest fine-tuning case. Hundreds of approved examples teach voice, structure and length more reliably than any style guide in a prompt. Pair it with RAG for product facts so the copy stays accurate.

Who should you hire for each?

RAG work needs an engineer who treats retrieval as a search problem: document parsing, chunking strategy, embeddings, hybrid search, reranking, evaluation of retrieval precision and recall, and access control. See hire RAG developers and RAG implementation services. Fine-tuning needs someone with hands-on training experience: dataset curation, LoRA and QLoRA, hosted fine-tuning APIs, evaluation harnesses and serving of tuned models. See hire LLM fine-tuning specialists. Before either, a strong prompt engineer can often close the gap with evals and prompt iteration.

Hyparz matches companies with pre-vetted remote AI engineers across RAG, fine-tuning and LLM application work, typically within days, with a one-week risk-free trial and a dedicated account manager, so you can start with the specialist your evaluation points at and add the other if the data says so.