All guides
RAFT Fine-Tuning Guide
A practical path from a working RAG pipeline to a small local model fine-tuned with RAFT: data, filtering, training, serving and an honest eval.
Practical chapters on C++, TypeScript and RAFT fine-tuning.
- 012 min read
Mental Model: What RAFT Teaches and What It Does Not
RAFT teaches a model how to use retrieved context. It does not teach it new facts, and knowing the difference decides whether it is worth doing.
- 022 min read
Before You Train: Retrieval, Eval Set and Holdout
Measure retrieval, build an eval set with a holdout, and record baseline numbers for the untuned model before you generate a single training example.
- 033 min read
Generating Training Data with a Teacher
Self-distill answers from a stronger local teacher, in exactly the inference format, with oracle and distractor-only contexts and hints the student never sees.
- 042 min read
Filtering and the Datasheet
Rejection-sample the teacher's answers with explicit checks, count every drop reason, and read a datasheet before you train.
- 052 min read
Citations Belong in Code
Let the model cite passage ids and verbatim quotes, then verify the quotes and generate timestamps and links in code.
- 062 min read
Training a LoRA Adapter
BF16 LoRA with Unsloth and TRL, loss on the assistant reply only, a token length check before training, and checkpoints that survive a shared GPU.
- 072 min read
Export and Serving
Merge the adapter, quantize to GGUF, register the model and an equally packaged base model in Ollama, and serve the full pipeline behind an OpenAI-compatible endpoint.
- 083 min read
Evaluating Against the Base Model
Identical retrieval for every model, metrics that separate knowledge from style, a paired comparison, and a look at every false abstention.
- 093 min read
Renting a GPU for the Bigger Model
What to upload, how to estimate cost, and the operational traps of unattended cloud training: slow mirrors, silent watchers and volumes tied to one host.
- 101 min read
Checklist
A one-page checklist for a RAFT run, from retrieval baseline to shipped model.
- 112 min read
Grounding and References
The RAFT paper, the libraries used in the example run, and how the numbers in this guide were produced.
About this guide
A compact guide for engineers who already have a retrieval pipeline and want a small local model that answers from retrieved passages better than a prompted base model does.
RAFT (Retrieval-Augmented Fine-Tuning) trains a model on questions paired with retrieved passages, some relevant and some distractors, so it learns to answer from the right passage, ignore the rest and say so when nothing fits. The training itself is a standard LoRA run. The work is in the data, the checks and the evaluation, and that is what this guide spends its pages on.
The running example is a real project: about 870 transcribed videos of a single creator, a local retrieval service, and Qwen3.5 4B and 9B models fine-tuned to answer in the creator’s first-person voice with a verified citation for every sentence. The 4B model trained on a 16 GB consumer GPU. The 9B model trained on a rented 48 GB card. The war stories from that run are in the companion post My First RAFT Fine-Tune on a Shared 16 GB GPU.
How to Use This Guide
Read the pages in order. Each one answers four questions:
- What problem does this step solve?
- What is the default choice?
- What fails silently if you skip it?
- How do you check that it worked?
What Is Genuinely New
If you have built RAG systems, most of this is familiar. Only a few ideas are specific to RAFT:
- The model learns from contexts that contain distractors, and from contexts where the answer is missing entirely.
- Training examples must use exactly the prompt format of inference, built from the same retriever.
- A stronger teacher can see hints at generation time that the student never sees.
- The gain is mostly in how the model uses context and how it sounds, not in what it knows.
Recommended Path
Phase 1: Foundations
- Mental Model: What RAFT Teaches and What It Does Not
- Before You Train: Retrieval, Eval Set and Holdout
Phase 2: Data
Phase 3: Model
Phase 4: Proof
- Evaluating Against the Base Model
- Renting a GPU for the Bigger Model
- Checklist
- Grounding and References
Fastest Practical Plan
A focused week is enough for a first run if your retrieval already works.
- Day 1: eval set, holdout and baseline numbers for the untuned model.
- Days 2 and 3: data generation with filters, then read the datasheet and fix the filters.
- Day 4: train a small model and export it.
- Day 5: evaluate against the base model with identical retrieval and write down what changed.
Default Target Stack
If you do not know where to start, aim for this combination:
Qwen3.5 4B + Unsloth BF16 LoRA + TRL SFTTrainer with response-only loss + GGUF Q4_K_M + Ollama, with a local teacher model for data generation.
It fits a 16 GB GPU, trains in a few hours and keeps every transcript on your machine.
Grounding
This is a field guide from one complete run, not a survey of the literature. Numbers are from that run and will differ on your data. For the paper, library documentation and version notes, see Grounding and References.