All guides

RAFT Fine-Tuning Guide

A practical path from a working RAG pipeline to a small local model fine-tuned with RAFT: data, filtering, training, serving and an honest eval.

Practical chapters on C++, TypeScript and RAFT fine-tuning.

  1. 01

    Mental Model: What RAFT Teaches and What It Does Not

    RAFT teaches a model how to use retrieved context. It does not teach it new facts, and knowing the difference decides whether it is worth doing.

    2 min read
  2. 02

    Before You Train: Retrieval, Eval Set and Holdout

    Measure retrieval, build an eval set with a holdout, and record baseline numbers for the untuned model before you generate a single training example.

    2 min read
  3. 03

    Generating Training Data with a Teacher

    Self-distill answers from a stronger local teacher, in exactly the inference format, with oracle and distractor-only contexts and hints the student never sees.

    3 min read
  4. 04

    Filtering and the Datasheet

    Rejection-sample the teacher's answers with explicit checks, count every drop reason, and read a datasheet before you train.

    2 min read
  5. 05

    Citations Belong in Code

    Let the model cite passage ids and verbatim quotes, then verify the quotes and generate timestamps and links in code.

    2 min read
  6. 06

    Training a LoRA Adapter

    BF16 LoRA with Unsloth and TRL, loss on the assistant reply only, a token length check before training, and checkpoints that survive a shared GPU.

    2 min read
  7. 07

    Export and Serving

    Merge the adapter, quantize to GGUF, register the model and an equally packaged base model in Ollama, and serve the full pipeline behind an OpenAI-compatible endpoint.

    2 min read
  8. 08

    Evaluating Against the Base Model

    Identical retrieval for every model, metrics that separate knowledge from style, a paired comparison, and a look at every false abstention.

    3 min read
  9. 09

    Renting a GPU for the Bigger Model

    What to upload, how to estimate cost, and the operational traps of unattended cloud training: slow mirrors, silent watchers and volumes tied to one host.

    3 min read
  10. 10

    Checklist

    A one-page checklist for a RAFT run, from retrieval baseline to shipped model.

    1 min read
  11. 11

    Grounding and References

    The RAFT paper, the libraries used in the example run, and how the numbers in this guide were produced.

    2 min read
About this guide

A compact guide for engineers who already have a retrieval pipeline and want a small local model that answers from retrieved passages better than a prompted base model does.

RAFT (Retrieval-Augmented Fine-Tuning) trains a model on questions paired with retrieved passages, some relevant and some distractors, so it learns to answer from the right passage, ignore the rest and say so when nothing fits. The training itself is a standard LoRA run. The work is in the data, the checks and the evaluation, and that is what this guide spends its pages on.

The running example is a real project: about 870 transcribed videos of a single creator, a local retrieval service, and Qwen3.5 4B and 9B models fine-tuned to answer in the creator’s first-person voice with a verified citation for every sentence. The 4B model trained on a 16 GB consumer GPU. The 9B model trained on a rented 48 GB card. The war stories from that run are in the companion post My First RAFT Fine-Tune on a Shared 16 GB GPU.

How to Use This Guide

Read the pages in order. Each one answers four questions:

  1. What problem does this step solve?
  2. What is the default choice?
  3. What fails silently if you skip it?
  4. How do you check that it worked?

What Is Genuinely New

If you have built RAG systems, most of this is familiar. Only a few ideas are specific to RAFT:

  • The model learns from contexts that contain distractors, and from contexts where the answer is missing entirely.
  • Training examples must use exactly the prompt format of inference, built from the same retriever.
  • A stronger teacher can see hints at generation time that the student never sees.
  • The gain is mostly in how the model uses context and how it sounds, not in what it knows.

Phase 1: Foundations

  1. Mental Model: What RAFT Teaches and What It Does Not
  2. Before You Train: Retrieval, Eval Set and Holdout

Phase 2: Data

  1. Generating Training Data with a Teacher
  2. Filtering and the Datasheet
  3. Citations Belong in Code

Phase 3: Model

  1. Training a LoRA Adapter
  2. Export and Serving

Phase 4: Proof

  1. Evaluating Against the Base Model
  2. Renting a GPU for the Bigger Model
  3. Checklist
  4. Grounding and References

Fastest Practical Plan

A focused week is enough for a first run if your retrieval already works.

  • Day 1: eval set, holdout and baseline numbers for the untuned model.
  • Days 2 and 3: data generation with filters, then read the datasheet and fix the filters.
  • Day 4: train a small model and export it.
  • Day 5: evaluate against the base model with identical retrieval and write down what changed.

Default Target Stack

If you do not know where to start, aim for this combination:

Qwen3.5 4B + Unsloth BF16 LoRA + TRL SFTTrainer with response-only loss + GGUF Q4_K_M + Ollama, with a local teacher model for data generation.

It fits a 16 GB GPU, trains in a few hours and keeps every transcript on your machine.

Grounding

This is a field guide from one complete run, not a survey of the literature. Numbers are from that run and will differ on your data. For the paper, library documentation and version notes, see Grounding and References.

Questions about the writing?

Ask about an article, a project, or the ideas behind it.