11 — Grounding and References
The RAFT paper, the libraries used in the example run, and how the numbers in this guide were produced.
This guide describes one complete run. The method follows the RAFT paper, and the numbers come from that run’s reports. They will differ on your data.
Method
- RAFT: Adapting Language Model to Domain Specific RAG (Zhang et al., 2024). The paper that defines training with distractor documents and verbatim citing.
- LoRA: Low-Rank Adaptation of Large Language Models (Hu et al., 2021).
Libraries and Tools
- Unsloth: model loading, LoRA,
train_on_responses_onlyand the 16-bit merge. - TRL SFTTrainer: the supervised fine-tuning loop.
- PEFT: the adapter implementation underneath.
- llama.cpp: GGUF conversion and quantization.
- Ollama: local serving and the Modelfile format.
- Qwen3.5-9B on Hugging Face: the base model of the larger run.
- Open WebUI: the chat UI in front of the serving wrapper.
- RunPod: the rented GPU for the 9B run.
How the Numbers Were Produced
- Corpus: about 870 videos of one creator, transcribed locally with Whisper. Training used solo videos only.
- Eval set: 129 reviewed questions (103 answerable, 26 unanswerable). Every document referenced by eval evidence was held out from training.
- Retrieval: one local service,
k = 8passages for answers,hit@10 = 0.738on the eval set. - Teacher and judge:
qwen3.5:9b, run locally. The judge being the teacher is a known bias. - Training data: 7,988 attempts, 3,012 kept, 2,858 training and 144 validation examples after export filters.
- Models: Qwen3.5-4B (RTX 5070 Ti, 16 GB) and Qwen3.5-9B (RTX 6000 Ada, 48 GB), BF16 LoRA rank 16, 2 epochs, served as GGUF Q4_K_M through Ollama.
Companion Post
The operational story of the run, including what broke, is in My First RAFT Fine-Tune on a Shared 16 GB GPU: 7 Gotchas the Paper Doesn’t Mention.