11 — Grounding and References

The RAFT paper, the libraries used in the example run, and how the numbers in this guide were produced.

Christian Pojoni 2 min read
Share

This guide describes one complete run. The method follows the RAFT paper, and the numbers come from that run’s reports. They will differ on your data.

Method

Libraries and Tools

  • Unsloth: model loading, LoRA, train_on_responses_only and the 16-bit merge.
  • TRL SFTTrainer: the supervised fine-tuning loop.
  • PEFT: the adapter implementation underneath.
  • llama.cpp: GGUF conversion and quantization.
  • Ollama: local serving and the Modelfile format.
  • Qwen3.5-9B on Hugging Face: the base model of the larger run.
  • Open WebUI: the chat UI in front of the serving wrapper.
  • RunPod: the rented GPU for the 9B run.

How the Numbers Were Produced

  • Corpus: about 870 videos of one creator, transcribed locally with Whisper. Training used solo videos only.
  • Eval set: 129 reviewed questions (103 answerable, 26 unanswerable). Every document referenced by eval evidence was held out from training.
  • Retrieval: one local service, k = 8 passages for answers, hit@10 = 0.738 on the eval set.
  • Teacher and judge: qwen3.5:9b, run locally. The judge being the teacher is a known bias.
  • Training data: 7,988 attempts, 3,012 kept, 2,858 training and 144 validation examples after export filters.
  • Models: Qwen3.5-4B (RTX 5070 Ti, 16 GB) and Qwen3.5-9B (RTX 6000 Ada, 48 GB), BF16 LoRA rank 16, 2 epochs, served as GGUF Q4_K_M through Ollama.

Companion Post

The operational story of the run, including what broke, is in My First RAFT Fine-Tune on a Shared 16 GB GPU: 7 Gotchas the Paper Doesn’t Mention.

Questions about the writing?

Ask about an article, a project, or the ideas behind it.