06 — Training a LoRA Adapter

BF16 LoRA with Unsloth and TRL, loss on the assistant reply only, a token length check before training, and checkpoints that survive a shared GPU.

Christian Pojoni 2 min read
Share

Core Idea

The training run is the most standard part of RAFT. Use known-good defaults, check the inputs before you spend GPU hours, and make the run resumable.

Check Before You Train

Run a CPU-only check over the training files first:

  • token length per example under the chat template, with the median and how many exceed max_seq_length
  • that the loss mask covers only the assistant reply

RAFT prompts are long because they carry passages. The example used max_seq_length = 6144. Examples above it were dropped, not truncated, since a truncated context can cut off the passage the answer cites.

Default Configuration

These settings trained both example models without tuning:

model, tok = FastLanguageModel.from_pretrained(
    "Qwen/Qwen3.5-4B", max_seq_length=6144, dtype=torch.bfloat16, load_in_4bit=False)
model = FastLanguageModel.get_peft_model(
    model, r=16, lora_alpha=16, lora_dropout=0, target_modules=TARGET_MODULES,
    use_gradient_checkpointing="unsloth", random_state=3407)
args = SFTConfig(
    per_device_train_batch_size=1, gradient_accumulation_steps=16,
    num_train_epochs=2, learning_rate=2e-4, lr_scheduler_type="linear",
    warmup_steps=0.03, weight_decay=0.001, optim="adamw_8bit",
    max_length=6144, packing=False, eval_steps=50, save_steps=50)
trainer = train_on_responses_only(trainer, instruction_part=INSTRUCTION_PART,
                                  response_part=RESPONSE_PART)

train_on_responses_only matters. Without it, most of the loss is spent predicting the passages, which teaches the model nothing about answering.

Resource Numbers

ModelGPUPeak VRAMStepsTimeFinal eval loss
Qwen3.5-4BRTX 5070 Ti 16 GB10.8 GB3582 h 49 min0.190
Qwen3.5-9BRTX 6000 Ada 48 GB25.1 GB3583 h 35 min0.139

BF16 LoRA rather than 4-bit QLoRA was a deliberate choice. The 4B model fits in BF16 on 16 GB, and it avoids quantization noise in the comparison with the base model.

Shared GPUs

If other jobs use the same card, set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True, save checkpoints every 50 steps, and make train resume from the latest checkpoint when it is started again with the same output directory. An out-of-memory crash then costs minutes, not hours. Refuse to start below a free-memory threshold instead of crashing mid-epoch.

Reading Eval Loss

Eval loss tells you the model is learning the format and is not overfitting. It does not tell you whether answers got better. Two epochs gave steadily falling eval loss in the example (0.154, 0.146, 0.143, 0.142 at steps 50 to 200 on 9B). The decision to keep the model came only from the task eval.

Common Mistakes

  • Computing loss on the whole prompt.
  • Truncating long RAFT prompts instead of dropping them.
  • Reading a lower eval loss as a better model.

Questions about the writing?

Ask about an article, a project, or the ideas behind it.