Fine-Tuning Open Source Models

Prompt engineering is fast, cheap, and reversible. For most tasks it is the right starting point. Fine-tuning is slower, more expensive, and harder to reverse. It addresses the specific problems prompting cannot solve: consistent output formats without multi-hundred-token instructions, specialised domain knowledge not present in the base model, dramatically reduced inference costs at high volume, and fully local deployments with no API dependency.

The key word is "cannot." If a well-crafted prompt reliably produces the output you need, fine-tuning adds cost and complexity without benefit. Exhaust prompting first. Fine-tuning is the answer when you have tried and the model demonstrably fails on your specific task distribution.

Full fine-tuning updates every weight in the model. A 7-billion-parameter model requires approximately 28 GB of GPU memory to store weights at FP32 (32-bit floating point) precision, plus additional memory for activations, gradients, and optimiser states. Total memory requirement: over 100 GB. This demands expensive multi-GPU clusters.

PEFT (Parameter-Efficient Fine-Tuning) methods update only a small fraction of weights while keeping the vast majority frozen. The most widely used PEFT technique is LoRA (Low-Rank Adaptation). Instead of updating the full weight matrix W, LoRA inserts two small trainable matrices A and B alongside it. During the forward pass, the effective weight becomes W + AB. Only A and B are updated during training.

At rank r=8, the number of trainable parameters drops by roughly 1,750x compared to full fine-tuning. For a 7B model: instead of 7 billion trainable parameters, you update roughly 4 million. Memory requirement: around 16 GB with standard LoRA, and under 8 GB with QLoRA.

QLoRA (Quantised LoRA), published by Dettmers et al. in 2023, adds one more step: it compresses the frozen base model weights to 4-bit precision before training. This reduces memory by roughly 4x compared to standard LoRA. The trainable adapter matrices still operate in higher precision (bfloat16), so training quality is maintained. QLoRA brings a 7B fine-tuning run under 8 GB, which puts it within reach of a single consumer GPU such as the NVIDIA RTX 3090 or 4090, or of a free Google Colab session with a T4 GPU.

Dataset quality determines fine-tuning quality more than any hyperparameter or architecture choice. The model can only learn patterns present in the training data. Garbage in, garbage out applies with particular force to fine-tuning, because the model will faithfully reproduce not just the style but the errors and inconsistencies in your examples.

The standard format for instruction fine-tuning pairs each example with an instruction, optional input, and the desired output. The model learns to follow the instruction pattern rather than the specific content. Training on 500 diverse, high-quality examples consistently outperforms training on 5,000 low-quality ones.

Before training, audit your dataset for these critical properties. Every example should have an instruction that is specific and consistent in format. Outputs must be correct and ideally reviewed by a domain expert. No example should contain personally identifiable information (PII). Edge cases and failure modes should be explicitly represented, not just easy, clean examples. Split the dataset into training, validation, and test sets at an 80/10/10 ratio minimum.

The Hugging Face ecosystem provides the standard Python libraries for LoRA and QLoRA fine-tuning. You need four packages: transformers (model loading and tokenisation), peft (the PEFT library, which implements LoRA),trl (the TRL library, which provides SFTTrainer for supervised fine-tuning), and bitsandbytes (4-bit quantisation for QLoRA).

The training configuration involves three key choices. First, the LoRA rank (r): higher rank means more trainable parameters and potentially higher quality, but more memory and slower training. Rank 8 to 16 is typical for most tasks. Second, the target modules: which transformer layers to add adapters to. The query and value projection layers (q_proj, v_proj) are standard choices. Third, the learning rate: 2e-4 is a common starting point for LoRA fine-tuning.

After training, evaluate on your held-out test set using task-specific metrics. Training loss decreasing is necessary but not sufficient. For extraction tasks, measure precision, recall, and F1 score. For generation, use ROUGE-L (overlap with reference outputs) and human evaluation. For JSON output tasks, measure schema validity rate and field-level accuracy. Compare the fine-tuned model against the base model and against prompting on the same test set.

Use prompting when your volume is low to medium, the base model can produce acceptable outputs with detailed instructions, and you need results immediately. Choose fine-tuning when you face high request volume (10,000 or more calls per day), the model fails consistently even with careful prompting, you need exact and consistent output formats, inference latency must decrease, or data privacy requirements prevent use of external APIs.

One common confusion: fine-tuning is not a cure for hallucination. A fine-tuned model can produce confident, well-formatted, completely incorrect output. Fine-tuning teaches style, format, and domain conventions. For factual accuracy, combine fine-tuning with retrieval-augmented generation (RAG), which provides the model with retrieved evidence at inference time rather than baking facts into weights.

EU AI Act Article 53 imposes baseline obligations on all providers of general-purpose AI (GPAI) models, and Article 55 adds further obligations for providers of GPAI models with systemic risk. If you fine-tune an open-source model and offer it to third parties, assess whether these obligations apply. Hugging Face model card standards require documenting training data, intended use, and known limitations before publishing a fine-tuned model to the Hub.