ReviseAlgo Logo

Fine-Tuning & Model Alignment

QLoRA & Parameter-Efficient Fine-Tuning

Low-rank adaptation, 4-bit quantization training, and Hugging Face PEFT.

Interview: High - Core skill for adapting open-source models to domain tasks.

Full Fine-Tuning is Dead For Most Use Cases

QLoRA Architecture

Full fine-tuning updates all 70 billion parameters simultaneously. At bfloat16, that requires ~448GB of GPU VRAM just for the optimizer states — a $200,000 infrastructure problem. LoRA sidesteps this entirely with a mathematical insight: weight updates during fine-tuning have low intrinsic rank. Instead of updating W (a large matrix), you learn two tiny matrices A and B where W' = W + AB. You only train ~0.1% of the total parameters.

QLoRA: 4-bit Quantization + LoRA

QLoRA goes further: it quantizes the frozen base model weights to 4-bit NF4 format (near lossless) and trains the LoRA adapters in bfloat16. This enables fine-tuning a Llama 3 70B model on a single A100 80GB GPU — a 10x VRAM reduction vs. full fine-tuning.

LoRA Hyperparameters That Actually Matter

  • r (rank): Size of the low-rank decomposition. r=8 for general tasks, r=64 for complex domain adaptation. Higher r = more parameters, more expressivity, more overfitting risk.
  • alpha: Scaling factor (typically set to 2*r). Controls the magnitude of LoRA weight updates.
  • target_modules: Which layers to apply LoRA to. Apply to all attention, gate, up, down projection layers for best results.

Use Cases

Fine-tuning Llama/Mistral to follow your company's specific JSON output format reliably

Domain adaptation for medical/legal/finance where generic LLMs lack vocabulary and reasoning patterns

Teaching a model your proprietary coding style or internal DSL

Common Mistakes

Using r=256 for a simple classification task — massive overfitting and unnecessary VRAM usage. Start with r=8 or r=16.

Not applying LoRA to gate_proj/up_proj/down_proj (FFN layers) — these layers are crucial for factual knowledge injection

Forgetting to call merge_and_unload() before deployment — serving with PEFT adapters adds latency overhead