DPO & RLHF: Aligning LLMs to Human Preferences

Fine-Tuning Makes Models Capable. Alignment Makes Them Useful.

DPO vs RLHF

A model fine-tuned on code will write code. But will it refuse to write malware? Will it explain its reasoning? Will it prefer concise answers over verbose ones? Alignment post-training (RLHF or DPO) is what separates raw capability from commercially deployable behavior.

RLHF: The Hard Way

RLHF (Reinforcement Learning from Human Feedback) requires training a separate Reward Model (RM) on human preference data, then using PPO (Proximal Policy Optimization) to optimize the LLM's policy against the RM's score. This requires running 3 models simultaneously during training (policy, reference, reward) — extremely expensive and unstable. This is how GPT-4 was aligned.

DPO: The Elegant Shortcut

DPO (Direct Preference Optimization) was a 2023 breakthrough that eliminates the reward model entirely. Instead of training a separate RM, DPO directly trains the LLM using preference pairs (chosen response vs. rejected response) with a closed-form classification loss. The math shows that the optimal policy can be expressed directly in terms of the preference data — no RL loop required.

DPO is now the industry standard for alignment at most labs. It is simpler to implement, more stable to train, and achieves comparable quality to PPO-RLHF for instruction following.

Synthetic Preference Data with Constitutional AI

Anthropic's Constitutional AI approach eliminates expensive human labelers for preference data. A "helpfulness" model generates a response. A "critique" model evaluates it against a set of principles (the constitution). The critique model rewrites the response to better satisfy the principles. The original vs. revised response becomes a (rejected, chosen) preference pair. This generates millions of alignment pairs automatically.