Three stages from pretraining to human-aligned LLM
RLHF (Reinforcement Learning from Human Feedback) is a three-step process. Stage 1, SFT (Supervised Fine-Tuning), takes a pretrained LLM and fine-tunes it on high-quality examples (e.g., 'good' responses to prompts), teaching it the baseline style and format. Stage 2, Reward Model, trains a separate model to score responses on a human-preference scale. Stage 3, PPO or DPO, uses the reward model to guide policy optimization, rewarding responses humans prefer and punishing those they don't.
This pipeline is necessary because human preferences are diverse and hard to capture in a loss function. Pretraining optimizes for prediction accuracy (next token); RLHF optimizes for usefulness and safety according to human judgment.
The reward model as a proxy for human judgment
The reward model is trained on pairwise human comparisons: given two responses, humans rank them. The model learns to output high scores for preferred responses. This avoids expensive per-response annotation (which would require a score for every possible output), scaling to millions of training examples.
Quality of the reward model directly impacts the final LLM: a weak reward model with human-annotation noise can train a jailbroken or unaligned policy.
PPO and DPO tradeoffs
PPO (Proximal Policy Optimization) is the original algorithm but is sample-inefficient and complex to tune. DPO (Direct Preference Optimization) avoids training a separate reward model entirely, learning directly from preference pairs. DPO is simpler and faster but assumes preference pairs are stationary (humans don't change their minds).