Fine-Tuning AI Models: RLHF, DPO, and Modern Alignment Techniques

Fine-tuning pre-trained large language models has become a critical step in building production AI systems. This article provides a technical examination of modern fine-tuning methodologies, from traditional supervised approaches to advanced preference learning techniques like RLHF, DPO, and KTO. We analyze the mechanics of each method, compare their resource requirements and trade-offs, and provide practical guidance for selecting the appropriate technique based on your use case and computational budget.

Introduction

The paradigm shift from training models from scratch to fine-tuning pre-trained foundation models has dramatically democratized AI development. Starting from a base model like LLaMA, Mistral, or Qwen, practitioners can adapt these models to specific tasks with a fraction of the compute required for full training.

However, the landscape of fine-tuning techniques has expanded significantly beyond simple supervised learning. Modern methods now include reinforcement learning from human feedback (RLHF), direct preference optimization (DPO), Kullback-Leibler trophic optimization (KTO), and parameter-efficient techniques like LoRA. Understanding these methods—their assumptions, requirements, and limitations—is essential for making informed engineering decisions.

This article examines each technique in depth, providing the technical foundation you need to implement them effectively.

Supervised Fine-Tuning

Supervised fine-tuning (SFT) is the most straightforward approach: given a dataset of input-output pairs, continue training the model using standard next-token prediction loss.

How It Works

Given a dataset D = {(x₁, y₁), (x₂, y₂), ..., (xₙ, yₙ)} where x is the input prompt and y is the desired completion, SFT minimizes the negative log-likelihood:

L_SFT = -Σᵢ Σₜ log P(yᵢ,ₜ | xᵢ, yᵢ,; θ)

The model is pretrained on next-token prediction, so this loss is well-defined and training is stable. SFT essentially continues the pretraining process but on task-specific data.

When to Use SFT

SFT is appropriate when you have a clean, well-curated dataset of input-output examples for your target task. It's the baseline approach and often the first step before applying more advanced techniques.

Strengths:

• Simple to implement and debug
• Stable training with standard optimizers
• Works well with sufficient data
• No preference annotations required

Limitations:

• Requires high-quality demo data (the model learns exactly what you show it)
• Can regression on other tasks if not combined with other techniques
• Doesn't capture implicit preference information in the data

Reward Modeling

Reward modeling is an indirect approach to alignment that trains a model to score outputs, which can then be used to optimize the base model.

How It Works

Given comparisons or rankings of outputs (output A is better than output B), train a reward model R(x, y; φ) to predict the quality score. The model is trained using pairwise ranking loss:

L_RM = -log σ(R(x, y₊) - R(x, y₋))

where y₊ is the preferred output and y₋ is the less preferred one.

This is essentially learning a preference function. The reward model can then guide optimization, though direct reinforcement learning on the reward model often leads to reward hacking—optimizing the model to exploit patterns in the reward function rather than genuinely improving quality.

When to Use Reward Modeling

Reward modeling is useful when you have natural preference data (e.g., human comparisons) but not clean reference outputs. It's particularly valuable when different outputs have different strengths and weaknesses, making a single "correct" answer inappropriate.

Strengths:

• Works with preference/Comparison data rather than exact outputs
• Captures nuanced quality differences
• Can combine with RL for full RLHF pipeline

Limitations:

• Requires separate training stage
• Reward models can have biases and limitations
• Direct optimization leads to reward hacking without careful mitigation

RLHF: Reinforcement Learning from Human Feedback

RLHF combines reward modeling with reinforcement learning to produce models aligned with human preferences. It's the technique that made ChatGPT possible.

The RLHF Pipeline

RLHF operates in three stages:

1. Supervised Fine-Tuning: Train the base model on the target task with quality demonstrations
2. Reward Modeling: Train a separate reward model on human preference comparisons
3. Reinforcement Learning: Optimize the SFT model using PPO (Proximal Policy Optimization) with the reward model as the learning signal

The RL objective uses the reward model to evaluate outputs, with an additional KL penalty to prevent the model from drifting too far from the SFT baseline:

L_PPO = E[R(x, y) - β · KL(y||y_SFT)]

How PPO Works in RLHF

PPO constrains policy updates to be conservative, preventing catastrophic forgetting. The key mechanism is the clip objective that prevents the policy from changing too quickly:

L_clipped = min(r(θ) · A(θ), clip(r(θ), 1-ε, 1+ε) · A(θ))

where r(θ) is the probability ratio between the new and old policy, and A is the advantage estimate.

When to Use RLHF

RLHF is the gold standard for instruction-following and chat models where you want the model to produce outputs that humans find helpful. It's computationally expensive (requires three models in memory) but produces the highest quality results for open-ended generation.

Strengths:

• Produces genuinely helpful, aligned outputs
• Well-understood with extensive tooling
• Works with proxy preferences (trained on human feedback)

Limitations:

• Computationally expensive (3+ models in memory)
• Complex training pipeline
• Can be unstable without careful tuning
• Reward model limitations propagate through optimization

DPO: Direct Preference Optimization

DPO eliminates the reinforcement learning step entirely by directly optimizing the policy on preference data. It's a simplification that achieves comparable results with significantly less compute.

How It Works

DPO reformulates the RLHF objective mathematically to eliminate the need for RL. Starting from the RLHF optimality condition, DPO shows that the policy satisfying preferences can be found directly via a simple binary cross-entropy objective:

L_DPO = -log σ(β · (log π(y₊|x) - log π(y₋|x)))

The sign is flipped compared to standard classification because y₊ is the preferred (higher reward) output.

This is remarkably simple: it's just a weighted classification loss where the model predicts which of two outputs is preferred. No reward model, no RL, no KL penalty required.

When to Use DPO

DPO is ideal when you have preference data and want RLHF-quality results with simpler training. It requires less memory (one model instead of three) and is more stable.

Strengths:

• Much simpler pipeline (one-stage training)
• Requires less memory (single model)
• More stable training
• No hyperparameter tuning for RL

Limitations:

• Still requires preference data
• Performance can match but not exceed RLHF in some cases
• Implicitly assumes the preference pairs are independent of the policy

KTO: Kullback-Leibler Trophic Optimization

KTO is a newer technique that simplifies preference learning further by eliminating the need for pairwise comparisons entirely.

How It Works

KTO requires only unary feedback: is an output good or bad? It uses a simple binary classification objective:

L_KTO = -p_positive · log σ(f(x, y)) - p_negative · log(1 - σ(f(x, y)))

where σ is the sigmoid function and f is the model's scoring function.

The key insight is that this implicit alignment emerges automatically through the KL divergence constraint. The model learns to produce outputs that maximize the probability of positive examples while maintaining similarity to the base model.

When to Use KTO

KTO is the most accessible preference learning method—it works with simple positive/negative labels rather than paired comparisons. It's ideal when you have binary feedback (thumbs up/down) rather than comparative data.

Strengths:

• Works with unary feedback (no pairs needed)
• Simplest data collection requirements
• Single-stage training
• No reward model needed

Limitations:

• Newer with less established track record
• Requires careful balancing of positive/negative examples
• May not capture nuanced preferences as well as pairwise methods

LoRA: Low-Rank Adaptation

LoRA is not a preference learning technique—it's a parameter-efficient fine-tuning method that can be combined with any of the above approaches. It dramatically reduces the compute requirements for fine-tuning.

How It Works

LoRA injects trainable low-rank matrices into each transformer attention layer. Instead of updating the full weight matrix W ∈ ℝᵈᵈ, LoRA adds a low-rank decomposition:

W' = W + BA

where B ∈ ℝᵈʳ and A ∈ ℝʳᵈ with r << d.

During training, only the LoRA parameters (B and A) are updated. At inference, the adapted matrix can be merged back into the original weights. The rank r (typically 8-32) controls the capacity of the adaptation.

When to Use LoRA

LoRA is essential when compute resources are limited. It reduces fine-tuning memory requirements by 2-3× and enables fine-tuning of larger models on consumer hardware.

Strengths:

• Dramatically reduced memory requirements
• Can combine with any fine-tuning method
• Multiple adapters can be swapped for different tasks
• Fine-tuned model maintains full inference capability

Limitations:

• Adds latency if not merged (minimal for most applications)
• May not capture all the capacity of full fine-tuning
• Requires careful rank selection

Comparing Fine-Tuning Methods

Method	Data Requirement	Memory (Training)	Complexity	Best For
SFT	Input-output pairs	High	Low	Well-defined tasks with clean data
Reward Modeling	Preference comparisons	Medium	Medium	Learning implicit quality functions
RLHF	Preference comparisons	Very High (3+ models)	High	Maximum quality instruction following
DPO	Preference comparisons	Medium	Low	RLHF-quality with less compute
KTO	Binary feedback	Medium	Low	Thumbs up/down feedback available
LoRA (add-on)	Any above	~2-3× lower	Low	Compute-constrained environments

Decision Framework

Use SFT when:

• You have high-quality demonstration data
• The task has a clear correct output
• Compute is not a constraint

Use DPO when:

• You have preference comparisons
• You want RLHF-quality with simpler training
• Memory is constrained

Use KTO when:

• You have binary feedback (not comparisons)
• Data collection is expensive
• Simplicity is paramount

Use RLHF when:

• You need the highest quality outputs
• You have extensive preference data
• Compute is not a constraint

Add LoRA when:

• Training on consumer hardware
• Multiple task adaptations needed
• Memory is limited

Practical Considerations

Data Requirements

All preference learning methods benefit from diverse, representative preference data. For DPO and RLHF, you'll typically need 1,000-10,000 preference pairs for meaningful improvement. KTO requires similar quantities of positive/negative examples.

Quality matters more than quantity—a smaller dataset with clear preference signals outperforms a large dataset with noisy or inconsistent preferences.

Compute Requirements

Training a 7B model with RLHF requires roughly 16-24GB of GPU memory for the model, value head, and optimizer states. DPO reduces this to ~8-12GB. LoRA brings this down further to ~6-8GB.

For larger models (70B+), LoRA is essentially required for most practitioners.

Training Stability

RLHF is notoriously unstable without careful learning rate scheduling and KL coefficient tuning. DPO and KTO are significantly more stable and require less hyperparameter tuning.

Conclusion

Fine-tuning methodology has evolved significantly beyond simple supervised learning. The choice of technique depends on your data, compute budget, and quality requirements:

• SFT remains the foundation for well-defined tasks with clean data
• RLHF produces the highest quality aligned models but at significant compute cost
• DPO offers a practical middle ground with RLHF-quality results and simpler training
• KTO democratizes preference learning by working with simple binary feedback
• LoRA enables fine-tuning on constrained hardware by reducing memory requirements

For most practitioners, the recommended approach is SFT followed by DPO with LoRA, combining the stability of supervised learning with efficient preference optimization. This pipeline produces competitive results at a fraction of the traditional RLHF compute cost.

As the field continues to evolve, expect further simplifications and efficiencies. The trend is clear: making high-quality model alignment accessible to more practitioners with fewer resources.

References

• Ouyang, L., et al. (2022). Training language models to follow instructions with human feedback. NeurIPS.
• Rafailov, D., et al. (2024). Direct Preference Optimization: Your Language Model is a Reward Model. NeurIPS.
• Ethayarahh, A., et al. (2024). KTO: Model Alignment via Direct Optimization. arXiv preprint.
• Hu, E. J., et al. (2022). LoRA: Low-Rank Adaptation of Large Language Models. ICLR.