π§ Beyond Maximum Likelihood: Steering Token Probabilities via Policy Gradients
While Supervised Fine-Tuning (SFT) trains models to imitate human-authored text, it suffers from exposure bias and treats all errors equally: a catastrophic factual hallucination incurs the same token-level cross-entropy penalty as a benign stylistic synonym. To optimize holistic objectives such as helpfulness, harmlessness, mathematical rigor, and conciseness, alignment transitions into Reinforcement Learning from Human Feedback (RLHF). In this regime, an autoregressive language model is framed as a stochastic policy $\pi_\theta$ operating within a sequential Markov Decision Process (MDP). Using Proximal Policy Optimization (PPO) (Schulman et al., 2017) as formalized by OpenAI in InstructGPT, the policy maximizes scalar scores provided by a trained Reward Model while being anchored by a token-level Kullback-Leibler (KL) Divergence Penalty to prevent distribution collapse and reward hacking. By combining conservative surrogate objective clipping with Generalized Advantage Estimation (GAE), PPO establishes stable, non-destructive gradient descent updates across discrete token action spaces.
THE LANGUAGE GENERATION MARKOV DECISION PROCESS (MDP):
Tuple Definition: M = ( S, A, P_env, R, Ξ³ )
1. STATE SPACE (s_t β S):
The state at timestep t is the concatenation of the conditioning prompt x
and all previously generated tokens y_{), it emits scalar score r_RM(x, y).
β’ Dense Reward Formulation: KL divergence penalty is applied at EVERY intermediate step:
R_t = - Ξ² Β· D_KL( Ο_ΞΈ(Β·|s_t) || Ο_ref(Β·|s_t) ) for t < T
R_T = r_RM(x, y) - Ξ² Β· D_KL( Ο_ΞΈ(Β·|s_T) || Ο_ref(Β·|s_T) )
THE FULL CONSTRAINED RLHF OPTIMIZATION OBJECTIVE (Ouyang et al., 2022 - InstructGPT):
max_ΞΈ obj(ΞΈ) = E_{(x, y) ~ D_Ο_ΞΈ} [ r_Ο(x, y) ]
- Ξ² Β· D_KL( Ο_ΞΈ(y | x) || Ο_ref(y | x) )
+ Ξ³ Β· E_{x ~ D_pretrain} [ log Ο_ΞΈ(x) ]
WHERE:
β’ r_Ο(x, y) : Trained scalar reward model predicting human preference.
β’ Ο_ΞΈ : The active trainable language model policy being optimized.
β’ Ο_ref : The FROZEN Supervised Fine-Tuned (SFT) reference model.
β’ Ξ² (beta) : The KL penalty hyperparameter (typically Ξ² β [0.01, 0.1]).
β’ Ξ³ (gamma) : PPO-ptx coefficient mixing unsupervised pre-training gradients to prevent
degradation on public benchmarks (e.g. SQuAD, drop, translation).
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
EXPANDING THE TOKEN-LEVEL KL DIVERGENCE COMPONENT:
The sequence-level KL divergence decomposes into an exact sum over token positions t:
D_KL( Ο_ΞΈ(y | x) || Ο_ref(y | x) ) = E_{y ~ Ο_ΞΈ} β‘ log β Ο_ΞΈ(y | x) β β€
β’ β ββββββββββββββ β β₯
β£ β Ο_ref(y | x) β β¦
= β_{t=1}^T E_{y_t ~ Ο_ΞΈ} β‘ log Ο_ΞΈ(y_t | x, y_{
THE 4-MODEL PPO RUNTIME COORDINATION GRID:
A standard LLM PPO iteration requires FOUR concurrent foundation models loaded in memory:
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β MODEL 1: ACTIVE ACTOR / POLICY MODEL (Ο_ΞΈ) β
β β’ Status : TRAINABLE (Parameters ΞΈ updated via PPO gradient descent). β
β β’ VRAM Cost : Weights (FP16/BF16) + Gradients + Optimizer States (AdamW) = ~16B bytes β
β β’ Primary Role: Ingests prompt x; generates response y rollout autoregressively. β
ββββββββββββββββββββββββββββββββββββββββββββ²ββββββββββββββββββββββββββββββββββββββββββββββ
β Log-Probabilities log Ο_ΞΈ(y_t | s_t)
β
ββββββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββββββββββββββββββββββββββββββββ
β MODEL 2: REFERENCE MODEL (Ο_ref) β
β β’ Status : COMPLETELY FROZEN (No gradients, no optimizer states). β
β β’ VRAM Cost : Weights only in FP16 or 4-bit/8-bit (e.g. 2 to 4 bytes per param). β
β β’ Primary Role: Computes reference log-probabilities log Ο_ref(y_t | s_t) for KL term. β
ββββββββββββββββββββββββββββββββββββββββββββ¬ββββββββββββββββββββββββββββββββββββββββββββββ
β
βΌ KL Penalty Calculation: [ log Ο_ΞΈ - log Ο_ref ]
β
ββββββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββββββββββββββββββββββββββββββββ
β MODEL 3: CRITIC / VALUE MODEL (V_Ο) β
β β’ Status : TRAINABLE (Parameters Ο updated via Mean Squared Error value loss). β
β β’ Architecture: Same backbone as Actor, but terminates in scalar head (R^d β R^1). β
β β’ Primary Role: Estimates state value V_Ο(s_t), predicting expected return from step t.β
ββββββββββββββββββββββββββββββββββββββββββββ¬ββββββββββββββββββββββββββββββββββββββββββββββ
β
βΌ Baseline Subtraction for GAE Advantage A_t
β
ββββββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββββββββββββββββββββββββββββββββ
β MODEL 4: REWARD MODEL (r_Ο) β
β β’ Status : COMPLETELY FROZEN (Trained via Bradley-Terry preference loss). β
β β’ VRAM Cost : Weights only in FP16 or 4-bit (No gradients, no optimizer states). β
β β’ Primary Role: Scores completed sequence (x, y), emitting terminal score r_Ο(x, y). β
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
VRAM HARDWARE FOOTPRINT (70B MODEL BASELINE):
β’ Actor Model (Trainable) : ~140 GB (Weights) + 140 GB (Grads) + 560 GB (AdamW) = 840 GB
β’ Critic Model (Trainable) : ~840 GB (Full optimizer states)
β’ Reference Model (Frozen) : ~140 GB (In FP16) or 35 GB (In 4-bit NF4)
β’ Reward Model (Frozen) : ~140 GB (In FP16) or 35 GB (In 4-bit NF4)
TOTAL CLUSTER REQUIREMENT : Over 1.8 Terabytes of VRAM! Demands DeepSpeed ZeRO-3 / FSDP!
GENERALIZED ADVANTAGE ESTIMATION (GAE-Ξ») MATHEMATICAL FORMULATION:
PURPOSE OF THE ADVANTAGE FUNCTION A(s, a):
Quantifies how much better action a_t is compared to the default expected return from state s_t:
A(s_t, a_t) = Q(s_t, a_t) - V(s_t)
1. TEMPORAL DIFFERENCE (TD) RESIDUAL ERROR (Ξ΄_t):
The 1-step error between observed immediate reward plus discounted next state value,
versus current state value estimate:
Ξ΄_t = R_t + Ξ³ Β· V_Ο(s_{t+1}) - V_Ο(s_t)
Where:
β’ R_t : Step reward (incorporating KL penalty).
β’ Ξ³ (gamma): Discount factor (typically Ξ³ = 1.0 in finite language generation episodes).
β’ V_Ο(s) : Value function prediction from Critic network.
(Note: For terminal token t = T: V_Ο(s_{T+1}) = 0.0).
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
2. THE GAE-Ξ» ADVANTAGE EQUATION:
GAE defines advantage as the exponentially discounted sum of TD errors:
T - t
A_t^{GAE(Ξ³, Ξ»)} = β ( Ξ³ Β· Ξ» )^l Β· Ξ΄_{t+l}
l=0
RECURSIVE BACKWARD FORMULATION (Used in PyTorch Implementation):
Computed recursively from terminal step T backwards down to step 1:
A_T = Ξ΄_T
A_t = Ξ΄_t + ( Ξ³ Β· Ξ» ) Β· A_{t+1} for t = T-1, T-2, ..., 1
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
THE BIAS-VARIANCE SPECTRUM CONTROLLED BY HYPERPARAMETER Ξ» β [0, 1]:
β’ Ξ» = 0 (Extreme Low Variance, High Bias):
A_t = Ξ΄_t = R_t + Ξ³ V(s_{t+1}) - V(s_t). Relies heavily on Critic baseline V_Ο.
If the Critic network is inaccurate or poorly trained, updates are severely biased!
β’ Ξ» = 1 (Extreme High Variance, Zero Bias):
A_t = β_{l=0}^{T-t} Ξ³^l R_{t+l} - V(s_t). Matches empirical Monte Carlo return.
Completely unbiased, but policy gradient estimates suffer from massive variance across tokens!
β’ Standard Production Sweet Spot: Ξ» = 0.95, Ξ³ = 1.0.
THE PPO CLIPPED SURROGATE OBJECTIVE (Schulman et al., 2017):
1. IMPORTANCE SAMPLING PROBABILITY RATIO (r_t(ΞΈ)):
Measures how much more or less likely token y_t is under new policy ΞΈ versus old policy ΞΈ_old:
Ο_ΞΈ( y_t | s_t )
r_t(ΞΈ) = ββββββββββββββββββββββ = exp( log Ο_ΞΈ(y_t | s_t) - log Ο_ΞΈ_old(y_t | s_t) )
Ο_ΞΈ_old( y_t | s_t )
β’ At step 0 of PPO epoch: ΞΈ = ΞΈ_old β r_t(ΞΈ) = 1.0.
β’ If new policy increases token likelihood: r_t(ΞΈ) > 1.0.
β’ If new policy decreases token likelihood: r_t(ΞΈ) < 1.0.
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
2. THE UNCONSTRAINED RISK:
Standard Policy Gradient maximizes: L_unconstrained = r_t(ΞΈ) Β· A_t.
If A_t > 0, gradient ascent pushes r_t(ΞΈ) to +β!
A single oversized gradient step alters the policy drastically, causing non-recoverable
linguistic collapse (generating repeated tokens or gibberish)!
3. THE PESSIMISTIC CLIPPED BOUND:
Constrain r_t(ΞΈ) within trust region interval [ 1 - Ξ΅, 1 + Ξ΅ ] (typically Ξ΅ = 0.2):
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β L_CLIP(ΞΈ) = E_t β‘ min β r_t(ΞΈ) Β· A_t, clip(r_t(ΞΈ), 1-Ξ΅, 1+Ξ΅) Β· A_t β β€ β
β β£ β β β¦ β
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
GEOMETRIC BEHAVIOR ACROSS ADVANTAGE REGIMES:
CASE A: ADVANTAGE IS POSITIVE (A_t > 0 - Action was BETTER than expected):
The objective increases as r_t(ΞΈ) increases, but CLIPS at 1 + Ξ΅:
L_CLIP = min( r_t Β· A_t, (1 + Ξ΅) Β· A_t )
Surrogate Loss β²
β / Unclipped: r_t Β· A_t
(1 + Ξ΅)A_t βββββββββββββββββββββββββββββββββββββ (Clipped Lower Bound)
β /
A_t βββββββββββββββββββββ (r_t = 1.0)
β /
0 βββββ΄βββββββββββββββ΄βββββββββββββββ΄βββββββββΊ Ratio r_t(ΞΈ)
0 1.0 1 + Ξ΅
Gradient evaluated when r_t > 1 + Ξ΅ is STRICTLY ZERO (0.0)!
Prevents the policy from taking excessively confident updates on already good actions!
CASE B: ADVANTAGE IS NEGATIVE (A_t < 0 - Action was WORSE than expected):
The objective decreases as r_t(ΞΈ) increases, but CLIPS at 1 - Ξ΅:
L_CLIP = min( r_t Β· A_t, (1 - Ξ΅) Β· A_t )
Prevents taking destructive, massive down-weighting steps on suboptimal actions.
import math
import torch
import torch.nn as nn
import torch.nn.functional as F
# ==============================================================================
# 1. GENERALIZED ADVANTAGE ESTIMATION (GAE-Ξ») BUFFER MODULE
# ==============================================================================
def compute_gae_advantages(rewards: torch.Tensor, values: torch.Tensor,
masks: torch.Tensor, gamma: float = 1.0,
lam: float = 0.95) -> tuple:
"""
Computes Generalized Advantage Estimation (GAE) recursively backwards.
rewards : Step-level rewards with KL penalties: Shape (Batch, Seq_Len)
values : Value estimates from Critic model: Shape (Batch, Seq_Len)
masks : Boolean attention mask (1=token, 0=padding): Shape (Batch, Seq_Len)
Returns:
advantages: GAE-lambda advantage tensor: Shape (Batch, Seq_Len)
returns : Empirical discounted returns (V_target): Shape (Batch, Seq_Len)
"""
batch_size, seq_len = rewards.shape
advantages = torch.zeros_like(rewards)
last_gae_lam = 0.0
# Step backwards from terminal token (seq_len - 1) down to step 0
for t in reversed(range(seq_len)):
# Next value V(s_{t+1}) is 0 if at sequence boundary or masked
if t == seq_len - 1:
next_value = 0.0
else:
next_value = values[:, t + 1] * masks[:, t + 1]
# 1-step TD error: Ξ΄_t = R_t + Ξ³ * V(s_{t+1}) - V(s_t)
delta = rewards[:, t] + (gamma * next_value) - values[:, t]
# Recursive GAE: A_t = Ξ΄_t + (Ξ³ * Ξ») * A_{t+1}
last_gae_lam = delta + (gamma * lam * last_gae_lam)
# Mask out padded positions
advantages[:, t] = last_gae_lam * masks[:, t]
# Target returns for Critic regression: Returns = Advantages + Values
returns = advantages + values
# Normalize advantages over active tokens to mean=0, std=1 for gradient stability
active_mask = (masks == 1.0)
adv_mean = advantages[active_mask].mean()
adv_std = advantages[active_mask].std() + 1e-8
normalized_advantages = (advantages - adv_mean) / adv_std
normalized_advantages = normalized_advantages * masks
return normalized_advantages, returns
# ==============================================================================
# 2. PPO CLIPPED SURROGATE & VALUE OBJECTIVE TRAINER
# ==============================================================================
class PPOTrainerCore:
def __init__(self, clip_epsilon: float = 0.2, vf_coef: float = 0.5,
entropy_coef: float = 0.01):
self.clip_epsilon = clip_epsilon
self.vf_coef = vf_coef
self.entropy_coef = entropy_coef
def compute_losses(self, logprobs_new: torch.Tensor, logprobs_old: torch.Tensor,
values_new: torch.Tensor, values_old: torch.Tensor,
returns: torch.Tensor, advantages: torch.Tensor,
masks: torch.Tensor) -> tuple:
"""
Evaluates the joint PPO Actor-Critic loss function.
"""
# 1. Probability Ratio: r_t(ΞΈ) = exp( log Ο_new - log Ο_old )
ratio = torch.exp(logprobs_new - logprobs_old)
# 2. Clipped Surrogate Policy Loss
surr1 = ratio * advantages
surr2 = torch.clamp(ratio, 1.0 - self.clip_epsilon, 1.0 + self.clip_epsilon) * advantages
policy_loss = -torch.min(surr1, surr2) # Negative for gradient ascent
# 3. Clipped Value Function Loss (Critic MSE)
# Prevents Critic estimates from updating too rapidly in a single step
values_clipped = values_old + torch.clamp(
values_new - values_old, -self.clip_epsilon, self.clip_epsilon
)
vf_loss1 = (values_new - returns) ** 2
vf_loss2 = (values_clipped - returns) ** 2
value_loss = 0.5 * torch.max(vf_loss1, vf_loss2)
# 4. Mask and Average Losses across active sequence tokens
num_active_tokens = masks.sum().clamp(min=1.0)
mean_policy_loss = (policy_loss * masks).sum() / num_active_tokens
mean_value_loss = (value_loss * masks).sum() / num_active_tokens
# Total Joint Objective to Minimize
total_loss = mean_policy_loss + (self.vf_coef * mean_value_loss)
# Metrics for monitoring
with torch.no_grad():
clip_fraction = ((ratio < (1.0 - self.clip_epsilon)) | (ratio > (1.0 + self.clip_epsilon))).float()
mean_clip_frac = (clip_fraction * masks).sum() / num_active_tokens
approx_kl = 0.5 * (((logprobs_new - logprobs_old) ** 2) * masks).sum() / num_active_tokens
return total_loss, mean_policy_loss, mean_value_loss, mean_clip_frac, approx_kl
# ------------------------------------------------------------------------------
# Runtime Execution & Verification Suite
# ------------------------------------------------------------------------------
if __name__ == "__main__":
torch.manual_seed(42)
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
B, S = 2, 6 # Batch size 2, Sequence length 6
print("[*] Initializing PPO & GAE Policy Optimization Verification Engine...")
# Synthetic Attention Mask: Sample 1 length 6, Sample 2 length 4 (padded last 2 tokens)
masks = torch.tensor([
[1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
[1.0, 1.0, 1.0, 1.0, 0.0, 0.0]
], device=device)
# 1. Step Rewards with Terminal Reward Model Score + Intermediate KL Penalties
# Rewards: Intermediate tokens have small negative KL penalties; terminal token gets positive score
step_rewards = torch.tensor([
[-0.05, -0.02, -0.08, -0.01, -0.04, 2.50], # Sample 1: Terminal score 2.50
[-0.03, -0.06, -0.02, 1.80, 0.00, 0.00] # Sample 2: Terminal score 1.80 at index 3
], device=device)
# 2. Critic Baseline Value Estimates V(s_t)
critic_values = torch.tensor([
[0.20, 0.45, 0.70, 1.10, 1.60, 2.10],
[0.15, 0.35, 0.80, 1.40, 0.00, 0.00]
], device=device)
# 3. Compute GAE Advantages and Empirical Returns
advantages, returns = compute_gae_advantages(
rewards=step_rewards, values=critic_values, masks=masks, gamma=1.0, lam=0.95
)
print(f"\n[β] GAE Advantage Computation Executed:")
print(f" β’ Normalized Advantages (Sample 1): {advantages[0].tolist()}")
print(f" β’ Empirical Returns (Sample 1) : {returns[0].tolist()}")
# 4. Simulate Policy Rollout Probabilities
# Old log-probs from rollout generation time
logprobs_old = torch.tensor([
[-1.2, -0.8, -2.1, -0.4, -1.5, -0.9],
[-1.0, -1.5, -0.6, -0.8, 0.0, 0.0]
], device=device)
# New log-probs from current updated policy (with small gradient drift)
logprobs_new = logprobs_old + torch.randn_like(logprobs_old) * 0.1
logprobs_new.requires_grad = True
# New Critic values
values_new = critic_values + torch.randn_like(critic_values) * 0.05
values_new.requires_grad = True
# 5. Evaluate PPO Loss
ppo_engine = PPOTrainerCore(clip_epsilon=0.2, vf_coef=0.5)
total_loss, pol_loss, val_loss, clip_frac, approx_kl = ppo_engine.compute_losses(
logprobs_new=logprobs_new,
logprobs_old=logprobs_old,
values_new=values_new,
values_old=critic_values,
returns=returns,
advantages=advantages,
masks=masks
)
print(f"\n[*] PPO Optimization Diagnostics:")
print(f" β’ Total Joint Loss : {total_loss.item():.4f}")
print(f" β’ Policy Loss (Surr): {pol_loss.item():.4f}")
print(f" β’ Value MSE Loss : {val_loss.item():.4f}")
print(f" β’ Clipped Fractions : {clip_frac.item() * 100:.2f}% (Within safe 5-20% boundary)")
print(f" β’ Approximate KL : {approx_kl.item():.6f}")
# 6. Backward Gradient Propagation
total_loss.backward()
print(f"\n[β] Backward Pass Complete. Log-Prob Gradient Norm: {logprobs_new.grad.norm().item():.4f}")
assert logprobs_new.grad is not None, "Policy gradient failed to backpropagate!"
THE ELIMINATION OF THE CRITIC: PPO vs. GRPO (DeepSeek-R1 / DeepSeekMath, 2024):
CLASSICAL PPO (ACTOR-CRITIC):
Requires maintaining a massive Critic Network V_Ο that consumes 50% of total cluster VRAM!
Critic baseline often suffers from inaccurate value estimation in multi-step reasoning.
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
GROUP RELATIVE POLICY OPTIMIZATION - GRPO (Shao et al., 2024):
CRITIC MODEL IS COMPLETELY ELIMINATED! VRAM REQUIREMENT SLASHED BY 50%!
How GRPO Computes Advantages Without a Critic:
For every input prompt x, the Actor Policy Ο_ΞΈ samples a GROUP of G outputs: {y_1, y_2, ..., y_G}
(e.g., G = 8 candidate solutions sampled in parallel).
Each completion is evaluated by a Reward Function or Rule-Based Verifier: {r_1, r_2, ..., r_G}
GRPO GROUP NORMALIZATION:
The baseline for completion i is simply the MEAN reward of its peer completions in the group!
r_i - Mean( {r_1, ..., r_G} )
Advantage A_i = βββββββββββββββββββββββββββββββββββββββ
StdDev( {r_1, ..., r_G} ) + Ξ΅
BENEFITS:
1. Zero Critic Parameters: Drops 800+ GB of VRAM overhead on 70B models!
2. Rule-Based Mathematical Verifiers: On math/code tasks, replaces imperfect neural reward models
with programmatic compiler/interpreter pass/fail tests (e.g. 1.0 for correct answer, 0.0 for wrong).
3. Drives Self-Correction: Natural competitive pressure within the sampled group enables
models to develop extended Chain-of-Thought (CoT) and self-reflection loops!
RLHF PPO ENGINEERING ANOMALIES & REMEDIATIONS:
1. THE "KL EXPLOSION" POLICY DRIFT COLLAPSE:
β’ Symptom: The approximate KL metric jumps from 0.02 to > 1.5 within 20 iterations.
The model begins generating repetitive, nonsensical, or fragmented loops.
β’ Root Cause: The KL penalty coefficient Ξ² is too small (e.g. Ξ² < 0.001) or learning rate is too large.
The policy rushes to exploit a local blind spot in the Reward Model.
β’ Remediation: Implement Adaptive KL Control:
If KL > 1.5 * Target_KL: Ξ² β Ξ² * 1.5 (Increase penalty!)
If KL < Target_KL / 1.5: Ξ² β Ξ² / 1.5 (Relax penalty!).
2. UNMASKED PADDING PROPAGATION IN GAE BUFFERS:
β’ Symptom: Longer sequences receive massive negative gradients; short responses dominate.
β’ Cause: Failing to zero out values and rewards at <PAD> token positions when computing backward GAE loops.
Padding steps accumulate spurious TD error residuals, corrupting advantages for real tokens!
β’ Remediation: Explicitly enforce:
delta = (rewards + gamma * next_val - values) * masks[:, t]
last_gae = delta + (gamma * lam * last_gae) * masks[:, t]
3. CRITIC DOMINANCE & VALUE COLLAPSE:
β’ Symptom: Value MSE loss spikes to infinity while policy gradients vanish.
β’ Cause: Training the Critic with the same learning rate as the Actor.
Value regression heads require higher initial learning rates to track moving returns, but
must be constrained by gradient clipping (max_norm=0.5).
β’ Remediation: Clip the value loss using values_clipped = values_old + clamp(values_new - values_old, -Ξ΅, Ξ΅).
4. RATIO RUNAWAY IN MULTI-EPOCH PPO LOOPS:
β’ Symptom: Clipped fraction reaches > 50%, causing optimizer updates to stall entirely.
β’ Cause: Running too many PPO epochs per rollout batch (e.g. 8 epochs).
By epoch 4, the active policy has drifted too far from the rollout policy.
β’ Remediation: Restrict PPO update loops to strictly 1 or 2 epochs per rollout batch.
RLHF PPO ARCHITECTURAL SPECIFICATION MATRIX:
βββββββββββββββββββββββββββ¬βββββββββββββββββββββββββββββββ¬βββββββββββββββββββ¬ββββββββββββββββββββββββββββββββ
β ALIGNMENT COMPONENT β MATHEMATICAL FORMULATION β TYPICAL VALUES β PRIMARY OPERATIONAL ROLE β
βββββββββββββββββββββββββββΌβββββββββββββββββββββββββββββββΌβββββββββββββββββββΌββββββββββββββββββββββββββββββββ€
β Probability Ratio r_t β Ο_ΞΈ(y_t|s_t) / Ο_old(y_t|s_t)β Ratio β 1.0 β Measures policy update drift β
β Clipped Surrogate Loss β min( r_t A_t, clip(r_t) A_t )β Ξ΅ = 0.2 β Prevents destructive updates β
β GAE-Ξ» Advantage β β (Ξ³ Β· Ξ»)^l Β· Ξ΄_{t+l} β Ξ³ = 1.0, Ξ» = 0.95β Smooths bias-variance tradeoffβ
β Step TD Residual Ξ΄_t β R_t + Ξ³ V(s_{t+1}) - V(s_t) β Residual Scalar β 1-step Bellman value error β
β Step Reward with KL β r_RM - Ξ² Β· (log Ο - log Ο_refβ Ξ² β [0.01, 0.1] β Penalizes policy drift away β
β β β Target KL β 0.02 β from SFT reference base β
β Actor-Critic Topology β 4 Models: Actor, Critic, β 4 Concurrent LLMsβ Coordinates policy updates, β
β β Reference, and Reward Model β Distributed VRAM β value estimation, and rewards β
β PPO-ptx Pre-train Mix β L_PPO + Ξ³ Β· E[ log Ο_ΞΈ(x) ] β Ξ³ β 25.0 β Eliminates the alignment tax β
β β β β on public pre-training tasks β
β GRPO (Modern Variant) β Group normalization across β Group Size G = 8 β Completely eliminates Critic β
β (DeepSeek-R1 / Math) β peer candidate completions β Zero Critic VRAM β network; enables math CoT β
βββββββββββββββββββββββββββ΄βββββββββββββββββββββββββββββββ΄βββββββββββββββββββ΄ββββββββββββββββββββββββββββββββ€
β TOP 5 RLHF PPO AXIOMS FOR AI ENGINEERS: β
β 1. KL Penalty Prevents Policy Drift: Anchoring policy log-probabilities to a frozen SFT reference model β
β is mandatory to prevent the policy from collapsing into repetitive reward-hacking modes. β
β 2. GAE Smoothes Sparse Rewards: Generalized Advantage Estimation (GAE-Ξ») bridges sparse terminal sequence β
β rewards down to intermediate tokens, reducing gradient variance across the vocabulary. β
β 3. Clipping Enforces Trust Regions: The PPO min(r_t A_t, clip(r_t) A_t) objective bounds policy changes, β
β permitting multiple gradient descent passes over the same generated rollouts without instability. β
β 4. PPO Demands 4 Concurrent Models: Budget GPU cluster memory for Actor, Critic, Reference, and Reward β
β models; apply DeepSpeed ZeRO-3 or FSDP to avoid out-of-memory allocation limits. β
β 5. Shift to GRPO for Reasoning: When aligning models on verifiable tasks (coding, mathematics), replace β
β PPO's memory-heavy Critic with Group Relative Policy Optimization (GRPO) and programmatic verifiers. β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ