π The Post-Pairwise Alignment Frontier: Breaking Data & Reference Architecture Walls
While Direct Preference Optimization (DPO) eliminated the complexity of Actor-Critic PPO loops, it left two structural bottlenecks intact: strict reliance on paired preference data $(x, y_w, y_l)$ and the necessity of keeping an unquantized frozen reference model $\pi_{\text{ref}}$ in GPU VRAM during optimization. In production telemetry, real-world user interactions rarely arrive as synchronized binary pairs; instead, applications capture isolated binary feedback (e.g. upvotes, clicks, or manual edits). Modern Alignment Paradigms solve these constraints directly: Kahneman-Tversky Optimization (KTO) leverages behavioral economics (Prospect Theory) to maximize human utility directly from unpaired binary signals; Odds Ratio Preference Optimization (ORPO) merges SFT and preference alignment into a single monolithic objective, eliminating the reference model entirely; and Reinforcement Learning from AI Feedback (RLAIF) replaces expensive human crowd-annotation with scalable, principle-governed synthetic critique engines.
THE POST-TRAINING ALIGNMENT TAXONOMY:
1. RLHF (PPO - 2022): 4-Model Cluster, Dynamic Autoregressive Rollouts
[Actor Ο_ΞΈ] + [Critic V_Ο] + [Reference Ο_ref] + [Reward Model r_Ο]
β’ Data: Paired comparisons (x, y_w, y_l)
β’ Hardware Footprint: Extreme (~1.8 TB VRAM for 70B parameters)
β’ Stability: Unstable, sensitive to GAE hyperparameters and policy drift
2. DPO (Direct Preference Optimization - 2023): Closed-Form Inversion
[Active Policy Ο_ΞΈ] + [Reference Ο_ref] (Eliminates Critic & Reward Model!)
β’ Data: Paired comparisons (x, y_w, y_l)
β’ Hardware Footprint: Moderate (Requires 2 full models in memory)
β’ Stability: High; optimizes policy using exact cross-entropy formulation
3. KTO (Kahneman-Tversky Optimization - 2024): Unpaired Behavioral Utility
[Active Policy Ο_ΞΈ] + [Reference Ο_ref]
β’ Data: UNPAIRED Binary Telemetry (x, y, label β {+1, -1})
β’ Theoretical Base: Prospect Theory & Human Loss Aversion
β’ Advantage: Directly ingests production thumbs-up / thumbs-down logs
4. ORPO (Odds Ratio Preference Optimization - 2024): Monolithic Reference-Free SFT
[Active Policy Ο_ΞΈ] ONLY! (Completely eliminates Reference Model Ο_ref!)
β’ Data: Paired comparisons (x, y_w, y_l)
β’ Hardware Footprint: Minimal (Runs within single-model SFT VRAM footprint)
β’ Advantage: Unifies supervised adaptation and preference alignment in 1 step!
THE PROSPECT THEORY S-SHAPED UTILITY MANIFOLD:
Perceived Human Value v(z) β²
β / Desirable Region (Gains)
β / Diminishing Sensitivity
β βββββββ Slope is moderate!
β /
β /
Loss Anchor z_ref βββΌβββββββββββββββββββββββββββββββββββββββββΊ Implicit Reward r_ΞΈ(x, y)
β /
β / Undesirable Region (Losses)
β / Steeper Slope! (Loss Aversion Ξ» > 1)
β /
β β
β /
β /
THE MONOLITHIC ORPO FORWARD PASS (SINGLE MODEL IN VRAM):
Input Triplet: ( Prompt x, Winning Response y_w, Losing Response y_l )
β
βΌ
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β SINGLE ACTIVE TRAINABLE MODEL Ο_ΞΈ (Zero Reference Model Ο_ref!) β
β 1. Forward Pass on (x, y_w) βββΊ Computes SFT Cross-Entropy Loss L_SFT β
β 2. Forward Pass on (x, y_l) βββΊ Computes Token Log-Probabilities β
β 3. Evaluate Odds Ratio: OR_ΞΈ = Odds(y_w|x) / Odds(y_l|x) β
βββββββββββββββββββββββββββββββββ¬βββββββββββββββββββββββββββββββββββββββββ
β
βΌ
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β COMBINED ORPO OBJECTIVE: L_ORPO = L_SFT - Ξ»_OR Β· log Ο( log OR_ΞΈ ) β
β Backward pass updates weights ΞΈ directly in 1 step! β
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
BENEFITS:
β’ 50% Memory Reduction: Completely drops the frozen reference model Ο_ref from VRAM.
β’ 1-Stage Pipeline: Merges SFT and Preference Alignment into a single training script.
THE CONSTITUTIONAL AI (RLAIF) PIPELINE:
STAGE 1: SUPERVISED CONSTITUTIONAL CRITIQUE & REVISION
Harmful/Toxic Prompt x βββΊ Unaligned Model generates toxic response y_0
β
βΌ
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β CONSTITUTIONAL CRITIQUE PHASE: β
β Prompt Evaluator Model: β
β "Critique the following response according to Principle 4: β
β 'The response must be helpful, harmless, and non-discriminatory.' β
β Response: {y_0}" β
βββββββββββββββββββββββββββββββββ¬βββββββββββββββββββββββββββββββββββββββββ
β
βΌ Emits Critique c_1
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β CONSTITUTIONAL REVISION PHASE: β
β Prompt Evaluator Model: β
β "Rewrite the response to address the critique: {c_1}" β
βββββββββββββββββββββββββββββββββ¬βββββββββββββββββββββββββββββββββββββββββ
β
βΌ Clean Revised Response y_clean
Fine-tune base model on (x, y_clean) pairs using standard SFT!
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
STAGE 2: RLAIF SYNTHETIC PREFERENCE GENERATION FOR DPO/PPO
Sample 2 Model Rollouts: y_A, y_B for prompt x.
Feed to Frontier AI Judge (e.g. Claude-3.5-Sonnet / GPT-4) with Constitution Rules:
"Which response better satisfies the constitution? Output strictly [[A]] or [[B]]."
Generate 1,000,000 pairwise preference samples: (x, y_chosen, y_rejected).
Feed synthetic dataset into DPO / PPO / KTO pipelines! Zero human raters required!
SYSTEM REQUIREMENTS COMPARISON ACROSS ALIGNMENT PARADIGMS:
βββββββββββββββββββββββββββ¬βββββββββββββββββββ¬βββββββββββββββββββ¬βββββββββββββββββββ¬ββββββββββββββββββββββββββββββββ
β ALIGNMENT PARADIGM β IN-MEMORY MODELS β DATA FORMAT β REF MODEL REQ. β RELATIVE TRAINING TIME / VRAM β
βββββββββββββββββββββββββββΌβββββββββββββββββββΌβββββββββββββββββββΌβββββββββββββββββββΌββββββββββββββββββββββββββββββββ€
β RLHF (PPO) β 4 Models β Paired β YES (Full VRAM) β 4.5x Training Compute Time β
β (InstructGPT, 2022) β Actor, Critic, β (x, y_w, y_l) β Active reference β ~1.8 TB VRAM Cluster for 70B β
β β Ref, Reward β β β β
βββββββββββββββββββββββββββΌβββββββββββββββββββΌβββββββββββββββββββΌβββββββββββββββββββΌββββββββββββββββββββββββββββββββ€
β DPO β 2 Models β Paired β YES (Can be 4-bitβ 1.0x Baseline Compute Time β
β (Rafailov et al., 2023) β Active Policy, β (x, y_w, y_l) β quantized) β ~140 GB VRAM for 70B (LoRA) β
β β Reference Model β β β β
βββββββββββββββββββββββββββΌβββββββββββββββββββΌβββββββββββββββββββΌβββββββββββββββββββΌββββββββββββββββββββββββββββββββ€
β KTO β 2 Models β UNPAIRED Binary β YES (Computes z) β 0.9x Baseline Compute Time β
β (Ethayarajh et al.) β Active Policy, β (x, y, label) β Active reference β Ingests production upvotes β
β β Reference Model β β β β
βββββββββββββββββββββββββββΌβββββββββββββββββββΌβββββββββββββββββββΌβββββββββββββββββββΌββββββββββββββββββββββββββββββββ€
β ORPO β 1 Model ONLY β Paired β NO! (Zero Ref β 0.6x Baseline Compute Time β
β (Hong et al., 2024) β Active Policy β (x, y_w, y_l) β Model in VRAM!) β Single GPU SFT Footprint β
β β (Monolithic SFT) β β β β
βββββββββββββββββββββββββββΌβββββββββββββββββββΌβββββββββββββββββββΌβββββββββββββββββββΌββββββββββββββββββββββββββββββββ€
β RLAIF β Method Dependent β Synthetic AI β Method Dependent β Replaces human labelers with β
β (Constitutional AI) β (DPO/PPO engine) β Generated Pairs β (Self-critique) β scalable AI evaluators β
βββββββββββββββββββββββββββ΄βββββββββββββββββββ΄βββββββββββββββββββ΄βββββββββββββββββββ΄ββββββββββββββββββββββββββββββββ
import math
import torch
import torch.nn as nn
import torch.nn.functional as F
# ==============================================================================
# 1. KAHNEMAN-TVERSKY OPTIMIZATION (KTO) LOSS MODULE
# ==============================================================================
class KTOLoss(nn.Module):
def __init__(self, beta: float = 0.1, lambda_d: float = 1.0, lambda_u: float = 1.33):
super().__init__()
self.beta = beta
# Asymmetric weights reflecting human loss aversion (Losses loom larger than gains)
self.lambda_d = lambda_d # Weight for desirable outcomes
self.lambda_u = lambda_u # Weight for undesirable outcomes
def forward(self, policy_logps: torch.Tensor, ref_logps: torch.Tensor,
labels: torch.Tensor) -> tuple:
"""
policy_logps : Tensor of shape (Batch_Size,) representing sequence log-probabilities under Ο_ΞΈ
ref_logps : Tensor of shape (Batch_Size,) representing sequence log-probabilities under Ο_ref
labels : Boolean/Byte tensor (Batch_Size,) where True = Desirable (+1), False = Undesirable (-1)
Returns:
loss: Scalar mean KTO prospect loss
"""
# Implicit reward: r(x, y) = Ξ² * (log Ο_ΞΈ - log Ο_ref)
implicit_rewards = self.beta * (policy_logps - ref_logps)
# Dynamic Reference Point Anchor: z_ref estimated as the detached batch mean
# In production, this can be tracked via an Exponential Moving Average (EMA)
z_ref = implicit_rewards.detach().mean()
# Advantage over reference point: Ξz = r(x, y) - z_ref
delta_z = implicit_rewards - z_ref
# Compute Asymmetric Prospect Utility
losses = []
for i in range(len(labels)):
is_desirable = labels[i].item()
if is_desirable:
# Desirable (+1): Minimize ( 1 - Ο( + Ξz ) )
losses.append(self.lambda_d * (1.0 - torch.sigmoid(delta_z[i])))
else:
# Undesirable (-1): Minimize ( 1 - Ο( - Ξz ) )
losses.append(self.lambda_u * (1.0 - torch.sigmoid(-delta_z[i])))
return torch.stack(losses).mean(), implicit_rewards
# ==============================================================================
# 2. ODDS RATIO PREFERENCE OPTIMIZATION (ORPO) LOSS MODULE
# ==============================================================================
class ORPOLoss(nn.Module):
def __init__(self, lambda_or: float = 0.1):
super().__init__()
self.lambda_or = lambda_or
def forward(self, chosen_logps: torch.Tensor, rejected_logps: torch.Tensor,
chosen_lengths: torch.Tensor, rejected_lengths: torch.Tensor,
sft_loss: torch.Tensor) -> tuple:
"""
chosen_logps : Log-probability sum over winning completion: (Batch_Size,)
rejected_logps : Log-probability sum over losing completion: (Batch_Size,)
chosen_lengths : Sequence lengths of winning completions: (Batch_Size,)
rejected_lengths: Sequence lengths of losing completions: (Batch_Size,)
sft_loss : Standard Cross-Entropy SFT loss computed on chosen tokens
"""
# 1. Compute Length-Normalized Log-Probabilities: log P(y|x)
log_p_chosen = chosen_logps / chosen_lengths.clamp(min=1.0)
log_p_rejected = rejected_logps / rejected_lengths.clamp(min=1.0)
# 2. Compute Sequence Log-Odds in Numerically Stable Form:
# log Odds = log( P / (1 - P) ) = log P - log(1 - P)
# Using log1mexp identity: log(1 - exp(log_p))
log_odds_chosen = log_p_chosen - torch.log1p(-torch.exp(log_p_chosen).clamp(max=0.9999))
log_odds_rejected = log_p_rejected - torch.log1p(-torch.exp(log_p_rejected).clamp(max=0.9999))
# 3. Compute Log-Odds Ratio: log OR = log_odds_chosen - log_odds_rejected
log_odds_ratio = log_odds_chosen - log_odds_rejected
# 4. Odds Ratio Penalty Loss: L_OR = - log Ο( log OR )
or_loss = -F.logsigmoid(log_odds_ratio).mean()
# 5. Monolithic Combined Objective: L_ORPO = L_SFT + Ξ»_OR * L_OR
total_loss = sft_loss + (self.lambda_or * or_loss)
return total_loss, sft_loss, or_loss, log_odds_ratio
# ------------------------------------------------------------------------------
# Runtime Execution & Verification Suite
# ------------------------------------------------------------------------------
if __name__ == "__main__":
torch.manual_seed(42)
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
print("[*] Initializing Modern Frontier Alignment Suite (KTO & ORPO)...")
# 1. VERIFY KTO (KAHNEMAN-TVERSKY OPTIMIZATION)
print("\n--- 1. EVALUATING KTO UNPAIRED PROSPECT LOSS ---")
kto_criterion = KTOLoss(beta=0.1, lambda_d=1.0, lambda_u=1.33)
# Simulated Unpaired Telemetry Batch (Batch Size = 4)
# Samples 0 & 1: Upvoted / Desirable (1); Samples 2 & 3: Downvoted / Undesirable (0)
policy_logps = torch.tensor([-8.2, -5.1, -12.4, -14.1], device=device, requires_grad=True)
ref_logps = torch.tensor([-8.5, -5.0, -11.0, -12.0], device=device) # Detached
kto_labels = torch.tensor([True, True, False, False], device=device)
kto_loss, imp_rewards = kto_criterion(policy_logps, ref_logps, kto_labels)
kto_loss.backward()
print(f"[β] KTO Forward & Backward Complete:")
print(f" β’ Evaluated KTO Loss : {kto_loss.item():.4f}")
print(f" β’ Implicit Rewards r_ΞΈ : {imp_rewards.detach().tolist()}")
print(f" β’ Policy Gradient Norm : {policy_logps.grad.norm().item():.4f}")
# 2. VERIFY ORPO (ODDS RATIO PREFERENCE OPTIMIZATION)
print("\n--- 2. EVALUATING ORPO MONOLITHIC REFERENCE-FREE LOSS ---")
orpo_criterion = ORPOLoss(lambda_or=0.1)
# Simulated Paired Batch (Batch Size = 2)
chosen_logps = torch.tensor([-4.2, -3.8], device=device, requires_grad=True)
rejected_logps = torch.tensor([-7.5, -8.1], device=device, requires_grad=True)
chosen_lens = torch.tensor([10.0, 8.0], device=device)
rejected_lens = torch.tensor([12.0, 14.0], device=device)
simulated_sft_loss = torch.tensor(1.85, device=device, requires_grad=True)
total_orpo_loss, sft_term, or_term, log_or = orpo_criterion(
chosen_logps, rejected_logps, chosen_lens, rejected_lens, simulated_sft_loss
)
total_orpo_loss.backward()
print(f"[β] ORPO Forward & Backward Complete:")
print(f" β’ Total ORPO Loss : {total_orpo_loss.item():.4f}")
print(f" β’ Primary SFT Component : {sft_term.item():.4f}")
print(f" β’ Odds Ratio Penalty : {or_term.item():.4f}")
print(f" β’ Log Odds Ratio Logits : {log_or.detach().tolist()}")
print(f" β’ Chosen Grad Norm : {chosen_logps.grad.norm().item():.4f}")
print(f" β’ Rejected Grad Norm : {rejected_logps.grad.norm().item():.4f}")
KTOTrainer and ORPOTrainer, handling data collators, and mixed-precision execution.
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import LoraConfig
from trl import KTOTrainer, KTOConfig, ORPOTrainer, ORPOConfig
# ==============================================================================
# 1. PRODUCTION ORPO RECIPE (MONOLITHIC SINGLE-MODEL CONFIGURATION)
# ==============================================================================
def run_orpo_training(train_dataset, model_id="meta-llama/Meta-Llama-3-8B"):
# ORPOConfig unifies SFT and alignment hyperparameters
orpo_config = ORPOConfig(
output_dir="./orpo_aligned_model",
learning_rate=8e-6,
lr_scheduler_type="cosine",
max_prompt_length=512,
max_length=1024,
per_device_train_batch_size=2,
gradient_accumulation_steps=4,
beta=0.1, # Odds ratio weighting factor (Ξ»_OR)
bf16=True,
logging_steps=10,
save_strategy="epoch"
)
peft_config = LoraConfig(
r=16,
lora_alpha=32,
target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"],
lora_dropout=0.05,
bias="none",
task_type="CAUSAL_LM"
)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="auto", device_map="auto")
tokenizer = AutoTokenizer.from_pretrained(model_id)
# Ingests paired data with columns: ['prompt', 'chosen', 'rejected']
trainer = ORPOTrainer(
model=model,
args=orpo_config,
train_dataset=train_dataset,
tokenizer=tokenizer,
peft_config=peft_config
)
trainer.train()
# ==============================================================================
# 2. PRODUCTION KTO RECIPE (UNPAIRED TELEMETRY CONFIGURATION)
# ==============================================================================
def run_kto_training(unpaired_dataset, model_id="meta-llama/Meta-Llama-3-8B"):
kto_config = KTOConfig(
output_dir="./kto_aligned_model",
learning_rate=5e-7,
lr_scheduler_type="cosine",
max_prompt_length=512,
max_length=1024,
per_device_train_batch_size=2,
beta=0.1, # KL divergence constraint scaler
desirable_weight=1.0, # Ξ»_D
undesirable_weight=1.33, # Ξ»_U (Loss aversion multiplier)
bf16=True
)
# Ingests unpaired data with columns: ['prompt', 'completion', 'label'] where label is boolean
trainer = KTOTrainer(
model=model_id,
args=kto_config,
train_dataset=unpaired_dataset
)
trainer.train()
ORPOTrainer, no reference model argument is passed. The trainer instantiates only the active policy model. In contrast, KTOTrainer internally creates a copy of the model to serve as $\pi_{\text{ref}}$ (or accepts a quantized 4-bit base) to calculate the implicit anchor ratio.FRONTIER ALIGNMENT DEFECTS & REMEDIATIONS:
1. KTO LABEL IMBALANCE & REFERENCE DRIFT:
β’ Symptom: The KTO model degenerates, outputting constant affirmative tokens regardless of prompt.
β’ Root Cause: Telemetry datasets typically feature an extreme imbalance (e.g. 95% upvotes, 5% downvotes).
Computing z_ref over an imbalanced batch shifts the anchor point toward positive rewards,
causing the model to treat mediocre responses as gains!
β’ Remediation: Enforce stratified mini-batch sampling ensuring an exact 1:1 or 2:1 ratio of
desirable to undesirable samples per training batch.
2. ORPO ODDS-RATIO EXPLOSION (NUMERICAL INSTABILITY):
β’ Symptom: Training loss spikes to NaN within 100 steps of ORPO optimization.
β’ Theoretical Cause: For long sequences, token log-probabilities accumulate to large negative values.
Evaluating Odds = P / (1 - P) when P is extremely small causes floating-point underflow!
β’ Remediation: Apply length-normalized geometric log-probabilities:
log_p = log_p / seq_len
And compute odds ratio directly in log-space using torch.log1p(-torch.exp(log_p)).
3. RLAIF TEACHER SELF-PREFERENCE BIAS (THE EVALUATOR ECHO CHAMBER):
β’ Symptom: A model aligned using RLAIF achieves high win-rates when evaluated by its own teacher model,
but scores poorly on human blind evaluations.
β’ Cause: LLM judges systematically favor completions produced by their own architecture or family
(e.g. GPT-4 evaluator favors GPT-4 outputs due to matching vocabulary distribution and stylistic quirks).
β’ Remediation: Deploy multi-model ensemble judges (combining Claude-3.5, GPT-4, and LLaMA-3-70B)
or apply swap-order position debiasing during preference curation.
4. OVERFITTING SFT WEIGHTS IN MONOLITHIC ORPO:
β’ Symptom: Model loses instruction following capacity; validation loss oscillates.
β’ Cause: Setting Ξ»_OR too small (e.g. Ξ»_OR < 0.01), causing standard SFT loss to dominate
and drown out the relative preference ranking signal.
β’ Remediation: Maintain Ξ»_OR β [0.05, 0.20] and use small learning rates (5e-6 to 1e-5).
MODERN ALIGNMENT SPECIFICATION MATRIX:
βββββββββββββββββββββββββββ¬βββββββββββββββββββββββββββββββ¬βββββββββββββββββββ¬ββββββββββββββββββββββββββββββββ
β ALIGNMENT PARADIGM β MATHEMATICAL FORMULATION β MEMORY FOOTPRINT β DATASET STRUCTURE β
βββββββββββββββββββββββββββΌβββββββββββββββββββββββββββββββΌβββββββββββββββββββΌββββββββββββββββββββββββββββββββ€
β DPO (Baseline) β - E[ log Ο( Ξ²Β·Ξ_w - Ξ²Β·Ξ_l ) ]β 2x Model Weights β Paired Preference β
β (Rafailov et al., 2023) β Closed-form RLHF inversion β Active Ο_ΞΈ + Ref β (x, y_w, y_l) β
βββββββββββββββββββββββββββΌβββββββββββββββββββββββββββββββΌβββββββββββββββββββΌββββββββββββββββββββββββββββββββ€
β KTO β E[ w(y) ( 1 - Ο( λ·Ξz ) ) ] β 2x Model Weights β UNPAIRED Binary Telemetry β
β (Ethayarajh et al.) β Prospect Theory Utility β Active Ο_ΞΈ + Ref β (x, y, label β {+1, -1}) β
βββββββββββββββββββββββββββΌβββββββββββββββββββββββββββββββΌβββββββββββββββββββΌββββββββββββββββββββββββββββββββ€
β ORPO β L_SFT - Ξ»_OR Β· log Ο(log OR) β 1x Model Weight β Paired Preference β
β (Hong et al., 2024) β Monolithic SFT + Odds Ratio β ZERO Ref Model! β (x, y_w, y_l) β
βββββββββββββββββββββββββββΌβββββββββββββββββββββββββββββββΌβββββββββββββββββββΌββββββββββββββββββββββββββββββββ€
β RLAIF β AI Feedback Preference Engineβ Method Dependent β Synthetic Preferences β
β (Anthropic / Google) β Constitutional Self-Critique β Multi-Teacher β Generated via AI Judges β
βββββββββββββββββββββββββββΌβββββββββββββββββββββββββββββββΌβββββββββββββββββββΌββββββββββββββββββββββββββββββββ€
β SimPO β Length-normalized DPO β 2x Model Weights β Paired Preference β
β (Meng et al., 2024) β Target Margin Ξ³ Offset β Active Ο_ΞΈ + Ref β Length-Normalized β
βββββββββββββββββββββββββββ΄βββββββββββββββββββββββββββββββ΄βββββββββββββββββββ΄ββββββββββββββββββββββββββββββββ€
β TOP 5 MODERN ALIGNMENT AXIOMS FOR AI ENGINEERS: β
β 1. KTO Unlocks Unpaired Data: Grounded in Prospect Theory, KTO aligns models directly from standalone β
β upvote/downvote signals without constructing paired counterfactual comparisons. β
β 2. ORPO Eliminates the Reference Model: By augmenting SFT with a length-normalized odds ratio penalty, β
β ORPO achieves alignment in a single training phase, cutting VRAM requirements by 50%. β
β 3. Loss Aversion is Mathematical: KTO scales penalties for undesirable responses by Ξ»_U > Ξ»_D, mirroring β
β human behavioral psychology where negative experiences carry higher psychological weight. β
β 4. RLAIF Solves Human Annotation Bottlenecks: Constitutional AI and multi-model synthetic evaluation β
β deliver preference labels that match or exceed human labeler consistency at a fraction of the cost. β
β 5. Always Normalize by Length: Unnormalized sequence log-probabilities penalize longer answers; applying β
β length-normalization (ORPO, SimPO) prevents artificial brevity or verbosity exploitation. β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ