⚙️ The Modern Fine-Tuning Toolchain: Unified Abstractions for Foundation Alignment
Building raw mathematical layers from scratch provides deep foundational intuition, but deploying industrial-scale Supervised Fine-Tuning (SFT) over 70B+ foundation models demands an integrated software stack designed to mitigate GPU memory bottlenecks, automate dataset preprocessing, and manage distributed training. In the modern Generative AI ecosystem, this standard is defined by three interconnected libraries: HuggingFace Transformers (model definitions and tokenization engines), BitsAndBytes (custom CUDA kernels for 4-bit NF4 quantization and 8-bit paged optimizers), PEFT (adapter lifecycle management across LoRA, DoRA, and Prefix-Tuning), and TRL (Transformer Reinforcement Learning). TRL's central abstraction, the SFTTrainer, eliminates the engineering friction of manual data collators by orchestrating constant-length sequence packing, multi-turn Jinja2 chat templating, prompt response token masking via data collators, and zero-latency adapter fusion. Understanding how these layers bind together enables engineers to train high-performance instruction models on constrained compute infrastructure.
THE ENTERPRISE FINE-TUNING ECOSYSTEM INTERACTION TOPOGRAPHY:
┌──────────────────────────────────────────────────────────────────────────┐
│ USER APPLICATION & DATASET LAYER: HuggingFace Datasets, Jinja2 Templates │
│ Structured Conversations: [{"role": "user", ...}, {"role": "assistant"}] │
└───────────────────────────────────┬──────────────────────────────────────┘
│
▼ Token Packing & Response Label Masking
┌────────────────────────────────────────────────────────────────────────┐
│ TRL (Transformer Reinforcement Learning - SFTTrainer / SFTConfig) │
│ • Manages training epochs, evaluation loops, dynamic batch collation │
│ • Enforces ConstantLengthDataset concatenation (Zero <PAD> waste) │
└──────────────────┬─────────────────────────────────┬───────────────────┘
│ │
▼ Model Parameter Wrapping ▼ Hardware Acceleration
┌───────────────────────────────────────┐ ┌──────────────────────────────┐
│ PEFT (Parameter-Efficient FT) │ │ ACCELERATE │
│ • Injects LoRA / DoRA / QLoRA adapters│ │ • DDP / FSDP / ZeRO Sharding │
│ • Freezes 99.8% of base model weights │ │ • Multi-GPU / Multi-Node │
│ • Manages adapter checkpoints & merge │ │ gradient synchronization │
└──────────────────┬────────────────────┘ └──────────────┬───────────────┘
│ │
▼ Base Model Ingestion │
┌────────────────────────────────────────────────────────┴───────────────┐
│ TRANSFORMERS (AutoModelForCausalLM & AutoTokenizer) │
│ • Model architecture topologies (LLaMA-3, Mistral, Qwen, Gemma) │
│ • Dispatches FlashAttention-2 / SDPA CUDA kernels │
└───────────────────────────────────┬────────────────────────────────────┘
│
▼ Low-Precision Weight Dequantization
┌────────────────────────────────────────────────────────────────────────┐
│ BITSANDBYTES (Custom CUDA Kernel Accelerator) │
│ • 4-bit NormalFloat (NF4) storage & Double Quantization (DQ) │
│ • Paged AdamW optimizers (Memory page-faulting via CUDA Unified Memory)│
└───────────────────────────────────┬────────────────────────────────────┘
│
▼ Hardware Execution
┌──────────────────────────────────────────────────────────────────────────┐
│ HARDWARE ACCELERATOR (NVIDIA A100 / H100 / RTX 4090 - VRAM & Tensor Core)│
└──────────────────────────────────────────────────────────────────────────┘
RAW DIALOGUE OBJECT (PYTHON LIST OF DICTIONARIES):
[
{"role": "system", "content": "You are a cloud architecture assistant."},
{"role": "user", "content": "What is an S3 bucket?"},
{"role": "assistant", "content": "An Amazon S3 bucket is a scalable object storage container."}
]
│
▼ Applied via tokenizer.apply_chat_template()
│
FORMATTED CHATML STRING (Jinja2 Rendering):
<|im_start|>system
You are a cloud architecture assistant.<|im_end|>
<|im_start|>user
What is an S3 bucket?<|im_end|>
<|im_start|>assistant
An Amazon S3 bucket is a scalable object storage container.<|im_end|>
│
▼ Subword BPE Tokenization
DISCRETE INPUT IDS:
[100264, 9125, 198, 2791, 527, ..., 100265, 198, 100264, 882, ..., 100265]
▲ ▲
│ │
Delimiter: <|im_start|> Delimiter: <|im_end|>
User: / Assistant: instead of <|im_start|>user), the model fails to recognize speaker turn boundaries, resulting in multi-turn role collapse or hallucinated dialogue.TOKEN SEQUENCE AND ASSOCIATED CROSS-ENTROPY LOSS LABELS:
Tokens: [<|im_start|>, user, \n, What, is, an, S3, bucket?, <|im_end|>, \n, <|im_start|>, assistant, \n, An, S3, bucket, is, a, storage, container., <|im_end|>]
│ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │
Labels: [ -100, -100, -100,-100, -100,-100,-100,-100, -100, -100, -100, -100, -100, 482, 912, 14202, 374, 264, 8912, 4102, 100265 ]
└──────────────────────────────────────────────────────────────────────────────────────────────┘ └──────────────────────────────────────────────────────────┘
PROMPT & INSTRUCTION REGION COMPLETION / TARGET REGION
(Labels set to -100 ➔ ZERO GRADIENT!) (Real Token IDs ➔ LOSS EVALUATED HERE!)
MATHEMATICAL FILTERING IN PYTORCH:
Inside PyTorch CrossEntropyLoss:
N
Loss = - ─── 1 ─── ∑ 1_{y_i ≠ -100} · log P( y_i | x_{<i} ; θ )
N_active i=1
Where 1_{y_i ≠ -100} acts as a Dirac indicator function:
• For all prompt tokens: Indicator evaluates to 0.0 (Zero backward gradient pass).
• For assistant tokens : Indicator evaluates to 1.0 (Full backpropagation into LoRA adapters).
labels = input_ids.clone(), the model computes cross-entropy loss over the user's question. Because user prompts vary wildly in style and grammar, the model expends optimization capacity learning to predict user typing styles rather than learning high-quality response synthesis. Completion masking forces 100% of the gradient budget onto response quality.NAIVE PADDED BATCHING (EXTREME COMPUTE & MEMORY INEFFICIENCY):
Batch Size = 3, Max Sequence Length = 16 tokens.
Sample 1: [ "How", "are", "you?" ] ──► Length = 3 tokens
Sample 2: [ "Write", "a", "Python", "script", "to", "scrape", "APIs", "fast" ] ──► Length = 8 tokens
Sample 3: [ "Explain", "quantum", "computing", "simply" ] ──► Length = 4 tokens
Batch Tensor (16x3 = 48 tokens evaluated in GPU memory):
Row 1: [ T, T, T, <PAD>, <PAD>, <PAD>, <PAD>, <PAD>, <PAD>, <PAD>, <PAD>, <PAD>, <PAD>, <PAD>, <PAD>, <PAD> ]
Row 2: [ T, T, T, T, T, T, T, T, <PAD>, <PAD>, <PAD>, <PAD>, <PAD>, <PAD>, <PAD>, <PAD> ]
Row 3: [ T, T, T, T, <PAD>, <PAD>, <PAD>, <PAD>, <PAD>, <PAD>, <PAD>, <PAD>, <PAD>, <PAD>, <PAD>, <PAD> ]
WASTE: 33 out of 48 tokens (68.75%) are useless <PAD> tokens that consume quadratic attention FLOPs!
─────────────────────────────────────────────────────────────────────────────────────────
CONSTANT-LENGTH TOKEN PACKING (TRL SFTTrainer: packing=True):
Concatenates conversations continuously, separated by <|endoftext|> / <EOS> tokens, into fixed blocks:
Single Packed Sequence (Window Length = 16):
[ S1_T1, S1_T2, S1_T3, <EOS>, S2_T1, S2_T2, S2_T3, S2_T4, S2_T5, S2_T6, S2_T7, S2_T8, <EOS>, S3_T1, S3_T2, S3_T3 ]
BENEFITS:
1. Zero Padding Waste: Exactly 100% of processed tokens are informative training tokens!
2. Training Throughput: Accelerates training throughput by 2x to 5x.
3. FlashAttention Integration: Paired with FlashAttention-2 varlen (variable-length) kernels, attention
boundaries are tracked via cumulative sequence length offsets (cu_seqlens), preventing tokens in
Sample 2 from attending across the boundary into Sample 1!
packing=True alongside FlashAttention-2 dispatches a non-padding flash attention kernel that enforces block-diagonal attention masking, preventing cross-document information leakage.THE THREE PILLARS OF MEMORY-BOUND FINE-TUNING:
1. BITSANDBYTES 4-BIT (NF4) QUANTIZATION:
Compresses 16-bit base weights to 4-bit representation (0.5 Bytes/param).
For LLaMA-3-8B: Weight VRAM drops from ~16.0 GB ➔ ~4.5 GB.
2. GRADIENT CHECKPOINTING (torch.utils.checkpoint):
Standard Training: Stores all forward activations across every layer in VRAM for the backward pass.
At sequence length 4096, activations consume > 25 GB of VRAM!
With Checkpointing: Discards intermediate activations; recomputes them on-the-fly during backward pass.
Activation memory drops from O(L) ➔ O(√L), saving ~65% VRAM at the cost of ~20% compute recomputation.
3. PAGED OPTIMIZERS (paged_adamw_8bit / paged_adamw_32bit):
Tracks optimizer states (momentum and variance).
If temporary activation spikes push VRAM to capacity, pages optimizer tensors over PCIe to host CPU RAM,
completely eliminating CUDA Out-of-Memory (OOM) fatal crashes!
┌────────────────────────────────────────────────────────────────────────────────────────┐
│ VRAM CONSUMPTION COMPARISON ON SINGLE NVIDIA GPU (8B MODEL, SEQ LEN = 4096): │
│ • Full Precision Fine-Tuning (FP16, AdamW32, No Checkpointing) : ~128.0 GB (Fails!) │
│ • Standard LoRA (Base in FP16, AdamW32, With Checkpointing) : ~24.5 GB (Tight) │
│ • QLoRA (Base in 4-bit NF4, Checkpointing, Paged AdamW) : ~8.2 GB (Fits!) │
└────────────────────────────────────────────────────────────────────────────────────────┘
import os
import torch
from datasets import Dataset
from transformers import (
AutoModelForCausalLM,
AutoTokenizer,
BitsAndBytesConfig,
TrainingArguments
)
from peft import (
LoraConfig,
get_peft_model,
prepare_model_for_kbit_training
)
from trl import SFTTrainer, DataCollatorForCompletionOnlyLM
# ==============================================================================
# 1. HARDWARE CAPABILITY CONFIGURATION
# ==============================================================================
def get_hardware_configs():
# Detect BFloat16 hardware support (Ampere, Ada Lovelace, Hopper)
supports_bf16 = torch.cuda.is_available() and torch.cuda.is_bf16_supported()
compute_dtype = torch.bfloat16 if supports_bf16 else torch.float16
# 4-bit NormalFloat BitsAndBytes Configuration
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4", # 4-bit NormalFloat
bnb_4bit_use_double_quant=True, # Nested Double Quantization
bnb_4bit_compute_dtype=compute_dtype # Dequantize to BF16/FP16 for compute
)
return bnb_config, compute_dtype, supports_bf16
# ==============================================================================
# 2. SYNTHETIC INSTRUCTION DATASET PREPARATION
# ==============================================================================
def build_instruction_dataset(tokenizer):
"""
Constructs a multi-turn conversational dataset formatted with standard chat templates.
"""
conversations = [
[
{"role": "system", "content": "You are a professional enterprise cloud engineering tutor."},
{"role": "user", "content": "Explain what a dead-letter queue (DLQ) is in distributed messaging."},
{"role": "assistant", "content": "A dead-letter queue (DLQ) is a secondary service queue that stores messages that cannot be successfully processed by consumer services due to formatting errors or network timeouts."}
],
[
{"role": "system", "content": "You are a professional enterprise cloud engineering tutor."},
{"role": "user", "content": "What is the difference between latency and throughput in networks?"},
{"role": "assistant", "content": "Latency measures the round-trip time required for a data packet to travel from source to destination, whereas throughput quantifies the total volume of data transferred over the channel per unit time."}
]
]
# Render conversations into formatted text using the model's Jinja2 chat template
formatted_texts = []
for convo in conversations:
# Check if tokenizer has a chat template; if not, apply standard ChatML
if tokenizer.chat_template is not None:
rendered = tokenizer.apply_chat_template(convo, tokenize=False)
else:
# Fallback ChatML formatting
rendered = (
f"<|im_start|>system\n{convo[0]['content']}<|im_end|>\n"
f"<|im_start|>user\n{convo[1]['content']}<|im_end|>\n"
f"<|im_start|>assistant\n{convo[2]['content']}<|im_end|>\n"
)
formatted_texts.append(rendered)
return Dataset.from_dict({"text": formatted_texts})
# ==============================================================================
# 3. END-TO-END SFTTRAINER ORCHESTRATION ENGINE
# ==============================================================================
def run_production_sft_pipeline():
MODEL_ID = "facebook/opt-125m" # Lightweight model for verification
OUTPUT_DIR = "./output_qlora_adapter"
print("[*] Configuring Hardware & Quantization Environments...")
bnb_config, compute_dtype, supports_bf16 = get_hardware_configs()
# 1. Initialize Tokenizer
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, use_fast=True)
if tokenizer.pad_token is None:
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "right"
# 2. Load Base Model in 4-bit Precision
print(f"[*] Ingesting Base Model ({MODEL_ID}) in 4-bit NF4...")
base_model = AutoModelForCausalLM.from_pretrained(
MODEL_ID,
quantization_config=bnb_config if torch.cuda.is_available() else None,
device_map="auto" if torch.cuda.is_available() else None,
torch_dtype=compute_dtype
)
# 3. Prepare Model for Low-Precision Fine-Tuning
if torch.cuda.is_available():
base_model = prepare_model_for_kbit_training(
base_model,
use_gradient_checkpointing=True
)
# 4. Construct Parameter-Efficient LoRA Configuration
peft_config = LoraConfig(
r=16, # Bottleneck rank
lora_alpha=32, # Scaling constant (alpha = 2 * r)
target_modules=["q_proj", "v_proj"], # Target attention projections
lora_dropout=0.05,
bias="none",
task_type="CAUSAL_LM"
)
model = get_peft_model(base_model, peft_config)
model.print_trainable_parameters()
# 5. Build Instruction Dataset
dataset = build_instruction_dataset(tokenizer)
# 6. Configure Response-Only Completion Masking
# Sets labels = -100 for all tokens prior to the assistant response!
response_template = "<|im_start|>assistant\n" if "<|im_start|>" in dataset[0]["text"] else "assistant\n"
response_template_ids = tokenizer.encode(response_template, add_special_tokens=False)
# In practice, use DataCollatorForCompletionOnlyLM:
collator = DataCollatorForCompletionOnlyLM(
response_template=response_template_ids,
tokenizer=tokenizer
)
# 7. Configure Training Hyperparameters
training_args = TrainingArguments(
output_dir=OUTPUT_DIR,
per_device_train_batch_size=1,
gradient_accumulation_steps=2,
learning_rate=2e-4,
lr_scheduler_type="cosine",
warmup_ratio=0.1,
logging_steps=1,
max_steps=5, # Quick verification run
fp16=not supports_bf16 and torch.cuda.is_available(),
bf16=supports_bf16,
optim="paged_adamw_8bit" if torch.cuda.is_available() else "adamw_torch",
save_strategy="no",
report_to="none"
)
# 8. Instantiate TRL SFTTrainer
trainer = SFTTrainer(
model=model,
train_dataset=dataset,
dataset_text_field="text",
max_seq_length=512,
tokenizer=tokenizer,
args=training_args,
peft_config=peft_config
)
print("[*] Initiating Supervised Instruction Training Loop...")
trainer.train()
# 9. Persist Trained LoRA Adapter
print(f"[✔] Training Concluded. Saving LoRA Adapter Weights to {OUTPUT_DIR}...")
model.save_pretrained(OUTPUT_DIR)
tokenizer.save_pretrained(OUTPUT_DIR)
print("[✔] Adapter Successfully Saved. Model ready for zero-latency merging or serving!")
if __name__ == "__main__":
run_production_sft_pipeline()
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
# ==============================================================================
# PRODUCTION ADAPTER FOLDING WORKFLOW
# Never merge adapters directly into 4-bit quantized base weights!
# ==============================================================================
def export_merged_production_model(base_model_id: str, adapter_dir: str, export_dir: str):
print(f"[*] Step 1: Loading Base Model in UNQUANTIZED 16-bit Precision: {base_model_id}")
# Base model MUST be loaded in native FP16 or BF16 to prevent underflow!
base_model = AutoModelForCausalLM.from_pretrained(
base_model_id,
torch_dtype=torch.float16,
device_map="cpu", # Load to CPU RAM to preserve GPU VRAM
low_cpu_mem_usage=True
)
tokenizer = AutoTokenizer.from_pretrained(adapter_dir)
print(f"[*] Step 2: Attaching Trained LoRA Adapter Weights from {adapter_dir}...")
lora_model = PeftModel.from_pretrained(
base_model,
adapter_dir,
torch_dtype=torch.float16
)
print("[*] Step 3: Executing Permanent Matrix Addition: W_merged = W_0 + (alpha/r) * (B · A)...")
# merge_and_unload mathematically folds the low-rank delta directly into the base weights
# and removes the PEFT wrapper, leaving a standard AutoModelForCausalLM instance!
merged_model = lora_model.merge_and_unload()
print(f"[*] Step 4: Serializing Standalone Merged Checkpoint to {export_dir}...")
merged_model.save_pretrained(export_dir, safe_serialization=True)
tokenizer.save_pretrained(export_dir)
print("[✔] Export Complete! Checkpoint has ZERO latency overhead and requires no PEFT library at inference.")
adapter="medical" or adapter="code"), the engine dynamically gathers the small adapter matrices from host RAM and computes fused batch matrix multiplications, serving hundreds of specialized models from a single GPU node.INDUSTRIAL PEFT & TRL DEFECTS & ENGINEERING REMEDIATIONS:
1. THE CAUSAL DECODER LEFT-PADDING VS. RIGHT-PADDING CONFLICT:
• Symptom: Training converges normally, but batched inference generation emits repetitive nonsense or
immediate EOS tokens!
• Root Cause: In causal autoregressive decoding, padding MUST be placed on the LEFT (tokenizer.padding_side = 'left').
If right-padding is used during batched generation, the model attends to trailing <PAD> tokens,
corrupting the next-token probability distribution!
• Production Rule:
- During SFT Training with Packing: tokenizer.padding_side = 'right' (or packing=True).
- During Batched Inference Generation: tokenizer.padding_side = 'left'!
2. THE "PAD_TOKEN == EOS_TOKEN" DEGENERATION LOOP:
• Symptom: The model generates text that never terminates, or terminates after a single word.
• Root Cause: LLaMA models lack a dedicated <PAD> token. Setting 'tokenizer.pad_token = tokenizer.eos_token'
causes the loss function to mask out true End-Of-Sequence tokens (ignore_index=-100)!
The model is penalized for emitting EOS, forcing it to generate text indefinitely until max_len.
• Remediation: Add a distinct, dedicated padding token to the vocabulary:
tokenizer.add_special_tokens({'pad_token': '[PAD]'})
base_model.resize_token_embeddings(len(tokenizer))
3. OMITTING FFN PROJECTION MATRICES IN LORA TARGET_MODULES:
• Defect: Setting target_modules=["q_proj", "v_proj"] only.
• Impact: Limits adaptation capacity. Up to 65% of foundation model parameters reside inside the Feed-Forward
Networks (gate_proj, up_proj, down_proj). The model struggles to adapt to new domains.
• Remediation: Target all linear layers:
target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"]
4. GRADIENT CHECKPOINTING SILENTLY BREAKING LORA UPDATES:
• Symptom: LoRA adapter gradients evaluate to None or 0.0 after calling model.gradient_checkpointing_enable().
• Root Cause: Gradient checkpointing detaches non-leaf input activations, dropping gradient tracking
for frozen base layers connected to LoRA adapters.
• Remediation: Always call prepare_model_for_kbit_training() before get_peft_model(),
or explicitly enable input gradient tracking:
model.enable_input_require_grads()
labels = batch["labels"][0]
print(tokenizer.decode([t for t in labels if t != -100]))
If any part of the user's prompt appears in the printed output, your response_template string is misaligned with the tokenizer's chat formatting, causing the model to train on user instructions.INDUSTRIAL FINE-TUNING ECOSYSTEM SPECIFICATION MATRIX:
┌─────────────────────────┬──────────────────────────────┬──────────────────┬───────────────────────────────┐
│ FRAMEWORK COMPONENT │ CORE PYTHON ABSTRACTION │ CONFIG PARAMETER │ PRIMARY OPERATIONAL ROLE │
├─────────────────────────┼──────────────────────────────┼──────────────────┼───────────────────────────────┤
│ Quantization Driver │ BitsAndBytesConfig │ load_in_4bit=True│ Loads base model in 4-bit NF4 │
│ │ (bitsandbytes library) │ bnb_4bit_quant=nf4│ memory; saves 75% VRAM │
├─────────────────────────┼──────────────────────────────┼──────────────────┼───────────────────────────────┤
│ Adapter Configuration │ LoraConfig │ r=16, alpha=32 │ Injects trainable low-rank │
│ │ (peft library) │ target_modules │ adapter matrices into GEMMs │
├─────────────────────────┼──────────────────────────────┼──────────────────┼───────────────────────────────┤
│ Model Preparation Hook │ prepare_model_for_kbit_train │ use_gradient_ckpt│ Casts LayerNorm to FP32; │
│ │ (peft library) │ =True │ enables input gradient hooks │
├─────────────────────────┼──────────────────────────────┼──────────────────┼───────────────────────────────┤
│ Instruction Trainer │ SFTTrainer │ dataset_text_field│ High-level training engine; │
│ │ (trl library) │ max_seq_length │ wraps epochs and evaluation │
├─────────────────────────┼──────────────────────────────┼──────────────────┼───────────────────────────────┤
│ Completion Loss Masking │ DataCollatorForCompletionOnly│ response_template│ Sets prompt labels to -100; │
│ │ (trl library) │ │ trains strictly on responses │
├─────────────────────────┼──────────────────────────────┼──────────────────┼───────────────────────────────┤
│ Constant-Length Packing │ ConstantLengthDataset │ packing=True │ Packs multiple conversations; │
│ │ (trl library) │ │ eliminates padding token waste│
├─────────────────────────┼──────────────────────────────┼──────────────────┼───────────────────────────────┤
│ Paged Memory Optimizer │ paged_adamw_8bit │ optim parameter │ Pages optimizer states over │
│ │ (bitsandbytes library) │ │ PCIe to prevent CUDA OOMs │
├─────────────────────────┼──────────────────────────────┼──────────────────┼───────────────────────────────┤
│ Zero-Latency Exporter │ model.merge_and_unload() │ safe_serialization│Folds adapter into base model;│
│ │ (peft library) │ =True │ removes PEFT inference latency│
├─────────────────────────┴──────────────────────────────┴──────────────────┴───────────────────────────────┤
│ TOP 5 ECOSYSTEM INTEGRATION AXIOMS FOR AI ENGINEERS: │
│ 1. Train on Completions Only: Always deploy DataCollatorForCompletionOnlyLM to mask user prompt tokens, │
│ forcing the model to spend 100% of its gradient budget optimizing assistant completions. │
│ 2. Pack Sequences to Kill Padding: Set packing=True in SFTTrainer to concatenate short dialogues into │
│ full context windows, eliminating up to 70% padding token compute waste. │
│ 3. Match the Chat Template: Use tokenizer.apply_chat_template() to guarantee that multi-turn turn │
│ delimiters (<|im_start|>, <|im_end|>) exactly match the base model's pre-trained format. │
│ 4. Never Merge Directly in 4-bit: To export an unquantized production checkpoint, reload the base model │
│ in full 16-bit precision first, then invoke model.merge_and_unload(). │
│ 5. Left-Pad for Generation, Right-Pad for SFT: Always set tokenizer.padding_side = 'left' during batched │
│ inference generation to prevent causal attention masks from attending to trailing padding. │
└───────────────────────────────────────────────────────────────────────────────────────────────────────────┘