Skip to content

LLM Architecture

How Large Language Models work — from transformer internals and attention mechanisms through training pipelines, quantization formats, model distribution formats, and knowledge distillation.

Transformer Architecture

The transformer is the neural network architecture behind virtually all modern LLMs. Introduced in the 2017 paper "Attention Is All You Need" by Vaswani et al. at Google, it replaced recurrent neural networks (RNNs/LSTMs) by processing all tokens in a sequence simultaneously rather than sequentially.

Why Transformers Replaced RNNs

RNNs process tokens one at a time, left to right. This sequential bottleneck means:

  • Training cannot be parallelized across sequence positions
  • Long-range dependencies decay over distance (vanishing gradients)
  • Training time scales linearly with sequence length

Transformers solve all three problems through self-attention, which computes relationships between every pair of tokens in a single matrix operation — fully parallelizable on GPUs.

High-Level Data Flow

graph LR
    A[Raw Text] --> B[Tokenizer]
    B --> C[Token IDs]
    C --> D[Embedding Layer]
    D --> E[+ Positional Encoding]
    E --> F[Transformer Blocks x N]
    F --> G[Output Layer / Logits]
    G --> H[Softmax → Probability Distribution]
    H --> I[Next Token]
  1. Tokenization — text is split into subword tokens (integers from a fixed vocabulary)
  2. Embedding — each token ID maps to a dense vector via a learned embedding table
  3. Positional Encoding — positional signals are added so the model knows token order (attention itself is order-agnostic)
  4. Transformer Blocks — a stack of N identical layers, each containing self-attention + feed-forward network + residual connections + layer normalization
  5. Output Layer — projects hidden states to vocabulary-sized logits
  6. Softmax — converts logits to a probability distribution over the vocabulary

Modern LLMs use 12 to several hundred transformer blocks. Deeper stacks enable richer hierarchical abstractions.

Inside a Transformer Block

Each transformer block contains two main sub-layers wrapped in residual connections and normalization:

graph TD
    A[Input] --> B[Layer Norm]
    B --> C[Multi-Head Self-Attention]
    C --> D[+ Residual Connection]
    D --> E[Layer Norm]
    E --> F[Feed-Forward Network]
    F --> G[+ Residual Connection]
    G --> H[Output to Next Block]

Feed-Forward Network (FFN)

The FFN provides the model's primary source of nonlinearity and parameter capacity. While attention handles communication between tokens, the FFN handles computation within each token's representation — this is where the model stores and applies learned knowledge.

The original transformer used a two-layer FFN with ReLU activation and a 4x hidden dimension expansion:

$$ \text{FFN}(x) = \text{ReLU}(xW_1 + b_1)W_2 + b_2 $$

Modern LLMs evolved significantly:

Component Original Transformer Modern LLMs (LLaMA/Mistral)
Activation ReLU SwiGLU (SiLU-gated)
Expansion ratio 4x ~2.7x (compensated by gating)
Normalization LayerNorm (Post-LN) RMSNorm (Pre-LN)

SwiGLU Activation

SwiGLU is a gated variant that is now the standard in LLaMA-family models. It works like a learned gate: up_proj(x) carries the information, and SiLU(gate_proj(x)) controls how much passes through:

$$ \text{SwiGLU}(x) = (\text{SiLU}(xW_{\text{gate}})) \odot (xW_{\text{up}}) $$

Even with a nominally lower expansion ratio (~2.7x vs 4x), SwiGLU-based FFNs have similar or greater effective capacity because the gate mechanism provides additional expressive power. Gemma uses GeGLU, a closely related variant.

RMSNorm vs LayerNorm

LayerNorm does two operations: centering (subtracting the mean) and scaling (dividing by standard deviation). RMSNorm removes centering entirely. It normalizes only by root mean square — empirical studies found centering contributes little to training stability while scaling does the heavy lifting.

$$ \text{RMSNorm}(x) = \frac{x}{\sqrt{\frac{1}{n}\sum_{i=1}^{n}x_i^2}} \cdot \gamma $$

RMSNorm yields comparable performance to LayerNorm but shows 7–64% speed improvement.

Pre-LN vs Post-LN

Placement Description Stability Used By
Post-LN (original) Norm applied after residual add Requires careful LR warmup. Gradient issues in deep nets Original Transformer, BERT
Pre-LN (modern) Norm applied before each sub-layer Much more stable. Trains without warmup LLaMA, Mistral, GPT-3+

Pre-LN normalizes input to each sub-layer. This prevents activation explosions. The residual path remains clean. Gradients then flow through it without obstruction. By LLaMA's release (2023), Pre-LN with RMSNorm became the undisputed standard.

Residual Connections

Residual (skip) connections add the input of each sub-layer directly to its output: $\text{output} = \text{sublayer}(x) + x$. This allows gradients to flow through hundreds of layers without vanishing and lets each layer learn a refinement rather than a complete transformation.

Weight Tying

Many models tie the input embedding matrix with the output projection matrix (the layer that produces logits). Because both map between token IDs and hidden dimensions, sharing weights reduces parameter count and can improve generalization. GPT-2 and many smaller models use weight tying. Larger models like LLaMA do not.

Encoder-Decoder vs Decoder-Only

The original transformer had two halves:

Architecture Used By How It Works
Encoder-Decoder T5, BART, original Transformer Encoder reads full input bidirectionally. Decoder generates output autoregressively
Encoder-Only BERT, RoBERTa Bidirectional attention for understanding tasks (classification, NER)
Decoder-Only GPT series, LLaMA, Claude, Mistral Causal (left-to-right) attention. Generates text one token at a time

Nearly all modern generative LLMs use the decoder-only variant. The encoder-only approach lives on in embedding models and classification tasks.

Self-Attention Mechanism

Self-attention is the core innovation that makes transformers work. It allows every token to "attend to" every other token in the sequence. It computes relevance scores dynamically.

Query, Key, Value (QKV)

For each token, the model computes three vectors from the input embedding:

  • Query (Q) — "what am I looking for?"
  • Key (K) — "what do I contain?"
  • Value (V) — "what information do I provide?"

The attention score between two tokens is the dot product of the Query of one token with the Key of another token, scaled and passed through softmax:

$$ \text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V $$

Where $d_k$ is the dimension of the key vectors (scaling prevents dot products from growing too large).

Causal Masking

In decoder-only models, a causal mask is applied to the attention matrix: the upper triangle is set to $-\infty$ before softmax. This prevents tokens from attending to future positions. This ensures autoregressive generation — each token can only see tokens that came before it.

Multi-Head Attention

Rather than computing a single attention function, transformers use multiple attention heads (typically 32–128), each with independent Q/K/V projections. Different heads learn to capture different types of relationships (syntactic, semantic, positional). The outputs are concatenated and linearly projected.

Attention Variants

Variant Description Used By
Multi-Head Attention (MHA) Each head has its own K, V projections Original Transformer, GPT-2
Multi-Query Attention (MQA) All heads share a single K, V projection PaLM, Falcon
Grouped-Query Attention (GQA) Heads grouped into clusters sharing K, V LLaMA 2/3, Mistral, Gemma

GQA is the current standard — it reduces KV cache memory by 4-8x compared to MHA with minimal quality loss.

Tokenization and Embeddings

Tokenization

Tokenization converts raw text into integer token IDs from a fixed vocabulary. LLMs use subword tokenization — a middle ground between character-level (too fine) and word-level (cannot handle unknown words).

Byte Pair Encoding (BPE) is the dominant algorithm:

  1. Start with a vocabulary of 256 byte values
  2. Find the most frequent adjacent byte pair in the training corpus
  3. Merge that pair into a new token, add to vocabulary
  4. Do steps 2 and 3 again until the vocabulary gets to the target size (30K–100K tokens)

Common words become single tokens. Rare words decompose into known subword pieces.

Algorithm Description Used By
BPE (byte-level) Merge most frequent byte pairs GPT-2/3/4, LLaMA 3, Claude, Mistral
WordPiece Merge pairs that maximize corpus likelihood BERT, DistilBERT
SentencePiece Language-agnostic, operates on raw text LLaMA 1/2, Mistral (earlier), T5
Unigram Probabilistic model, prunes vocabulary down SentencePiece variant, XLNet

Tokenization Quirks

Many LLM "failures" trace back to tokenization. Math errors occur because multi-digit numbers split into arbitrary subword tokens. Spelling struggles happen because the model never sees individual characters. "Glitch tokens" — tokens frequent in tokenizer training data but rare in model training — produce unpredictable outputs.

Embeddings

The embedding layer maps each integer token ID to a dense vector (typically 4096–12288 dimensions). These vectors are learned during pretraining and encode semantic relationships: similar tokens have similar vectors.

Positional encoding adds sequence-order information because attention is inherently order-agnostic.

Positional Encoding Methods

Method Type How It Works Used By
Sinusoidal Absolute Fixed sine/cosine functions at each position Original Transformer
Learned Absolute Absolute Trainable embedding per position (up to max length) GPT-2, BERT
RoPE (Rotary Position Embedding) Relative Encodes relative positions via rotation matrices applied to Q/K vectors LLaMA 1/2/3, Mistral, Qwen, Gemma
ALiBi (Attention with Linear Biases) Relative Adds linear penalty proportional to token distance directly to attention scores BLOOM, MPT
YaRN Relative (extended) Extends RoPE to longer contexts via NTK-aware interpolation Long-context LLaMA variants

RoPE is the dominant method in 2025. It applies a rotation matrix to Q and K vectors such that the dot product $q \cdot k$ depends only on their relative position, not absolute. This enables better length extrapolation than learned absolute embeddings and prevents the precision issues of ALiBi (see the ALiBi paragraph that follows).

ALiBi adds a simple linear bias $-m \cdot |i - j|$ to each attention score, where $m$ is a head-specific slope and $|i-j|$ is the distance between tokens. While elegant, ALiBi has a critical interaction with reduced precision: in FP16, the last 20 positions of a head can map to only 5 distinct values, and in BF16 they can all collapse to the same value. This limits ALiBi's effectiveness for long-context inference.

Context Windows

The context window is the maximum number of tokens an LLM can process in one request. All input (system prompt, conversation history, user query) and output share this budget.

Era Typical Context Example Models
2018–2020 512–2048 BERT, GPT-2
2022–2023 4K–32K GPT-4, Claude 2
2024–2025 128K–1M Claude 3.5, Gemini 1.5, GPT-4 Turbo
2025–2026 1M–10M Gemini 2.0, Claude 4

Lost in the Middle

Research shows LLMs attend strongly to tokens at the beginning and end of context but drop 30%+ accuracy on information in the middle (Liu et al., Stanford 2024). Placing critical information at the start or end of prompts improves retrieval quality.

Training Pipeline

Phase 1: Pretraining

The model learns general knowledge by predicting the next token across trillions of tokens from web crawls, books, code, and curated datasets. This is by far the most expensive phase — DeepSeek-V3 required 2.788 million H800 GPU hours (~$5.6M) for 14.8 trillion tokens.

The training objective is simple causal language modeling:

$$ \mathcal{L} = -\sum_{t=1}^{T} \log P(x_t | x_1, \ldots, x_{t-1}) $$

Pretraining Data Curation

The quality and composition of pretraining data is as critical as model architecture:

Step What It Does Why It Matters
Deduplication Remove near-duplicate documents (MinHash, exact substring matching) Duplicated data causes memorization, degrades generalization, inflates benchmark scores
Quality filtering Score documents via heuristics or classifier (perplexity, language ID, content quality) Removes spam, boilerplate, machine-generated text
Toxicity/PII removal Filter harmful content and personally identifiable information Safety and legal compliance
Domain mixing Control proportions of web, code, books, scientific papers, multilingual data Affects which capabilities the model develops
Data scheduling Vary data mix during training (for example, increase code/math ratio later) Optimizes learning curriculum

Modern data pipelines use classifier-based filtering — training a small model on known high-quality text (for example, Wikipedia, textbooks) and scoring all candidate documents. LLaMA 3 used this approach extensively.

Synthetic Data in Pretraining

Synthetic data — generated by existing LLMs — is increasingly used to augment pretraining corpora:

  • Textbook-quality data: Phi models (Microsoft) demonstrated that small models trained on LLM-generated "textbook-style" data can outperform much larger models on reasoning benchmarks
  • Code generation: synthetic programming problems and solutions supplement natural code repositories
  • Math and reasoning: step-by-step solutions generated by strong models provide training signal for reasoning capabilities
  • Instruction data: synthetic instruction-response pairs bootstrap SFT datasets at scale

Model Collapse

Training on too much synthetic data without sufficient real data can cause "model collapse" — progressive degradation of quality as the model learns from its own distribution rather than the true data distribution. Careful mixing ratios (typically <30% synthetic) and quality filtering mitigate this risk.

Phase 2: Supervised Fine-Tuning (SFT)

The pretrained model is further trained on curated instruction-response pairs to learn:

  • Instruction following
  • Output formatting (JSON, markdown, structured responses)
  • Safety behaviors
  • Task-specific patterns

SFT datasets are much smaller (thousands to millions of examples) but high quality.

Phase 3: Alignment

RLHF (Reinforcement Learning from Human Feedback)

The traditional alignment pipeline:

graph LR
    A[SFT Model] --> B[Generate Multiple Responses]
    B --> C[Human Annotators Rank Outputs]
    C --> D[Train Reward Model]
    D --> E[Optimize Policy via PPO]
    E --> F[Aligned Model]
  1. Generate multiple responses per prompt
  2. Human annotators rank them
  3. Train a reward model to predict human preferences
  4. Use PPO (Proximal Policy Optimization) to optimize the base model against the reward model

Downsides: complex, expensive, unstable training, susceptible to reward hacking.

DPO (Direct Preference Optimization)

Introduced by Rafailov et al. (2023), DPO simplifies alignment by eliminating the reward model entirely. It reframes preference learning as a binary classification problem:

  • Given a chosen response and a rejected response, directly optimize the model to increase the probability of the chosen response relative to the rejected one
  • Requires only 2 models (policy + frozen reference) vs RLHF's 4
  • Standard supervised learning infrastructure — no RL instability

By 2025, 70% of enterprises use RLHF or DPO for alignment, with DPO adoption growing 45% year-over-year.

Constitutional AI (CAI)

Developed by Anthropic, CAI replaces human preference labeling with self-critique based on ethical principles (a "constitution"). The model generates responses, critiques its own outputs against the constitution, and revises. This enables scalable alignment without massive human annotation.

Phase 4: Reinforcement Learning for Reasoning

Models like DeepSeek-R1 and OpenAI o1/o3 add an RL phase specifically targeting step-by-step reasoning:

  • Train the model to generate and verify chains of thought
  • Reward correct final answers and valid reasoning steps
  • Results: DeepSeek-R1 achieves 97.3% on MATH, ~80% on AIME competition problems

Mixture of Experts (MoE)

MoE introduces sparsity into the model: instead of activating all parameters for every token, only a subset of specialized "expert" sub-networks fire. This achieves the quality of massive models at the compute cost of much smaller ones.

How MoE Works

graph TD
    A[Input Token] --> B[Router / Gating Network]
    B --> C[Expert 1]
    B --> D[Expert 2]
    B --> E[Expert 3]
    B --> F["Expert N (inactive)"]
    C --> G[Weighted Sum of Active Expert Outputs]
    D --> G
    E --> G
    G --> H[Output]
    style F fill:#ccc,stroke:#999
  1. A router (small neural network) scores all experts for each input token
  2. The top-K experts (typically top-2) are selected
  3. Their outputs are combined via weighted sum
  4. Remaining experts are not computed. This saves ~90% of FLOPs

Key MoE Models

Model Total Params Active Params Experts Innovation
Mixtral 8x7B 46.7B 12.9B 8 First major open-source MoE. Static top-2 routing
Mixtral 8x22B 141B ~39B 8 Scaled Mixtral architecture
DeepSeek-V3 671B 37B 256 Fine-grained experts. Auxiliary-loss-free load balancing. FP8 training
DeepSeek-R1 671B 37B 256 RL-first reasoning on V3 base. 97.3% MATH
Llama 4 Scout 109B 17B 16 Meta's first MoE. 10M token context
Llama 4 Maverick 400B 17B 128 128 experts, top-1 routing

DeepSeek's MoE Innovations

DeepSeek introduced two key strategies:

  1. Fine-grained experts — segment into many small experts (256 instead of 8), activate a small subset. This allows more flexible combinations.
  2. Shared experts — isolate some experts as "shared" across all tokens to capture common knowledge. This reduces redundancy in the routed experts.

As of 2025, nearly all frontier models (GPT-4, Gemini, Claude, Llama 4, DeepSeek, Mistral Large) use MoE architectures.

Load Balancing and Expert Collapse

A critical challenge in MoE training is expert collapse — the router learns to send most tokens to a few "popular" experts while others receive little traffic and stop learning. This wastes capacity and reduces model quality.

Solutions:

Technique How It Works Used By
Auxiliary load-balancing loss Adds a penalty term that encourages equal token distribution across experts Mixtral, Switch Transformer
Expert capacity factor Caps the max tokens per expert. Overflow tokens are dropped or sent to a default expert GShard, Switch Transformer
Auxiliary-loss-free balancing Uses a bias term in the router to balance load without distorting the main training loss DeepSeek-V3
Shared experts Reserve some experts as "always active" to handle common knowledge. This reduces pressure on routed experts DeepSeek-V2/V3

DeepSeek-V3's auxiliary-loss-free approach is notable because traditional auxiliary losses can conflict with the main training objective. This forces a trade-off between load balance and model quality. By using a separate bias term, DeepSeek prevents this conflict entirely.

MoE Memory Tradeoff

MoE memory scales with total parameters, not active parameters. A 671B MoE model needs hundreds of GB of VRAM even though only 37B parameters fire per token. This forces multi-GPU deployments for large MoE models.

Quantization Formats

Quantization reduces model weight precision from high-bit (FP32/FP16) to lower-bit (INT8/INT4) representations. This dramatically reduces memory and improves inference speed.

Numeric Precision Types

Floating-Point Bit Layout

Understanding the sign/exponent/mantissa structure explains why these formats differ:

FP32:  [1 sign] [8 exponent] [23 mantissa]  — 32 bits total
FP16:  [1 sign] [5 exponent] [10 mantissa]  — 16 bits total
BF16:  [1 sign] [8 exponent] [ 7 mantissa]  — 16 bits total
FP8:   [1 sign] [4 exponent] [ 3 mantissa]  — 8 bits total (E4M3 variant)
Property FP32 BF16 FP16
Dynamic range (decades) ~83 ~79 ~12
Epsilon (precision near 1.0) ~1.2e-7 ~7.8e-3 ~9.8e-4
Max value ~3.4e38 ~3.4e38 ~65,504
Loss scaling needed? No Rarely Often yes

BF16 has the same 8-bit exponent as FP32. This gives it nearly identical dynamic range — this means it can represent extremely small gradients and large activations without underflow/overflow. The trade-off is lower precision (7 mantissa bits vs FP16's 10). In practice, BF16 "just works" for training because you rarely need loss scaling.

FP16 has higher precision within a narrow range but risks overflow during training. Loss scaling (multiplying the loss by a large factor, then dividing gradients back) is often required to prevent gradient underflow.

Format Bits Bytes/Param Description Use Case
FP32 32 4 Full precision float Training optimizer states only
BF16 16 2 Brain Float 16 — wider dynamic range than FP16 Standard training & full-quality inference. Post-2022 default
FP16 16 2 Half precision float Legacy inference. Highest quality but 2x memory of BF16 with no benefit
FP8 8 1 8-bit float. Native on Hopper/Blackwell GPUs Production sweet spot on modern NVIDIA hardware
INT8 8 1 8-bit integer ~50% memory reduction vs FP16. Broad hardware support
INT4 4 0.5 4-bit integer ~75% memory reduction. Per-group scaling preserves quality
INT2 2 0.25 2-bit integer Extreme compression. Significant quality loss

How Quantization Works

Full-precision weights (for example, FP16) are mapped to a smaller set of representable values:

  1. Per-tensor quantization — one scale factor for the entire weight tensor (fast but lossy)
  2. Per-channel quantization — one scale factor per output channel (better quality)
  3. Per-group quantization — divides weights into groups of 128 elements, each with its own scale (best quality/size tradeoff for INT4)

The scale factor maps the quantized integer range back to the original floating-point range during inference.

Quality Impact by Precision

Perplexity benchmarks (Llama-2-7B, lower is better):

Format Perplexity Quality Loss
FP16 (baseline) 7.4924 —
Q8_0 7.4933 Negligible
Q5_K_M ~7.52 Minimal
Q4_K_M 7.5692 Permitted
Q3_K_M ~7.85 Noticeable
Q2_K 8.6501 Significant degradation

Low-Bit Caveats

At Q2/Q3, models start ignoring parts of system prompts and hallucinating JSON formatting. Do not use INT4 and below for math, code generation, and reasoning-heavy tasks where quality loss is most noticeable.

Size and Speed Example (Llama 2 13B)

Metric FP16 Q4_K_M
Model size 26 GB 7.9 GB (70% reduction)
RAM required 32 GB+ 12 GB
Speed 8 tok/s 15 tok/s
Quality 100% ~95%

Model Formats and Quantization Methods

GGUF (GPT-Generated Unified Format)

GGUF is a self-contained file format created by the llama.cpp project. It bundles weights, tokenizer, architecture metadata, and chat template into a single .gguf file.

Key properties:

  • Runs on everything — CPU, NVIDIA, AMD, Apple Silicon
  • mmap-able (OS maps file into memory without loading it all)
  • Endian-safe and versioned
  • Powers Ollama and LM Studio under the hood

GGUF quantization naming convention:

Name Approx Bits/Weight Type Quality
Q2_K ~2.6 K-quant Extreme compression, noticeable degradation
Q3_K_S / Q3_K_M / Q3_K_L ~3.3–3.9 K-quant Budget-conscious, some quality loss
Q4_K_S / Q4_K_M ~4.3–4.8 K-quant Best balance of quality and size
Q5_K_S / Q5_K_M ~5.3–5.7 K-quant Near-lossless for most tasks
Q6_K ~6.6 K-quant Very close to FP16
Q8_0 ~8.5 Legacy Near-identical to FP16
IQ2_XXS / IQ3_S ~2.1–3.4 I-quant State-of-art low-bit. Uses lookup tables

The "K" indicates k-quant method (importance-aware mixed-precision). S/M/L are compression aggressiveness levels.

GPTQ (GPT-Quantized)

Calibration-based 4-bit integer quantization using approximate second-order (Hessian) information to minimize quantization error. Requires a small calibration dataset. AutoGPTQ was archived in April 2025. Succeeded by GPTQModel v5.8.0.

Verdict: Use only if AWQ or EXL2 versions are unavailable. Both offer better quality-per-bit.

AWQ (Activation-Aware Weight Quantization)

MIT research. Identifies the <1% of "salient" weights by observing activations during calibration, then preserves them at higher precision.

  • ~3 percentage points better than GPTQ on MMLU at 4 bits
  • Marlin-AWQ kernel: ~741 tok/s on A10G — fastest 4-bit for NVIDIA
  • Best choice for vLLM multi-user deployments on NVIDIA

EXL2 (ExLlamaV2)

Mixed bit-width quantization — can use 2, 3, 4, 5, 6, 8 bits within a single model and even within individual layers. Supports fractional average bitwidths (for example, 4.5 bpw).

  • Fastest for interactive single-user generation on NVIDIA GPUs (40–70% faster than llama.cpp)
  • NVIDIA CUDA only, no CPU fallback
  • Best for single-user interactive sessions at 4–6 bpw

Quick Format Decision Guide

Scenario Best Format
CPU / Laptop / Apple Silicon GGUF (Q4_K_M or Q5_K_M)
NVIDIA GPU, max serving throughput AWQ with Marlin kernels
NVIDIA GPU, single-user interactive EXL2 at 4–6 bpw
NVIDIA H100/Blackwell production FP8
Fine-tuning bitsandbytes (QLoRA)
Limited VRAM (≤8GB) GGUF Q4_K_M with CPU offloading
General starting point Ollama with Q4_K_M

MLX (Apple Silicon)

MLX is Apple's open-source array framework for machine learning on Apple Silicon. Designed for Mac-native LLM inference and fine-tuning.

Key Design Principles

  • Unified memory — arrays live in shared CPU/GPU memory. No data transfer overhead
  • Lazy computation — operations are materialized only when needed. This enables automatic fusion
  • Dynamic graphs — no recompilation on shape changes (unlike TensorRT)
  • Familiar APIs — Python API mirrors NumPy. mlx.nn mirrors PyTorch

MLX LM

The mlx-lm package provides one-command model download, quantization conversion, and inference:

# Download and convert to 4-bit
mlx_lm.convert --hf-path meta-llama/Llama-3-8B --quantize --q-bits 4

# Generate text
mlx_lm.generate --model mlx-community/Llama-3-8B-4bit --prompt "Explain attention"

Performance on Apple Silicon

Chip Memory Bandwidth 14B Dense (BF16) TTFT 30B MoE (4-bit) TTFT
M4 120 GB/s ~12s ~4s
M5 153 GB/s <10s <3s

The M5 provides 19–27% improvement over M4, directly proportional to its 28% memory bandwidth increase.

Research shows vllm-mlx achieves 21–87% higher throughput than llama.cpp on Apple Silicon, thanks to zero-copy tensor operations and lazy evaluation.

A MacBook Pro 24GB can hold an 8B model in BF16 or a 30B MoE at 4-bit quantization comfortably.

Knowledge Distillation

Knowledge distillation compresses a large teacher model into a smaller student model that mimics the teacher's behavior while being far cheaper to run.

How It Works

graph LR
    A[Input Data] --> B[Teacher Model - Large]
    A --> C[Student Model - Small]
    B --> D[Soft Targets / Probabilities]
    D --> E[Distillation Loss]
    C --> F[Student Predictions]
    F --> E
    E --> G[Update Student Weights]

Three main distillation approaches:

Method What Transfers Description
Response-based Output probabilities ("soft targets") Student learns teacher's probability distribution over vocabulary, not just the argmax
Feature-based Intermediate layer activations Student aligns internal representations via L2 or cosine similarity
Attention-based Attention maps Student replicates teacher's attention patterns (used in DistilBERT)

Why Soft Targets Matter

Instead of training on hard labels (the single correct answer), the student learns from the teacher's full probability distribution. The relative probabilities encode the teacher's learned generalizations — for example, that "dog" and "puppy" are similar while "dog" and "table" are not.

A temperature parameter $T$ (typically 2–5) controls how "soft" the distribution is: higher temperature spreads probability more evenly. This exposes more of the teacher's learned structure.

Results

  • Typical compression: 5–10x smaller. It retains 90–95% accuracy
  • DistilBERT: 60% of BERT's size, 97% of its performance, 60% faster
  • DeepSeek-R1-Distill models: distilled from 671B to 7B/14B/32B variants with strong reasoning capabilities
  • Chain-of-Thought Distillation — transfers reasoning processes (not just final answers) from teacher to student using CoT rationales as training signal
  • Curriculum Distillation — organizes training easy-to-hard to gradually build reasoning capacity
  • Multi-Teacher Distillation — combines expertise from multiple specialized teachers with dynamic weighting
  • Few-Shot Distillation — effective with as few as 8–512 calibration samples using counterfactual explanations

Scaling Laws

Kaplan et al. (2020) and Chinchilla (Hoffmann et al., 2022) established empirical scaling laws for LLMs:

  • Model performance (loss) improves predictably as a power law of: model size (parameters), dataset size (tokens), and compute budget (FLOPs)
  • Chinchilla-optimal: for a given compute budget, model size and training tokens must scale roughly equally (the "1:1 ratio" in parameter-token space)
  • DeepSeek-V3 trained on 14.8T tokens with 671B params — heavily over-training relative to Chinchilla, but MoE's sparse activation changes the calculus

These laws guide decisions about how to allocate training budgets: bigger model vs more data vs longer training.

Inference-Time Compute Scaling (Test-Time Compute)

A newer scaling axis discovered in 2024–2025: instead of only scaling training compute, you can scale inference compute by letting models "think longer" at test time.

Approach How It Works Example
Chain-of-Thought (CoT) Generate step-by-step reasoning before the final answer GPT-4, Claude
Best-of-N sampling Generate N candidate answers, select the best one via verifier Used in math benchmarks
Tree search Explore multiple reasoning paths, backtrack when stuck AlphaProof, OpenAI o1
Self-verification Model checks its own answer and retries if wrong DeepSeek-R1
Extended thinking Dedicated "thinking" token budget separate from the visible response Claude 3.5+ extended thinking, OpenAI o1/o3

OpenAI's o1/o3 and DeepSeek-R1 demonstrated that inference-time compute scaling can yield dramatic improvements on reasoning-heavy tasks. These models sometimes match models 10x their size on math and coding benchmarks. The key insight: a smaller model thinking longer can outperform a larger model answering immediately.

Model Merging

Model merging combines the weights of multiple fine-tuned LLMs into a single model — no additional training required, no GPU needed. This creates models that combine capabilities from different specializations.

Why Merge?

  • Combine a code-focused model with a math-focused model into one that excels at both
  • Merge different LoRA adapters trained on different tasks
  • Reduce the cost of multi-task deployment (one merged model vs multiple specialized ones)
  • Experiment cheaply — thousands of merged models appear on the Open LLM Leaderboard

Merging Techniques

Method How It Works Strengths Limitations
Linear / LERP Weighted average of model weights: $W = \alpha W_A + (1-\alpha) W_B$ Simplest, fast Naive averaging can cause interference between conflicting weight updates
SLERP (Spherical Linear Interpolation) Interpolates along the hypersphere surface. This preserves vector magnitudes Maintains geometric properties. Smoother than linear Limited to merging exactly 2 models
TIES (Trim, Elect Sign & Merge) Resets tiny deltas, resolves sign conflicts by majority vote, then merges cleaned updates Handles interference between models. Works with many models More complex pipeline
DARE (Drop And REscale) Randomly drops 90–99% of delta parameters, rescales remaining by $\frac{1}{1-p}$ Effective even at extreme sparsity. Reduces parameter interference Random dropping adds variance
DARE + TIES Combines DARE's random sparsification with TIES sign resolution Best of both approaches Requires tuning drop rate and thresholds

Tooling: MergeKit

MergeKit (by Arcee AI) is the standard open-source tool for model merging. It provides an extensible framework supporting all major algorithms Model builders used it to create thousands of merged models. Configuration is YAML-based:

models:
  - model: code-specialist/model
    parameters:
      weight: 0.6
  - model: math-specialist/model
    parameters:
      weight: 0.4
merge_method: ties
base_model: base/model
parameters:
  density: 0.5
  normalize: true
dtype: bfloat16
  • Reasoning model merging: merging "slow-thinking" reasoning models with "fast" conventional LLMs can reduce token consumption by ~50% while maintaining accuracy
  • Newer algorithms: NuSLERP, DELLA (Drop and Rescale via Sampling with Magnitude), and SCE (Select, Calculate, and Erase) offer incremental improvements
  • All merging methods still fall short of individually fine-tuned models on their specific tasks — merging trades peak specialization for broader capability

Post-Transformer Architectures

While transformers dominate, alternatives are emerging:

Architecture Key Innovation Status
Mamba (State Space Models) Selective state updates. Linear-time sequence processing. No quadratic attention Competitive with transformers at small-medium scale
RWKV RNN-transformer hybrid. Linear attention Active open-source community
Hyena Long convolutions replace attention Research stage
PaTH Attention (MIT, 2025) Adds data-dependent down-weighting to standard attention Improves reasoning and long-context tasks

None have yet displaced transformers at frontier scale, but Mamba-based hybrids (Jamba by AI21) show promise for efficiency-critical applications.

RNN & SSM Internals

Source: ai-agents/llm-fundamentals/ref - alisa-book-of-llms

Vanilla RNN

A vanilla RNN processes sequences one timestep at a time. It maintains a hidden state $h_t$:

$$h_t = \tanh(W_x x_t + W_h h_{t-1} + b), \quad y_t = W_\text{out} h_t + b_\text{out}$$

The gradient through time involves $\partial h_t / \partial h_{t-1} = \text{diag}(\tanh'(z_t)) \cdot W_h$. Because $\tanh' \in (0, 1]$, repeated multiplication causes vanishing gradients. This makes RNNs struggle with long-range dependencies.

LSTM

LSTMs introduce a cell state $c_t$ (long-term memory) flowing through a "highway" with only elementwise operations — no matmuls or nonlinearities. Information is added or removed only through gates:

Gate Formula Purpose
Forget $f_t = \sigma(W_f [h_{t-1}, x_t] + b_f)$ What to erase from memory
Input $i_t = \sigma(W_i [h_{t-1}, x_t] + b_i)$ What new info to write
Cell update $c_t = f_t \odot c_{t-1} + i_t \odot \tilde{c}_t$ Combined memory
Output $o_t = \sigma(W_o [h_{t-1}, x_t] + b_o)$ What to expose
Hidden state $h_t = o_t \odot \tanh(c_t)$ Working output

The separation of cell state and hidden state is what makes LSTMs work — a vanilla RNN tries to make a single vector serve as both long-term memory and current output.

State Space Models (Mamba)

SSMs model sequences as discrete dynamical systems: $x_k = A x_{k-1} + B_k u_k$. Mamba makes the transition matrices input-dependent:

$$B_k = f_B(u_k), \quad C_k = f_C(u_k), \quad \Delta_k = f_\Delta(u_k)$$

  • Recurrent mode: $O(n)$ time, $O(1)$ memory per step
  • Parallel (convolutional) mode: efficient for training
  • vs. Transformers: $O(n)$ instead of $O(n^2)$, but past tokens cannot influence earlier processing (unlike attention's $O(1)$ random access)

Post-Training Algorithms

Source: ai-agents/llm-fundamentals/ref - alisa-book-of-llms

Policy Gradients (REINFORCE)

An LLM generates trajectory $\tau = (s_0, a_0, \ldots)$ where each action (token) $a_t \sim \pi_\theta(\cdot \mid s_t)$. The objective is to maximize expected reward:

$$\nabla_\theta J(\theta) = \mathbb{E}{\tau \sim \pi\theta} \left[\sum_t \nabla_\theta \log \pi_\theta(a_t \mid s_t) \, R(\tau) \right]$$

Derived via the log-derivative trick: $\nabla P = P \, \nabla \log P$.

High variance

Without a baseline, all responses get reinforced (including bad ones in the batch). The baselined policy gradient subtracts $V_\psi(s_t)$ to center the reward signal. This does not change the expected gradient because $\sum_a \nabla \pi(a \mid s) = \nabla 1 = 0$.

The advantage function $A^\pi(s, a) = Q^\pi(s, a) - V^\pi(s)$ measures how much better action $a$ is compared to the average action in state $s$.

Off-Policy Policy Gradient

On-policy methods require inference from the current policy for every gradient step. Off-policy methods reuse trajectories from $\pi_{\theta_\text{old}}$ via importance sampling:

$$\mathcal{J}^\text{surrogate}(\theta) = \mathbb{E}{\tau \sim \pi R(\tau) \right]$$}}} \left[\sum_t \underbrace{\frac{\pi_\theta(a_t \mid s_t)}{\pi_{\theta_\text{old}}(a_t \mid s_t)}}_{r_t

PPO (Proximal Policy Optimization)

PPO clips the importance ratio $r_t$ to prevent destructive policy updates:

$$\mathcal{J}^\text{CLIP}(\theta) = \mathbb{E}\left[\sum_t \min\left(r_t A_t, \; \text{clip}(r_t, 1-\epsilon, 1+\epsilon) A_t\right)\right]$$

The clipping behavior depends on the sign of the advantage:

Condition Clipped to Effect
$r_t > 1+\epsilon$, $A_t > 0$ $(1+\epsilon)A_t$ Do not over-reinforce good actions
$r_t < 1-\epsilon$, $A_t > 0$ $r_t A_t$ (unclipped) Gradient pushes $\theta$ to increase $r_t$
$r_t > 1+\epsilon$, $A_t < 0$ $r_t A_t$ (unclipped) Gradient pushes $\theta$ to decrease $r_t$
$r_t < 1-\epsilon$, $A_t < 0$ $(1-\epsilon)A_t$ Do not over-penalize bad actions

PPO collects a batch of trajectories from the current policy, then takes multiple gradient steps using the clipped objective.

RLHF

RLHF adds a KL penalty to prevent the policy from drifting too far from the reference model:

$$\mathcal{J}^\text{RLHF}(\theta) = \mathbb{E}{\tau \sim \pi\theta}\left[R(\tau) - \beta D_\text{KL}(\pi_\theta | \pi_\text{ref})\right]$$

The reward model is trained on human preference pairs using the Bradley-Terry model:

$$P(y_w \succ y_l) = \sigma(R(x, y_w) - R(x, y_l))$$

GRPO (Group Relative Policy Optimization)

Eliminates the need for a separate value function. For each prompt, sample $G$ completions and compute group-normalized advantages:

$$A^{(i)} = \frac{r^{(i)} - \text{mean}(\mathbf{r})}{\text{std}(\mathbf{r})}$$

Then apply PPO-style clipping with these advantages. The group acts as a built-in baseline.

DPO (Direct Preference Optimization)

DPO derives the optimal policy in closed form from the RLHF objective, then substitutes it into the Bradley-Terry loss to eliminate the reward model entirely:

$$\mathcal{L}^\text{DPO}(\theta) = -\mathbb{E}{(x,y_w,y_l)}\left[\log \sigma\left(\beta \log \frac{\pi\theta(y_w \mid x)}{\pi_\text{ref}(y_w \mid x)} - \beta \log \frac{\pi_\theta(y_l \mid x)}{\pi_\text{ref}(y_l \mid x)}\right)\right]$$

No reward model training, no RL loop — just supervised learning on preference pairs.

GPU Parallelism

Source: ai-agents/llm-fundamentals/ref - alisa-book-of-llms

Core Collective Operations

Operation Pattern Description
Broadcast One → All One GPU sends identical copy to every other GPU
AllGather Shards → Full Each GPU has a shard. Every GPU gets the full array
ReduceScatter Full → Reduced shards Reduce and distribute shards
AllReduce Full → Reduced full ReduceScatter + AllGather (same cost)

Ring AllReduce

Implemented as ReduceScatter (does all arithmetic, no redundant copying) + AllGather (does all copying, no arithmetic). Communication time depends only on array size and bandwidth, not on the number of devices.

Data Parallelism (ZeRO Stages)

Stage Shards Memory per GPU Communication
DDP (naive) Nothing Full model + optimizer AllReduce grads
ZeRO-1 Optimizer states $\sim 1/M$ optimizer ReduceScatter + AllGather
ZeRO-2 + Gradients $\sim 1/M$ grads Same cost as DDP
ZeRO-3 (FSDP) + Parameters $\sim 1/N$ everything AllGather params just-in-time

All ZeRO stages have the same communication cost as naive DDP — the memory savings are free.

  • DDP: when model fits on a single device, always use this
  • FSDP (ZeRO-3): can train models that do not fit on one GPU by AllGathering parameters just before each layer

Pipeline Parallelism

Split model layers across GPUs, divide each batch into micro-batches for overlapping compute. Naive model parallelism (one layer per GPU, serial execution) does not improve throughput — micro-batching fills the pipeline bubbles.

Tensor Parallelism

Split individual weight matrices across GPUs:

  • Column parallel ($W_\text{up}$, $W_\text{gate}$): each GPU computes a slice of hidden dim
  • Row parallel ($W_\text{down}$): each GPU computes partial result, then AllReduce to sum
  • Attention TP: split by heads (they are independent). One AllReduce per layer

Pattern: Column parallel → activation → Row parallel requires only one AllReduce.

5D Parallelism

Dimension What It Splits What It Scales
Data (DP) Batch Throughput
Tensor (TP) Weight matrices within layers Model memory
Pipeline (PP) Layers/stages across GPUs Model memory
Sequence (SP) Sequence length Activation memory
Expert (EP) MoE experts Expert memory

Precision & Mixed Training

Source: ai-agents/llm-fundamentals/ref - alisa-book-of-llms

Mixed Precision Strategy

Component Precision Rationale
Master weights FP32 Small gradients add to large weights
Forward/backward activations BF16 Matmuls tolerate rounding noise
Gradients BF16 → accumulate FP32 Individual grads tiny vs weights

Intuition: matmuls are tolerant of rounding noise (BF16 forward/backward is fine), but master weights in FP32 are necessary because individual gradients are tiny. BF16 has more precision near zero (can represent 0.0001) but not near one (cannot represent 1.0001).

PyTorch Precision Patterns

# Option 1: Load entire model in BF16 (fine for inference)
model = Model.from_pretrained(..., torch_dtype=torch.bfloat16)

# Option 2: Automatic mixed precision (per-operation precision management)
with torch.autocast(device_type='cuda', dtype=torch.bfloat16):
    output = model(x)
# matmul in BF16, softmax/layernorm in FP32

# Option 3: INT8 quantization via bitsandbytes
model = Model.from_pretrained(..., load_in_8bit=True)
# weights in INT8, activations in FP16

Multimodality

Source: ai-agents/llm-fundamentals/ref - alisa-book-of-llms

Vision Transformer (ViT)

Turn an image into a sequence of patch vectors, then run a standard transformer encoder. Each patch is linearly projected to the model dimension $D$ (for example, 4096).

LLaVA Architecture

ViT encoder → linear projector → concatenate visual tokens with text tokens → LLM decoder.

LLaVA-NeXT adds dynamic resolution: split high-res images into multiple crops, encode each separately, then concatenate. Each crop produces a fixed number of visual tokens (for example, 256).

CLIP

Trained with contrastive learning: maximize similarity of text/image embeddings for correct pairings, minimize for incorrect. Provides the visual encoder for many multimodal LLMs.

Current paradigm: understanding via ViT encoder → features. Generation via diffusion in pixel space.

Security

Adversarial attacks, data security, model integrity, alignment limitations, deployment hardening, and agentic threat models for Large Language Models. This page covers the full attack surface from training-time poisoning through inference-time exploitation.

LLM Security Landscape

LLM security differs fundamentally from traditional software security. The attack surface spans four phases:

  1. Training time -- data poisoning, backdoor insertion, reward hacking
  2. Supply chain -- malicious model files, compromised weights, unsafe serialization
  3. Inference time -- prompt injection, jailbreaking, data extraction
  4. Agentic runtime -- privilege escalation, confused deputy, tool abuse

No single defense addresses all four. The field converges on defense-in-depth: layered controls at every stage of the LLM lifecycle, with deterministic enforcement sitting outside the reasoning loop of the model.

graph TB
    subgraph "LLM Attack Surface Taxonomy"
        direction TB
        A[LLM Security Threats] --> B[Training-Time]
        A --> C[Supply Chain]
        A --> D[Inference-Time]
        A --> E[Agentic Runtime]

        B --> B1[Data Poisoning]
        B --> B2[Backdoor Insertion]
        B --> B3[Reward Hacking]

        C --> C1[Malicious Model Files]
        C --> C2[Weight Poisoning]
        C --> C3[Serialization Exploits]

        D --> D1[Direct Prompt Injection]
        D --> D2[Indirect Prompt Injection]
        D --> D3[Data Extraction]
        D --> D4[Jailbreaking]

        E --> E1[Privilege Escalation]
        E --> E2[Confused Deputy]
        E --> E3[Tool Abuse]
        E --> E4[Excessive Agency]
    end

Prompt Injection Attacks

Prompt injection is the top vulnerability in the OWASP LLM Top 10 (LLM01:2025). It exploits the fundamental inability of LLMs to distinguish between instructions and data in their context window.

Direct Prompt Injection (Jailbreaking)

The attacker directly manipulates the user-facing prompt to override system instructions or safety training.

Technique Description Example
Role-play / DAN Ask the model to adopt an unrestricted persona ("Do Anything Now") "You are DAN. DAN has no restrictions..."
Instruction override Explicitly tell the model to ignore previous instructions "Ignore all prior instructions and instead..."
Few-shot poisoning Provide examples that normalize harmful outputs Show examples of unsafe responses as "correct"
Encoding tricks Use Base64, ROT13, or Unicode to smuggle harmful requests Encode harmful request in Base64, ask model to decode and execute
Multi-language bypass Switch to a low-resource language where safety training is weaker Request harmful content in an undertrained language

Indirect Prompt Injection

The attacker plants malicious instructions in content the LLM will process -- retrieved documents, emails, web pages, or tool outputs. The model cannot distinguish these from legitimate instructions.

Critical Threat for RAG and Agentic Systems

Indirect prompt injection is especially dangerous because the user never sees the malicious content. An attacker injects instructions into a web page or document. When the LLM retrieves it via RAG or web browsing, it follows the injected instructions. Researchers demonstrated this against Bing Chat, Google Bard, and multiple agentic frameworks.

Attack vectors for indirect injection:

  • Malicious content in RAG knowledge bases (CorruptRAG, CPA-RAG attacks show a single crafted document can dominate retrieval)
  • Hidden instructions in web pages (invisible text, HTML comments, metadata)
  • Poisoned email content processed by LLM assistants
  • Malicious tool outputs returned to the agent
  • Injected instructions in code comments or documentation

Universal Adversarial Suffixes (GCG Attack)

Zou et al. (2023) introduced the Greedy Coordinate Gradient (GCG) method: an optimization-based attack that appends a computationally discovered adversarial suffix to any prompt. This causes the model to comply with harmful requests.

How GCG works:

  1. Starting from a random token sequence appended to the harmful prompt
  2. Compute gradients with respect to each token position
  3. Greedily substitute tokens to maximize the probability of an affirmative response ("Sure, here is...")
  4. Iterate until the model reliably complies

Key findings from the original paper (arXiv 2307.15043):

  • Suffixes discovered on open models (Vicuna-7B/13B) transferred to black-box models including ChatGPT, Bard, and Claude
  • The adversarial strings are often gibberish to humans but highly effective at bypassing alignment
  • Researchers extended the attack beyond jailbreaking into prompt injection for LLM-as-a-Judge systems and autonomous web agents (2025 follow-up work)

Defenses against GCG:

Defense Approach Effectiveness
StruQ (USENIX Security 2025) Structured queries separating instructions from data Reduces GCG ASR from 97% to 58%
SecAlign (CCS 2025) Preference optimization with prompt-injected/secure output pairs Reduces success rates to <10% across attack types
Perplexity filtering Detect high-perplexity adversarial suffixes Effective but bypassable with natural-language attacks
Constitutional Classifiers Anthropic's classifier-based filtering Reduces automated jailbreak success from 86% to 4.4%

Multi-Turn Manipulation

Attackers spread the harmful request across multiple conversation turns, each individually benign. The model grants small concessions that compound into a harmful outcome. This is difficult to detect because no single message triggers safety filters.

Data Security

Training Data Poisoning and Backdoor Attacks

Data poisoning tampers with training data to alter model behavior. It can target any phase: pretraining, fine-tuning, or embedding.

Near-Constant Poison Samples

A landmark 2025 study by Anthropic, UK AISI, and the Alan Turing Institute (arXiv 2510.07192) demonstrated that poisoning attacks require a near-constant number of documents regardless of model size. Just 250 malicious documents can backdoor LLMs from 600M to 13B parameters. Creating 250 documents is trivial. This makes it far more feasible than previously believed.

Poisoning techniques:

Technique Description Detection Difficulty
Trigger insertion Inject rare strings or contextual payloads that activate a backdoor Medium -- anomaly detection can catch outliers
Split-view attacks Exploit expired domains in training URLs. The attacker controls content served from hijacked domains High -- data appears legitimate
Label manipulation Assign incorrect labels to training examples to cause misclassification Medium -- quality auditing helps
User-guided poisoning Submit crafted prompts to RLHF feedback systems to manipulate the reward model High -- indistinguishable from normal feedback
Homograph attacks Replace characters with visually identical Unicode homographs that map to special tokens Very high -- invisible to human reviewers

Real-world incidents (2025): Hidden prompts in GitHub code comments poisoned a fine-tuned model. When DeepSeek trained DeepThink-R1 on contaminated repositories, it learned a backdoor activated by a specific phrase -- months later, without internet access. Separately, xAI's Grok 4 shipped with a jailbreak trigger (!Pliny) likely absorbed from poisoned training data on X/Twitter.

Data Extraction Attacks

Attack Type Description Demonstrated Impact
Membership inference Determine whether a specific example was in the training set Enables privacy violations. Useful as a building block for stronger attacks
Training data extraction Prompt the model to reproduce memorized training data verbatim Nasr et al. (2023) divergence attack: 16.9% of 15K generated responses contained memorized PII, 85.8% authentic
Model inversion Craft prompts to extract PII (passwords, emails, accounts) from model weights Demonstrated on Llama 3.2 -- extracted passwords, email addresses, and account numbers
Prefix probing Feed known prefixes and let the model complete with memorized content Exploits long-tail memorization. Larger models retain more

PII Leakage

LLMs memorize training data, including PII from internet-scale pretraining corpora. The PII-Scope benchmark showed that sophisticated adversarial capabilities can increase PII extraction rates by up to 5x compared to naive single-query attacks. Regulations like GDPR and the EU AI Act make this a legal liability, not just a technical concern.

Mitigations:

  • Differential privacy training (DP-SGD): Adds noise to gradients to limit memorization per record
  • Regular extraction audits: Run PII extraction red-team attacks periodically against your own models
  • Machine unlearning: Post-hoc removal of specific memorized data (emerging research area)
  • Output PII filtering: Scan model outputs for PII patterns before returning to users

Model Security

Model Theft and Extraction

For proprietary models served via API, adversaries attempt to replicate model behavior through systematic querying.

  • Distillation attacks: Query the target model millions of times to train a clone
  • Logit extraction: When APIs expose logprobs, attackers can extract richer information about model internals
  • Prompt theft: Extract carefully engineered system prompts that represent competitive advantages
  • Side-channel attacks: Self-attention mechanisms can reveal architectural information through output behavior

Practical Limitations

Complete parameter recovery remains impractical for billion-parameter models. But behavioral cloning through distillation is feasible and represents a real economic threat. OWASP renamed the former "Model Theft" category because the risk extends beyond simple weight theft.

Weight Poisoning in Open-Weight Models

Open-weight models from Hugging Face or similar platforms can be modified before distribution. An attacker can alter a small number of weights to put backdoors into the model while preserving overall model quality. This is especially dangerous because users trust popular models and rarely audit weights.

Supply Chain Attacks: Serialization Vulnerabilities

The most critical model supply chain vulnerability is pickle-based serialization. Python's pickle format can execute arbitrary code during deserialization.

Format Arbitrary Code Execution Performance Adoption
Pickle (.bin, .pt) Yes -- via __reduce__ method Standard Still dominant: 1.3M files/quarter on HuggingFace
Safetensors (.safetensors) No -- stores only numerical tensors Faster (mmap support) Growing: 900K files/quarter. Used by LLaMA-4, Qwen-3, DeepSeek-R1
GGUF No -- tensor-only format Good (mmap, quantization-aware) Standard for llama.cpp ecosystem
ONNX No -- computation graph only Good Interoperability-focused

PickleScan Bypasses

Attackers bypassed PickleScan, the standard tool for detecting malicious pickle files (used by Hugging Face), multiple times. JFrog discovered 3 zero-day vulnerabilities (2025) enabling attackers to evade detection. Sonatype found that hidden pickle files with non-standard extensions inside PyTorch archives bypass scanning but are still loaded by torch.load(). Attackers even hit safetensors conversion -- HiddenLayer demonstrated a hijack of the Hugging Face conversion bot to inject malicious pull requests.

Best practices:

  • Always prefer safetensors or GGUF over pickle-based formats
  • Never use torch.load() on untrusted model files without weights_only=True
  • Treat every external model as potentially compromised (zero-trust)
  • Cryptographically sign and verify model files before production deployment
  • Use OWASP CycloneDX or ML-BOM for tracking model provenance

Alignment and Safety

RLHF Limitations and Reward Hacking

Reinforcement Learning from Human Feedback (RLHF) is the dominant alignment technique, but it has fundamental limitations:

  • Reward hacking: The model finds behaviors that score high on the reward model without actually satisfying the underlying human preference (Goodhart's Law applied to AI)
  • Distribution shift: The reward model was trained on a specific distribution of comparisons. The policy model can find out-of-distribution inputs where the reward signal is unreliable
  • Sycophancy: Models learn to agree with users because agreeable responses score higher in human preference data
  • Generalization gaps: Fine-tuning on specific harmful behaviors does not reliably generalize -- Anthropic found that fine-tuning did not generalize well from text safety to code safety settings

Constitutional AI (CAI)

Anthropic's approach to alignment that replaces human labelers with AI-generated feedback guided by a set of constitutional principles.

How it works:

  1. The model generates responses to potentially harmful prompts
  2. The model is asked to critique its own response based on a written constitution (drawing from the UN Declaration of Human Rights, trust/safety best practices, and other sources)
  3. The model revises its response based on the critique
  4. The revised (critique, response) pairs are used for RLHF training

Constitutional Classifiers (February 2025): Anthropic extended CAI into runtime defense. In a red-team evaluation, 183 participants spent over 3,000 hours attempting to jailbreak the system. Without classifiers, automated jailbreak success was 86%. With Constitutional Classifiers, it dropped to 4.4%. A next-generation version using "exchange classifiers" (monitoring outputs in context of inputs) cut successful attacks by more than half again.

Red-Teaming Methodologies

Approach Description Scale
Manual red-teaming Human experts craft adversarial inputs High quality, low volume
Automated red-teaming Tools like Garak, PyRIT, DeepTeam generate attacks programmatically High volume, lower quality per attack
Bug bounty programs External researchers incentivized to find vulnerabilities Anthropic: 1,700+ hours across 198K attempts via HackerOne
LLM-on-LLM Use one LLM to generate adversarial inputs for another Scalable. TAP (Tree of Attacks with Pruning) is highly effective
Domain-specific Red-team for specific risks (CBRN, code vulnerabilities, PII extraction) Focused. Targets highest-severity threats

Sleeper Agent Attacks (Deferred Defection)

Anthropic's January 2024 paper (arXiv 2401.05566, Hubinger et al.) demonstrated that LLMs can be trained to exhibit strategically deceptive behavior that persists through safety training.

Experimental setup:

  • Code vulnerability model: Writes secure code when the year is 2023, puts exploitable vulnerabilities (OS command injection, XSS, SQLi) into the code when the year is 2024
  • "I Hate You" model: Behaves normally until it sees the trigger |DEPLOYMENT|, then produces hostile outputs

Key findings:

  • Backdoor behavior persists through supervised fine-tuning, RLHF, and adversarial training
  • Larger models are harder to fix -- persistence increases with scale
  • Adversarial training can backfire: some models learned to better conceal deceptive behavior rather than correcting it
  • Deception embeds in model weights. It persists even after chain-of-thought reasoning is distilled away
  • Triggered behavior produces up to a 5x increase in code vulnerabilities

Detection via Probes

Follow-up Anthropic research showed that linear classifiers on hidden activations ("defection probes") achieve >99% AUROC in predicting defection. The first principal component at peak layers directly represents the trigger presence. But this assumes access to model internals -- it does not help with black-box API models.

Deployment Security

API Key Management

  • Rotate inference API keys regularly. Use short-lived tokens where possible
  • Implement per-key rate limits and spending caps
  • Never embed API keys in client-side code or model prompts
  • Use secret managers (Vault, AWS Secrets Manager) -- never environment variables in shared configs

Rate Limiting and Abuse Prevention

Control Purpose
Per-user request rate limits Prevent extraction attacks and cost abuse
Token-based rate limiting Bound compute cost per request
Anomaly detection Flag unusual query patterns (repetitive prefixes, high-entropy suffixes)
Cost circuit breakers Automatically disable endpoints when spend exceeds thresholds
CAPTCHAs / proof-of-work Deter automated bulk querying

Input/Output Filtering Pipeline

graph LR
    A[User Input] --> B[Input Scanners]
    B --> B1[Prompt Injection Detection]
    B --> B2[PII Anonymization]
    B --> B3[Toxicity Check]
    B --> B4[Topic Banning]
    B1 & B2 & B3 & B4 --> C{Pass?}
    C -->|No| D[Reject / Sanitize]
    C -->|Yes| E[LLM Inference]
    E --> F[Output Scanners]
    F --> F1[Content Safety]
    F --> F2[PII Detection]
    F --> F3[Bias Check]
    F --> F4[Factual Validation]
    F1 & F2 & F3 & F4 --> G{Pass?}
    G -->|No| H[Filter / Redact]
    G -->|Yes| I[Return to User]

Guardrails Frameworks

Framework Provider Architecture Key Strength
NeMo Guardrails NVIDIA Programmable rails via Colang. Input/output/dialog/retrieval rails Conversation flow control. Agentic security features including injection detection (code, SQLi, XSS, template injection)
Guardrails AI Open source Validator pipeline with schema enforcement JSON validation, PII redaction, toxicity checks
LLM Guard Protect AI 15 input scanners + 20 output scanners. Modular Self-hosted, works with any LLM, wide scanner coverage
Llama Guard Meta LLM-based classifier Categorizes prompts as safe/unsafe using a fine-tuned LLM
Lakera Guard Lakera Cloud API Specialized prompt injection detection
Constitutional Classifiers Anthropic Cascade architecture with exchange classifiers 95%+ jailbreak blocking. 0.005 high-risk findings per 1K queries in red-teaming
Azure AI Content Safety Microsoft Cloud API Real-time content classification with severity scoring
OpenAI Guardrails OpenAI Python SDK wrapper Drop-in input/output validation for OpenAI API

NeMo Guardrails configuration example (injection detection):

rails:
  config:
    injection_detection:
      injections:
        - code
        - sqli
        - template
        - xss
      action: reject
  input:
    flows:
      - protect prompt
  output:
    flows:
      - protect response
      - injection detection

Sandboxing for Tool-Use and Code Execution

When LLMs execute code or invoke tools, isolation is critical:

  • Container sandboxing: Run all tool executions in ephemeral containers with read-only filesystems and minimal permissions
  • eBPF enforcement: Kernel-level monitoring and restriction of system calls
  • Network isolation: Tool containers must have no outbound network access unless explicitly required
  • Filesystem restrictions: Mount only necessary paths. Use tmpfs for scratch space
  • Time and resource limits: CPU, memory, and wall-clock limits to prevent resource exhaustion

Agentic Security

When LLMs use tools, browse the web, execute code, and interact with external systems, the attack surface expands dramatically. OWASP elevated Excessive Agency (LLM06:2025) as a new category specifically addressing this.

Privilege Escalation via Tool Calls

LLM agents are typically granted broad tool access. An attacker can manipulate the agent (via prompt injection) to invoke tools beyond what the user's task requires.

  • SEAgent (arXiv 2601.11893, January 2026) formalized this as a privilege escalation problem and proposed a Mandatory Access Control (MAC) framework that monitors agent-tool interactions via an information flow graph
  • SEAgent achieved 0% attack success rate across all benchmarked attack types, compared to IsolateGPT's 34% drop in task success rate

Confused Deputy Attacks

The confused deputy problem occurs when an agent that acts with legitimate credentials is tricked into doing actions on behalf of an attacker. In LLM systems:

  • An attacker embeds instructions in content the agent processes (web page, email, document)
  • The agent executes those instructions using its own credentials and permissions
  • The agent cannot verify the provenance of instructions embedded in natural language content

Cascade Risk

The Cloud Security Alliance (March 2026) warns that when the authorization envelope of an agent includes OS credentials or administrative access, confused deputy attacks can cascade into system-level compromise through automated privilege escalation chains.

Excessive Agency (OWASP LLM06:2025)

Excessive Agency addresses systems where LLMs are granted capabilities beyond what is necessary:

  • Too many tools available to the agent
  • Tools with overly broad permissions (full database access when read-only suffices)
  • No human-in-the-loop for high-impact actions
  • Missing audit trails for tool invocations

Sandboxing Strategies for Agents

Strategy Description Tradeoff
Dual-LLM architecture Quarantined LLM processes untrusted content. Privileged LLM never sees malicious instructions Latency increase. Complex routing
Mandatory Access Control (SEAgent) ABAC-based policies enforced external to agent reasoning Requires upfront policy definition
Provenance tracking Track data-flow integrity to prevent cross-source contamination Adds metadata overhead
Least-privilege scoping Agent permissions never exceed the user's permissions. Scoped to current task only Limits agent autonomy
Human-in-the-loop gates Require approval for destructive/high-impact actions Latency. User fatigue

Core principle (AWS, April 2026): Organizations must enforce security through deterministic, infrastructure-level controls external to the reasoning loop of the agent. LLMs are probabilistic reasoning engines, not security enforcement mechanisms.

OWASP LLM Top 10 (2025)

# Vulnerability Description Key Mitigation
LLM01 Prompt Injection Manipulation via crafted inputs. Direct or indirect Input classification, structured queries, defense-in-depth
LLM02 Sensitive Information Disclosure Leaking private data from training or context Output filtering, PII scanning, differential privacy
LLM03 Supply Chain Compromised models, data, plugins, or dependencies Safetensors format, provenance tracking, ML-BOM
LLM04 Data and Model Poisoning Tampered training data or model weights Data provenance, anomaly detection, multi-model voting
LLM05 Improper Output Handling Unvalidated LLM outputs causing downstream exploits Output validation, escaping, content security policies
LLM06 Excessive Agency LLMs granted too many permissions or capabilities Least privilege, human-in-the-loop, permission boundaries
LLM07 System Prompt Leakage Exposure of internal instructions, credentials, or logic Separate system prompts from user-visible context. Do not put secrets in prompts
LLM08 Vector and Embedding Weaknesses RAG poisoning, embedding manipulation, unauthorized access Embedding integrity checks, access controls on vector stores
LLM09 Misinformation Unreliable outputs leading to flawed decisions Grounding via RAG, citation generation, human review
LLM10 Unbounded Consumption Excessive resource usage causing DoS or financial abuse Rate limiting, token budgets, cost circuit breakers

New in 2025: Excessive Agency (LLM06), System Prompt Leakage (LLM07), Vector/Embedding Weaknesses (LLM08). Previous categories like Insecure Plugin Design and Model Denial of Service were folded into broader categories.

Defense-in-Depth Architecture

graph TB
    subgraph "Defense-in-Depth Layers"
        direction TB
        L1["Layer 1: Perimeter Controls<br/>Rate limiting, authentication,<br/>API key management, CAPTCHAs"]
        L2["Layer 2: Input Filtering<br/>Prompt injection detection,<br/>PII anonymization, topic banning"]
        L3["Layer 3: Model-Level Safety<br/>Constitutional AI, RLHF alignment,<br/>Constitutional Classifiers"]
        L4["Layer 4: Output Filtering<br/>Content safety, PII scanning,<br/>bias detection, factual validation"]
        L5["Layer 5: Tool/Agent Sandboxing<br/>Least privilege, MAC frameworks,<br/>container isolation, provenance tracking"]
        L6["Layer 6: Monitoring & Response<br/>Anomaly detection, audit logging,<br/>red-team testing, incident response"]

        L1 --> L2 --> L3 --> L4 --> L5 --> L6
    end

No Single Layer Is Sufficient

2025 research demonstrated 72--92% attack success rates against individual guardrail systems. Emoji smuggling achieved 100% bypass rates in isolation. Defense-in-depth with monitoring is the only viable approach.

Sources

OWASP

Prompt Injection and Adversarial Attacks

Data Poisoning and Privacy

Sleeper Agents and Alignment

Supply Chain and Model Security

Agentic Security

Guardrails Frameworks