Skip to content

LLM Fundamentals Reference

Look-up tables for the LLM stack: current models and context windows, numeric and quantization formats, serving-engine and tooling versions, GPU specs, benchmarks, vector databases, guardrail frameworks, the OWASP LLM Top 10, and a glossary. Concepts behind these tables live in Explanation; tasks that use them live in How-to Guides.

These tables go stale fast

Model lineups, prices, and engine versions change monthly. Every table carries an "as of" date. Values marked secondary were cross-checked only against third-party trackers because the vendor page was unreachable during the 2026-09-25 refresh. Confirm them on the vendor page before you rely on them.

Model Landscape (2026-09)

Frontier API Models

Prices are standard-tier USD per million tokens (MTok), input / output. Batch APIs are typically 50% off, and cached input is billed at a fraction of the base input price.

Vendor Model Context window Max output Price in / out per MTok Verification
Anthropic Claude Fable 5.1 1M 128K $10 / $50 Vendor docs, 2026-09-25
Anthropic Claude Opus 5.5 1M 128K $4 / $20 Vendor docs, 2026-09-25
Anthropic Claude Sonnet 5 1M 128K $2 / $10 Vendor docs, 2026-09-25
Anthropic Claude Haiku 4.5 200K 64K $1 / $5 Vendor docs, 2026-09-25
OpenAI GPT-6 Astra (released 2026-09-03) ~1.05M (prompts over 272K are repriced) 128K $10 / $50 Secondary (several pricing trackers agree)
Google Gemini 3.1 Pro 1M TBD $2 / $12 for prompts up to 200K, higher above Secondary (pricing guides disagree on context: 1M vs 2M)

Anthropic details

Anthropic's models overview (2026-09-25) lists Claude Fable 5.1, Opus 5.5, Sonnet 5, and Haiku 4.5 as current. It lists Fable 5, Opus 5, Opus 4.8/4.7/4.6/4.5, and Sonnet 4.6/4.5 as legacy models that remain available. Prompt-cache reads cost 10% of base input (2.5% on Fable 5.1, 5% on Opus 5.5).

Open-Weight Models

Model Developer Released Total / active params Context License Notes
DeepSeek-V4-Pro DeepSeek 2026-04-24 (preview), later GA 1.6T / 49B 1M MIT Hybrid Compressed Sparse Attention (CSA) + Heavily Compressed Attention (HCA). Secondary for parameter counts
DeepSeek-V4-Flash DeepSeek 2026 284B / 13B 1M MIT Secondary
Qwen3.5 (flagship 397B-A17B) Alibaba Qwen 2026-02 to 2026-03 397B / 17B (family spans 0.8B to 397B) TBD Apache-2.0 vLLM lists Qwen3.5 under hybrid attention / state-space models
Qwen3.6-35B-A3B, Qwen3.6-27B Alibaba Qwen 2026 35B / 3B (MoE), 27B (dense) TBD Apache-2.0 Open-weight tier of Qwen 3.6. Qwen3.6-Plus is API-only
gpt-oss-120b, gpt-oss-20b OpenAI 2025-08-05 117B / 5.1B, 21B / 3.6B 128K Apache-2.0 MoE weights shipped in MXFP4
Llama 4 Scout Meta 2025-04-05 109B / 17B (16 experts) 10M (advertised) Llama 4 Community License Meta's first MoE
Llama 4 Maverick Meta 2025-04-05 400B / 17B (128 routed experts + shared expert) 1M Llama 4 Community License Alternates dense and MoE layers
DeepSeek-V3 / R1 DeepSeek 2024-12 / 2025-01 671B / 37B (256 routed experts + 1 shared) 128K MIT (V3-0324 and R1) R1 = RL-trained reasoning model on the V3 base
Mixtral 8x7B Mistral AI 2023-12 46.7B / 12.9B (8 experts, top-2) 32K Apache-2.0 First widely used open MoE
Mixtral 8x22B Mistral AI 2024-04 141B / 39B 64K Apache-2.0 Scaled Mixtral

Other 2026 open-weight families have day-0 engine support: Gemma 4 (Ollama and llama.cpp READMEs), Kimi K3 (SGLang, 2026-07), GLM-5.x (SGLang), and NVIDIA Nemotron 3 (SGLang). Qwen3.8 appeared in 2026-08. Their parameter counts and licenses were not verified in this refresh (TBD).

Context Window Growth

Era Typical context Example models
2018–2020 512–2,048 BERT, GPT-2
2022–2023 4K–32K (100K–200K at the top end by late 2023) GPT-4 (8K/32K), Claude 2 (100K, then 200K in Claude 2.1)
2024–2025 128K–1M GPT-4 Turbo (128K), Claude 3.x (200K), Gemini 1.5 Pro (1M–2M), Llama 4 Scout (10M advertised)
2026 1M is the frontier default Claude Opus 5.5 / Sonnet 5 / Fable 5.1 (1M), Gemini 3.1 Pro (1M), GPT-6 Astra (~1.05M), DeepSeek-V4 (1M)

Numeric Formats

Format Bits Bytes/param Layout / scaling Typical use (2026)
FP32 32 4 E8M23 Optimizer states and master weights in training
BF16 16 2 E8M7. Same range as FP32 Default training and full-quality inference
FP16 16 2 E5M10. Max ~65,504 Legacy. Needs loss scaling in training
FP8 (E4M3 / E5M2) 8 1 Per-tensor or per-block scale Production inference and training on Hopper, Blackwell, and MI300-class GPUs
MXFP8 8 ~1.03 OCP microscaling: 32-element blocks share an E8M0 scale Blackwell training and inference
INT8 8 1 Per-channel or per-group scale Broad hardware support. W8A8 via llm-compressor
MXFP4 4 ~0.53 E2M1 values, 32-element blocks share an E8M0 scale gpt-oss weights. Supported in vLLM, llama.cpp, and MLX
NVFP4 4 ~0.56 E2M1 values, 16-element blocks with an FP8 (E4M3) scale plus a per-tensor FP32 scale Blackwell-native FP4 inference. Supported in vLLM, TensorRT-LLM, SGLang, and MLX
INT4 4 0.5 (+ scales) Per-group scale (group size 32–128) GPTQ, AWQ, and GGUF Q4 variants
INT2 / ~2-bit 2 0.25 (+ scales) Codebook or lattice (IQ2, QTIP/EXL3) Extreme compression with noticeable quality loss

Quantization support per engine comes from the project docs: vLLM lists FP8, MXFP8/MXFP4, NVFP4, INT8, INT4, GPTQ/AWQ, GGUF, and compressed-tensors. mlx_lm.convert --q-mode accepts affine, mxfp4, nvfp4, and mxfp8.

GGUF Quantization Types

These are the official llama.cpp measurements for Llama 3.1 8B (tools/quantize/README.md, retrieved 2026-09-25). Bits/weight is the whole-file average, so it includes tensors that are kept at higher precision. Throughput numbers are for the maintainers' test machine and are only useful for relative comparison.

Type Bits/weight Size (GiB) Text generation t/s @128 Family
IQ1_S 2.00 1.87 79.7 I-quant (needs imatrix)
IQ2_XXS 2.38 2.23 79.9 I-quant
IQ2_M 2.93 2.74 74.4 I-quant
Q2_K 3.16 2.95 79.9 K-quant
IQ3_XXS 3.25 3.04 74.0 I-quant
Q3_K_S / Q3_K_M / Q3_K_L 3.64 / 4.00 / 4.30 3.41 / 3.74 / 4.02 69.8 / 71.7 / 69.4 K-quant
IQ4_XS 4.46 4.17 77.5 I-quant
Q4_K_S / Q4_K_M 4.67 / 4.89 4.36 / 4.58 76.7 / 71.9 K-quant. Q4_K_M is the common default
Q5_K_S / Q5_K_M 5.57 / 5.70 5.21 / 5.33 69.5 / 67.2 K-quant
Q6_K 6.56 6.14 58.7 K-quant
Q8_0 8.50 7.95 50.9 Legacy block quant
F16 16.00 14.96 29.2 Unquantized

The same README gives Q4_K_M sizes for Llama 3.1: 8B 32.1 GB to 4.9 GB, 70B 280.9 GB to 43.1 GB, 405B 1,625.1 GB to 249.1 GB (the original sizes are FP32).

Quality Impact by Precision

This table gives community perplexity measurements for Llama-2-7B (lower is better). The absolute values depend on the evaluation text and context length, so compare only the relative gaps. Use KL divergence against the BF16 model for a more reliable quality signal.

Format Perplexity Quality loss
FP16 (baseline) 7.4924 —
Q8_0 7.4933 Negligible
Q5_K_M ~7.52 Minimal
Q4_K_M 7.5692 Small
Q3_K_M ~7.85 Noticeable
Q2_K 8.6501 Significant degradation

Size and Speed Example (Llama 2 13B)

This is an illustrative community measurement on consumer hardware. Treat the tok/s figures as relative only.

Metric FP16 Q4_K_M
Model size 26 GB 7.9 GB (70% reduction)
RAM required 32 GB+ 12 GB
Speed 8 tok/s 15 tok/s
Quality 100% ~95%

Serving Engines and Runtimes

Versions come from PyPI release metadata unless noted (as of 2026-09-25).

Engine Latest version (date) License Maintainer Notes
vLLM 0.30.0 (2026-09-22) Apache-2.0 vLLM project (originated at UC Berkeley Sky Computing Lab) PagedAttention, continuous batching, prefix caching, disaggregated prefill/decode. OpenAI-compatible plus Anthropic Messages API. Roughly biweekly minor releases
SGLang 0.5.20 (2026-09-18) Apache-2.0 sgl-project RadixAttention prefix cache, zero-overhead scheduler, prefill/decode disaggregation. Default engine for DFlash "Spec V2"
TensorRT-LLM 1.2.1 stable (2026-04-20). 1.3.0rc28 pre-release (2026-09-23) Apache-2.0 NVIDIA PyTorch-based runtime. trtllm-serve <hf-model> serves Hugging Face checkpoints directly
llama.cpp build b11100 (2026-09-22, per release trackers) MIT ggml-org GGUF, CPU+GPU hybrid, many backends (CUDA, HIP, Metal, Vulkan, SYCL, WebGPU). A llama CLI with llama cli / llama serve subcommands now sits beside llama-cli and llama-server. Numbered v0.x releases sit beside nightly b-builds
Ollama v0.34.x (2026-09, per release trackers) MIT Ollama Wraps llama.cpp. REST API on port 11434. ollama launch integrates with coding agents
MLX / mlx-lm mlx 0.32.2 (2026-08-25). mlx-lm 0.31.3 (2026-04-22) MIT Apple Apple Silicon unified memory. LoRA fine-tuning built in
vllm-mlx 0.5.0 (2026-09-17) TBD Community vLLM-style serving on MLX
ExLlamaV3 1.5.1 (2026-09-22) MIT turboderp EXL3 format (QTIP-based). TabbyAPI is the recommended server
Text Generation Inference (TGI) Maintenance mode since 2025-12-11 Apache-2.0 Hugging Face Hugging Face recommends vLLM or SGLang (or llama.cpp or MLX locally)
LM Studio Desktop app. SDK lmstudio 1.5.0 (2025-08-22) Proprietary app, MIT SDK Element Labs GUI for GGUF and MLX models

Distributed Serving Orchestrators

Project Latest (date) License Governance Role
NVIDIA Dynamo 1.5.0 (2026-09-19). Nightly 1.6.0.dev builds Apache-2.0 NVIDIA, open source Orchestration above vLLM, SGLang, and TensorRT-LLM. Disaggregated prefill/decode, KV-aware routing, multi-tier KV cache. Built in Rust and Python
llm-d v0.7 (2026-05) Apache-2.0 CNCF Sandbox (joined 2026-03-24). Founded by Red Hat, Google Cloud, IBM Research, CoreWeave, and NVIDIA Kubernetes-native. Prefix-cache-aware routing, tiered KV offload, wide expert parallelism, P/D disaggregation

Quantization and Fine-Tuning Tooling

Tool Latest version (date) Status
llm-compressor (vLLM) 0.14.0 (2026-09-22) Active. Produces compressed-tensors checkpoints (FP8, INT8, INT4, NVFP4) for vLLM. Absorbed AutoAWQ
GPTQModel 7.5.0 (2026-09-15) Active successor to AutoGPTQ
AutoGPTQ 0.7.1 (2024-03-01) Unmaintained. The README points users to GPTQModel
AutoAWQ 0.2.9 (2025-05-11) Deprecated. Adopted by vLLM llm-compressor. MLX-LM supports AWQ on Mac
bitsandbytes 0.50.2 (2026-08-27) Active. 4-bit NF4 and 8-bit loading for QLoRA
Transformers 5.17.0 (2026-09-09) Active. v5 line
PEFT 0.21.0 (2026-09-15) Active. LoRA, DoRA, IA3, prefix and prompt tuning
Unsloth 2026.9.11 (2026-09-23) Active. Apache-2.0
Axolotl 0.19.0 (2026-09-10) Active
MergeKit 0.1.4 (2025-10-31) LGPL-3.0

Structured Output Libraries

Library Latest version (date) Approach Overhead Integration
XGrammar 0.2.8 (2026-09-24) Pushdown automaton with precomputed token bitmasks Near zero (~1%, vendor-reported) vLLM, SGLang, MLC, TensorRT-LLM
llguidance 1.8.0 (2026-08-11) Earley-style grammar engine (Microsoft) Low Guidance, vLLM (guidance backend), llama.cpp
Outlines 1.3.3 (2026-08-06) Regex/CFG compiled to token masks Low to moderate Transformers, vLLM, llama.cpp
Guidance — Template language with token healing Low Standalone

vLLM exposes these through structured_outputs (json, regex, choice, grammar, structural_tag) and OpenAI-style response_format. The legacy guided_* request fields were removed in vLLM v0.12.0.

GPU Hardware Specs

NVIDIA Data Center GPUs

The TFLOPS figures are dense Tensor Core throughput (without 2:4 sparsity) from NVIDIA datasheets. Datasheets often quote sparse numbers, which are 2x higher.

GPU Memory Bandwidth FP16/BF16 dense TFLOPS FP8 dense TFLOPS Notes
A100 40GB / 80GB 40 GB HBM2 / 80 GB HBM2e 1.6 / 2.0 TB/s 312 — Ampere. No FP8
A10G 24 GB GDDR6 600 GB/s ~70 (AWS does not publish an official figure) — AWS g5 instances
L4 24 GB GDDR6 300 GB/s 121 242 Ada. Edge and cost-sensitive inference
L40S 48 GB GDDR6 864 GB/s 362 733 Ada. Inference-optimized
H100 SXM 80 GB HBM3 3.35 TB/s 989 1,979 Hopper
H100 NVL 94 GB HBM3 3.9 TB/s ~835 ~1,671 PCIe dual-card form factor with a lower power budget than SXM
H200 141 GB HBM3e 4.8 TB/s 989 1,979 Hopper compute with more memory
B200 180–192 GB HBM3e 8 TB/s ~2,250 ~4,500 Blackwell. Adds FP4 (NVFP4)
B300 (Blackwell Ultra) 288 GB HBM3e 8 TB/s TBD TBD Higher FP4 throughput than B200

AMD Data Center GPUs

GPU Memory Bandwidth Notes
MI300X 192 GB HBM3 5.3 TB/s CDNA 3. Supported by vLLM, SGLang, and llm-d
MI325X 256 GB HBM3e 6 TB/s CDNA 3 refresh
MI355X 288 GB HBM3e 8 TB/s CDNA 4. Adds FP4/FP6

Consumer and Workstation GPUs

GPU VRAM Bandwidth Practical use
RTX 3090 24 GB GDDR6X 936 GB/s Popular on the used market for QLoRA
RTX 4080 16 GB GDDR6X 717 GB/s 7B–8B Q4 with room for KV cache
RTX 4090 24 GB GDDR6X 1,008 GB/s 13B–14B Q4 comfortably. ~30B Q4 with short context
RTX 5090 32 GB GDDR7 1,792 GB/s ~30B Q4. Blackwell consumer card with FP4 support
RTX PRO 6000 Blackwell 96 GB GDDR7 (ECC) ~1.8 TB/s Workstation card. 70B Q4 or ~100B-class MoE at 4-bit

Apple Silicon

Chip Unified memory Bandwidth Notes
M4 Pro 24–64 GB 273 GB/s 8B BF16 or ~30B Q4
M4 Max 36–128 GB 410–546 GB/s (by GPU bin) 70B Q4 on the 128 GB configuration
M3 Ultra (Mac Studio, 2025) 96–512 GB 819 GB/s Can hold very large MoE models at 4-bit (for example 671B-class) in 512 GB
M5 (2025-10) Up to 32 GB 153 GB/s ~28% more bandwidth than M4 (120 GB/s). Apple reports 19–27% faster LLM token generation
M5 Pro / M5 Max (2026) TBD TBD Shipping (the DFlash 2 notes reference M5 Max), specs not verified in this refresh

There is no M4 Ultra

Apple's 2025 Mac Studio paired the M4 Max with the M3 Ultra. Earlier versions of this page listed a non-existent "M4 Ultra".

Evaluation Benchmarks

Benchmark What it tests Format Status (2026-09)
MMLU General knowledge (57 subjects) Multiple choice Saturated. Use MMLU-Pro
MMLU-Pro Harder MMLU with 10 choices Multiple choice Near saturation at the frontier
GPQA (Diamond) Graduate-level science Expert-written multiple choice Near saturation at the frontier
HLE (Humanity's Last Exam) 2,500 expert questions Multi-modal, short answer and multiple choice Hard frontier benchmark
HumanEval Code generation (Python functions) Generate code, run unit tests Saturated (top models >90%)
SWE-bench Verified Real GitHub issues, human-validated subset Navigate repo, write patch Top models vendor-reported around 80% in 2026. Harder successors include SWE-bench Pro
Terminal-Bench Agentic tasks in a terminal Execute multi-step shell tasks Common agentic-coding benchmark
LiveCodeBench Rolling competitive-programming problems Generate code Contamination-resistant (date-filtered)
GSM8K Grade-school math (2–8 steps) Chain-of-thought Saturated. Contamination concerns
MATH / MATH-500 Competition math Multi-step symbolic reasoning DeepSeek-R1 97.3% (2025-01). Saturated
AIME Olympiad-qualifier math Integer answers Refreshed yearly. DeepSeek-R1 79.8% on AIME 2024
ARC-AGI-2 Abstract visual reasoning puzzles Grid transformations Hard. Designed to resist memorization
IFEval Instruction following Verifiable format constraints Tests structured output compliance
LMArena (formerly Chatbot Arena) Human preference Blind pairwise voting, Elo-style rating Widely watched. Biased toward chat-style tasks

Use benchmarks carefully

  • Saturation: MMLU, GSM8K, and HumanEval are largely solved. Score differences at 90%+ are often noise.
  • Contamination: GSM1k (Scale AI, 2024) found accuracy drops of up to ~13% on fresh GSM8K-style problems for some model families.
  • Task mismatch: match benchmarks to your use case. MMLU does not measure coding, and HumanEval does not measure real engineering.
  • Vendor-reported scores: harnesses, scaffolds, and attempt counts differ. Compare scores only within one harness.

Vector Databases

Database Strength Scale Managed option
Pinecone Zero-ops managed service Billions Yes (primary offering)
Weaviate Hybrid search + modules Millions–billions Yes
Milvus Distributed, billion-scale Billions Zilliz Cloud
Qdrant Rich payload filtering (Rust) Millions–billions Yes
Chroma Developer-friendly, lightweight Millions Chroma Cloud
pgvector PostgreSQL extension. No new infra Millions Any managed Postgres

Additional PEFT Methods

Beyond LoRA, QLoRA, and adapters (see How-to Guides):

Method How it works Trainable params Best for
DoRA (Weight-Decomposed LRA) Decomposes weights into magnitude + direction. LoRA on direction only ~same as LoRA Better quality at the same rank. Drop-in LoRA replacement
Prefix Tuning Prepends trainable "virtual tokens" to each layer's K and V ~0.1% Few-shot task adaptation
Prompt Tuning Adds trainable embeddings to the input only (not each layer) ~0.01% Extremely lightweight. Classification
IA3 Learns rescaling vectors for K, V, and FFN activations ~0.01% Few-shot with minimal parameters

Guardrail Frameworks

Framework Provider Architecture Key strength
NeMo Guardrails NVIDIA (v0.24.1, 2026-09-16) Programmable rails in Colang. Input, output, dialog, retrieval, and execution rails Conversation flow control. Injection detection (code, SQLi, XSS, template)
Guardrails AI Open source Validator pipeline with schema enforcement JSON validation, PII redaction, toxicity
LLM Guard Protect AI 15 input scanners + 20 output scanners Self-hosted, works with any LLM
Llama Guard Meta Fine-tuned LLM safety classifier Categorizes prompts and responses as safe or unsafe per a hazard taxonomy
Lakera Guard Lakera Cloud API Specialized prompt-injection detection
Constitutional Classifiers Anthropic Cascade: cheap exchange classifier escalates to a stronger classifier Next-gen (2026-01): ~1% compute overhead on Claude Opus 4.0 traffic. One high-risk finding in 198,000 red-team attempts (0.005 per 1K queries)
Azure AI Content Safety Microsoft Cloud API Real-time classification with severity scores. Prompt Shields for injection
OpenAI Guardrails OpenAI Python SDK wrapper Drop-in input/output validation for the OpenAI API

Rail Types

Rail type When What
Input rails Before the LLM processes a request Reject harmful prompts, mask PII, detect injection
Output rails After the LLM generates a response Filter toxic content, validate format
Dialog rails During multi-turn conversation Enforce flow, prevent topic drift
Retrieval rails In RAG pipelines Filter harmful retrieved chunks

OWASP LLM Top 10 (2025)

# Vulnerability Description Key mitigation
LLM01 Prompt Injection Manipulation via crafted inputs. Direct or indirect Input classification, structured queries, defense-in-depth
LLM02 Sensitive Information Disclosure Leaking private data from training or context Output filtering, PII scanning, differential privacy
LLM03 Supply Chain Compromised models, data, plugins, or dependencies Safetensors, provenance tracking, ML-BOM
LLM04 Data and Model Poisoning Tampered training data or model weights Data provenance, anomaly detection
LLM05 Improper Output Handling Unvalidated LLM outputs causing downstream exploits Output validation, escaping, content security policies
LLM06 Excessive Agency LLMs granted too many permissions or capabilities Least privilege, human-in-the-loop, permission boundaries
LLM07 System Prompt Leakage Exposure of internal instructions, credentials, or logic Keep secrets out of prompts. Enforce authorization outside the LLM
LLM08 Vector and Embedding Weaknesses RAG poisoning, embedding inversion, unauthorized access Access controls on vector stores, source validation
LLM09 Misinformation Unreliable outputs leading to flawed decisions Grounding via RAG, citations, human review
LLM10 Unbounded Consumption Excessive resource usage (DoS, denial of wallet, model extraction) Rate limiting, token budgets, cost circuit breakers

New in 2025: System Prompt Leakage (LLM07) and Vector/Embedding Weaknesses (LLM08). Excessive Agency was expanded for agentic systems. Insecure Plugin Design and Model Denial of Service were folded into broader categories.

Glossary

Term Definition
Active parameters Parameters used per token in an MoE model (for example 37B of DeepSeek-V3's 671B)
AWQ Activation-aware Weight Quantization. Protects salient weight channels found from activation statistics
BPE Byte Pair Encoding. Subword tokenizer built by merging frequent byte pairs
Continuous batching Scheduler that adds and removes requests from the running batch every decode step
Decode Token-by-token generation phase. Memory-bandwidth bound
Disaggregated serving Running prefill and decode on separate GPU pools, with KV cache transfer between them
DPO Direct Preference Optimization. Preference alignment without a reward model
EXL3 ExLlamaV3 quantization format based on QTIP trellis quantization
GGUF llama.cpp's single-file model format (weights, tokenizer, metadata, chat template)
GQA / MQA / MLA Grouped-query, multi-query, and multi-head latent attention. Ways to shrink the KV cache
GRPO Group Relative Policy Optimization. PPO variant with group-normalized advantages and no value model
imatrix Importance matrix from calibration text, used by llama.cpp to weight quantization error
ITL / TPOT Inter-token latency / time per output token
KV cache Stored attention keys and values of previous tokens, reused at every decode step
LoRA / QLoRA Low-rank adapter fine-tuning. QLoRA trains adapters on a 4-bit (NF4) frozen base
MoE Mixture of Experts. Router sends each token to top-k expert FFNs
MTP Multi-token prediction heads, usable as a built-in speculative drafter
MXFP4 / NVFP4 Block-scaled 4-bit float formats (OCP microscaling, NVIDIA)
PagedAttention vLLM's block-based KV cache allocator, modeled on OS virtual memory paging
Prefill Processing the whole prompt in parallel to build the KV cache. Compute bound
RadixAttention SGLang's radix-tree prefix cache that shares KV blocks across requests
RoPE Rotary position embedding. Encodes relative position by rotating Q and K
Speculative decoding A cheap drafter proposes tokens that the target model verifies in one pass. Lossless
TTFT Time to first token. Dominated by queueing + prefill

Sources