Skip to content

LLM Inference

Techniques, engines, and serving infrastructure for making large language model generation faster and cheaper — speculative decoding, KV-cache engineering, scheduling, and hardware-aware serving.

This domain covers the optimization layer that sits between a trained model and its users. Since 2025 the field has consolidated around a small set of levers: parallel (non-autoregressive) drafting, target-conditioned draft models, overlap scheduling that hides host latency, and block sizes tuned to hardware verification costs. Agent workloads — long, token-hungry, latency-sensitive — are the demand driver; see AI Agents for the consumer side.

Topics

Topic What It Is Status
DFlash 2 Block-diffusion speculative-decoding drafter with candidate path selection and dynamic short convolutions — lossless ~3x decoding at ~1/3 compute per token active, Aug 2026

Landscape Notes (2026)

  • Speculative decoding is the default. Every major engine (SGLang, vLLM, TensorRT-LLM, llama.cpp, ollama) ships at least one speculation path, and frontier open models (Qwen, Gemma, Kimi, DeepSeek) ship native MTP drafters. The differentiator is draft quality per unit of draft latency.
  • Parallel drafting displaced autoregressive drafting. DFlash's block-diffusion approach — draft an entire block in one forward pass, condition on target hidden states via KV injection — set the pattern that successors (DFlash 2, Domino, DSpark, JetSpec) refine.
  • Verification is not the bottleneck on datacenter hardware. UCSD's TPU v5p measurements ("K-flat verification") show verifying 1024 tokens costs nearly the same as verifying 16; draft quality, not width, is the frontier.
  • Gains shrink with concurrency. Speculation speedups are largest at batch size 1-8 and compress toward 1x at concurrency 32+; engine-level scheduling (overlap, Spec V2) matters as much as the drafter at high load.

Engine Landscape (2026)

Engine Niche Speculation Path
SGLang Production serving; Spec V2 overlap scheduler is the current performance frontier --speculative-algorithm DFLASH + drafter path
vLLM Broadest model coverage; GPU via Speculators library, TPU via tpu-inference (JAX) --speculative-config JSON, "method": "dflash"
TensorRT-LLM NVIDIA-tuned serving on Blackwell/Hopper DFlash supported; flags not yet publicly documented
llama.cpp Local GGUF inference, single request --spec-type draft-dflash (speculation blocks concurrency scaling)
ollama Easiest local UX DFlash 2 branch (PR #17865), experimental draft models
oMLX Apple Silicon GUI server (MLX) DFlash via model-manager settings

Comparisons

Canonical comparison notes live in comparisons/ — currently DFlash 2 vs EAGLE-3 vs MTP: the three deployable speculative-decoding draft paths, measured head-to-heads, and a weighted decision matrix.

FAQ

  • Is speculative decoding output really identical? Yes when implemented correctly — rejection sampling guarantees the target distribution. Engine bugs, not the algorithm, are the historical integrity risk; see DFlash 2 Security.
  • Why not just add a bigger draft model? Draft cost, not draft size, is the old constraint — parallel (block-diffusion) drafting flattened it, shifting the frontier to draft quality per cycle.
  • Do I need a new drafter per model? Yes — drafters are per-target artifacts; check the Hugging Face collections before assuming support for a given checkpoint.
  • Does any of this help at concurrency 64+? Published DFlash 2 data stops at 32 (1.0-1.45x); at high concurrency, engine scheduling usually matters more than the drafter.
  • Where do drafters come from? Official HF orgs (incoai/z-lab, plus vendor orgs like nvidia, RedHatAI, modal-labs) or self-training via the NVIDIA NeMo recipe — see DFlash 2 Operations.

Reading Paths

  • "Why is my LLM serving slow?" — start with DFlash 2; speculative decoding is the single largest lever for interactive (batch 1-8) workloads.
  • "Should I roll it out?" — read the concurrency table in DFlash 2 Architecture first, then the rollout checklist in DFlash 2 Operations.
  • "Can I trust speculative output?" — DFlash 2 Security covers losslessness as an output-integrity guarantee and the drafter supply chain.

Key Concepts

Concept Meaning
Speculative decoding A small drafter proposes tokens; the target LLM verifies the block in one pass and accepts the longest valid prefix — lossless by rejection sampling
Acceptance length Mean committed tokens per draft-verify cycle; the single best health/quality metric for a speculation setup
Block size (K) Tokens drafted per cycle; draft cost is nearly flat in K on modern hardware ("K-flat verification")
KV injection Conditioning technique: target hidden states feed the drafter's KV cache at every layer (DFlash's approach), keeping deep drafters accurate
Suffix decay Accuracy loss toward the end of a drafted block; fixed by local mixing (DFlash 2's convolutions) rather than depth
Interactivity Per-user output token rate (tok/s/user) — the latency-side metric of the throughput-latency Pareto curve
Overlap scheduling Hiding host-side scheduler work behind GPU work (SGLang Spec V2); worth >30% at high concurrency
Drafter The auxiliary model itself; per-target, typically 1-2B params, distributed via Hugging Face collections

Reference Sources

  • AI Agents — the workload class that makes inference latency the binding constraint
  • Unified Tools Catalogue — inference and serving tools under "Inference, Serving & Fine-tuning"