Skip to content

DFlash 2

Summary

DFlash 2 is the second-generation block-diffusion speculative-decoding drafter from Inco AI, built on the DFlash paper (Chen, Liang, Liu — Z Lab at UCSD, ICML 2026). It drafts an entire block of tokens in a single forward pass, then adds two cheap modules on top: a pairwise candidate path selector that stitches independently-predicted positions into one coherent path, and a two-tap dynamic convolution that stops draft accuracy from decaying toward the end of the block. Output is provably unchanged from the target model (lossless). Result: 16-25% more accepted tokens per verification pass for ~1.3% added cycle latency, and 2.7-3.4x the throughput of autoregressive decoding at batch size 1 (roughly a third of the compute per token).

Evaluation

  • Why it's better: Autoregressive drafters (EAGLE-3, native MTP) pay a serial forward pass per draft token, capping speculation depth. DFlash 2 keeps the one-pass parallel draft and recovers the two accuracy losses it had — incoherent top-1 picks (fixed by the path selector) and suffix decay (fixed by the convolution) — while beating both its own v1 drafter, native MTP, and community DSpark drafters on every benchmark tested (GSM8K, MATH-500, HumanEval, MBPP, MT-Bench).
  • When it fits: Latency-sensitive, low-concurrency serving (batch size 1-8) — interactive coding, reasoning, and agent workloads. Local single-user inference on Apple Silicon (oMLX, llama.cpp, ollama). TPU serving (vLLM tpu-inference).
  • When it does not fit: High-concurrency throughput serving, and targets without a published drafter (see Cons). As of Aug 2026 the DFlash v1 collection covers ~20 more models beyond the two launch drafters; native MTP even dips below 1x on some tasks at concurrency 32.
Pros Cons
Lossless — greedy output matches target exactly; sampling preserves its distribution Needs a drafter trained per target model; no official training code in the repo ("checkpoint-locked", z-lab/dflash Issue #1)
One-pass parallel drafting: draft cost is nearly independent of block size (K-flat verification on TPUs/v5p) Gains compress at high concurrency (1.01-1.45x at concurrency 32)
Tiny overhead: +1.3% draft-verify cycle latency combined (per-module split in Architecture) Only two DFlash 2 drafters published at launch
Broad engine support out of the gate: SGLang, vLLM, TensorRT-LLM, llama.cpp, ollama, oMLX llama.cpp speculative decoding prevents concurrency scaling (single-request oriented)
Beats DSpark correction with ~40x fewer added parameters and 16x lower latency overhead MLX quantized targets must drop to block size <= 5 (quantized matmul kernel efficiency)
  • Common Use Cases:
    • Agent serving stacks that need per-user interactivity (500-600 tok/s) at high concurrency — NVIDIA measured 15x higher throughput than autoregressive decoding at the same interactivity for gpt-oss-120b on 8x Blackwell Ultra.
    • Local agentic coding on Apple Silicon: Qwen3.8-27B 4-bit + DFlash 2 drafter on M5 Max via oMLX.
    • Production endpoints: CoreWeave's Kimi K2.7 Code endpoint (fastest on Artificial Analysis for that model) runs the DFlash v1 drafter by default; DFlash 2 is the drop-in upgrade path.
  • Licensing & Commercial Use: The z-lab/dflash code is MIT (c) 2026 Z Lab. Drafter checkpoints are on Hugging Face (incoai and z-lab orgs); checkpoint-level license terms are not stated on the model cards — TBD, verify per checkpoint before commercial redistribution.
  • Ecosystem & Data Connections: Runs inside SGLang (default Spec V2 engine), vLLM (via the Speculators library), TensorRT-LLM, llama.cpp, ollama, oMLX, and vLLM TPU / tpu-inference (JAX). Engine integration points (PR references and pinned builds) are tracked in Operations and Security. Drafters distributed through Hugging Face collections: incoai/dflash-2, z-lab/dflash, z-lab/dflash-2 (all linked in Sources below).
  • Compatibility & Requirements: A matching DFlash 2 drafter for the exact target model. CUDA (Blackwell and Hopper recipes published), Google TPU v5p, Apple Silicon (MLX). The drafter is not a standalone LLM — it requires a speculation-aware server. Python 3.10+ for the dflash CLI/benchmark client.
  • Latest Versions: DFlash paper arXiv:2602.06036 (v1 Feb 2026, v2 May 2026, ICML 2026); DFlash 2 announced Aug 2026 with two drafters; >3.5M Hugging Face downloads across DFlash models (Aug 2026).
  • Alternatives: EAGLE-3 (autoregressive drafter, input-only conditioning), native MTP modules (ships with the model, 7-token serial draft), DSpark (sequential correction heads, +77.8M params/+9.6% latency), Domino (serial GRU correction), JetSpec (causal in-block attention + forward-KL distillation), SpecDiff-2 (full diffusion LLM as drafter — high memory footprint), Medusa (parallel heads, older generation).
  • Migration & Lock-in Risks: Migration from EAGLE/MTP is a config change in SGLang/vLLM (swap algorithm + drafter path — no application refactoring). Lock-in is model-shaped, not vendor-shaped: each target model needs its own drafter, and third-party drafter training currently means NVIDIA NeMo AutoModel recipes or Z Lab/Modal engagement. The algorithm itself is open (paper + MIT code).
  • Community Health: Active and industry-backed. Z Lab (UCSD) maintains the repo; SGLang/Modal/Z Lab co-engineered the serving path; NVIDIA published Blackwell/Hopper recipes and 20 v1 checkpoints; Google Cloud co-published the TPU port; drafters published by NVIDIA, Red Hat, Modal, Meta, Poolside, Xiaomi. Known friction: training code not yet released (Issue #1), a CUDA-graph crash under load (Issue #146).

Notes In This Folder

  • Architecture — draft/verify cycle, KV injection, path selector, two-tap convolution, benchmarks across NVIDIA/TPU/H200
  • Operations — serve with SGLang/vLLM/llama.cpp/ollama/oMLX, dflash CLI, block-size tuning, training drafters with NeMo
  • Security — losslessness as an output-integrity guarantee, drafter supply chain, privacy and data flow, threat model
  • Comparisons — index pointing to the canonical DFlash 2 vs EAGLE-3 vs MTP note

Sources

All URLs verified HTTP 200 on 2026-08-28.

Questions

  • Will Z Lab/Inco release official drafter training code, ending the "checkpoint-locked" limitation? (Operations documents the interim NeMo path)
  • What are the checkpoint-level license terms for the incoai/z-lab HF drafters? (model cards are silent — TBD)
  • How large is the DFlash 2 gain at concurrency 64+ and with chunked-prefill production traffic mixes? (published data stops at concurrency 32)
  • Does a DFlash 2 drafter arrive for Qwen3.5-397B-A17B and Kimi-class MoE models, where v1 DFlash already beat native MTP at every concurrency tested?

Quick Facts

Field Value
Full name DFlash 2: Keep Drafting Parallel (Inco AI, Aug 2026)
Lineage DFlash — Chen, Liang, Liu (Z Lab at UCSD), arXiv:2602.06036, ICML 2026
Type Speculative-decoding drafter (not a standalone LLM)
Code license MIT, (c) 2026 Z Lab
Launch drafters Qwen3.8-27B (2B params, BF16) and Muse-Glimmer-30B
Headline numbers 2.7-3.4x AR throughput @ batch 1; +16-25% accepted tokens/pass; +1.3% cycle latency
Engines SGLang, vLLM, TensorRT-LLM, llama.cpp, ollama, oMLX, vLLM TPU (JAX)