Skip to content

DFlash 2 Reference

Context

Look-up facts for DFlash 2 and its DFlash (v1) base: release timeline, engine support matrix with first-supporting releases, drafter checkpoints and their licenses, the drafter config keys every engine reads, per-engine flags, the NeMo training config, and all published benchmark tables with their test conditions. For how the drafter works, see Explanation. For deployment steps, see How-to Guides.

Release Timeline

Date Event Source
2026-01-06 z-lab/dflash Issue #1 opened (request for training code). Still open Issue #1
2026-02-05 DFlash paper v1 on arXiv (2602.06036). The Inco blog says DFlash was "released in January" arXiv
2026-05-28 DFlash paper v2 on arXiv. Accepted at ICML 2026 arXiv v2, ICML poster
2026-06-15 LMSYS blog: DFlash on the SGLang Spec V2 engine, Qwen3.5-397B-A17B drafter LMSYS
2026-06-23 NVIDIA blog: up to 15x throughput on Blackwell Ultra (TensorRT-LLM, DFlash v1) NVIDIA
2026-07-09 z-lab/dflash Issue #146 (CUDA-graph crash under load on vLLM 0.22.1) Issue #146
Aug 2026 DFlash 2 announced by Inco AI with two drafters (Qwen3.8-27B, Muse-Glimmer-30B) Inco blog
2026-08-18 dflash 0.1.0 on PyPI (MIT, Python >= 3.10) PyPI
2026-08-19 SGLang PR #35371 (DFlash2) merged PR #35371
2026-08-21 vLLM PR #52816 (DFlash2) merged PR #52816
2026-08-22 NeMo AutoModel PR #3605 (DFlash 2 draft model, trainer, recipe) merged PR #3605
2026-08-26 vLLM v0.28.0: first vLLM release with DFlash 2 vLLM v0.28.0
2026-08-27 llama.cpp PR #27342 (DFlash2) merged PR #27342
2026-09-04 SGLang v0.5.19: first SGLang release with DFlash 2 SGLang v0.5.19
2026-09-18 Inco AI releases Splash, an Apple-silicon engine that ships DFlash 2 drafts incoai/splash
2026-09-23 TensorRT-LLM 1.3.0rc28: first tagged build with the DFlash 2 selector path (pre-release) PyPI

Release dates for the engine releases and the dflash package come from PyPI upload times (UTC). The first-supporting release is confirmed from the source at each tag: the DFlash 2 draft class (DFlash2DraftModel) is in SGLang v0.5.19 but not v0.5.18, and in vLLM v0.28.0 but not v0.27.1.

Engine Support Matrix

Status as of 2026-09-25.

Engine DFlash (v1) DFlash 2 First release with DFlash 2 Latest release (date) Status
SGLang --speculative-algorithm DFLASH (PR #22077, then Spec V2 in PR #23000) Same DFLASH algorithm. DFlash2DraftModel is picked from the checkpoint config (PR #35371) v0.5.19 (2026-09-04) 0.5.20 (2026-09-18) Mainline
vLLM method: dflash in --speculative-config Same method. DFlash2DraftModel maps to qwen3_dflash2 (PR #52816) v0.28.0 (2026-08-26) 0.30.0 (2026-09-22) Mainline
TensorRT-LLM DFlashDecodingConfig / decoding_type: DFlash (in 1.3.0rc15, 2026-05-21, and later) Same config, no extra arguments 1.3.0rc28 (2026-09-23, pre-release) Stable 1.2.1 has no DFlash. Pre-release 1.3.0rc28 Main branch and pre-releases only
llama.cpp --spec-type draft-dflash Auto-detected from the GGUF (selector_top_k > 0) under the same type (PR #27342) master after 2026-08-27 Rolling bNNNN builds Mainline
ollama — PR #17865 "mlx: add DFlash2 support" (MLX engine only) None — Open PR, branch build
oMLX Fork z-lab/omlx-fork release 0.6.2-dflash2 (signed arm64 dmg) Fork only — Fork build
Splash (Inco AI) — Ships a DFlash 2 draft inside each model package 2026-09-18 launch — Vendor engine, Apache-2.0
vLLM TPU (tpu-inference) JAX port (Google blog) TBD — no DFlash 2 port found as of 2026-09-25 — — v1 only

Community ports reported after the DFlash 2 launch (not verified individually): vLLM-Ascend, SGLang on Ascend NPU, exllamav3. Other runtimes (for example pegainfer, rMLX) track DFlash 2 ports in open issues.

Drafter Checkpoints

DFlash 2 Drafters

Checkpoint Target Format License Notes
incoai/Qwen3.8-27B-DFlash2 (mirror z-lab/Qwen3.8-27B-DFlash2) Qwen/Qwen3.8-27B Safetensors, BF16, ~2B params Apache-2.0 (HF metadata) Launch drafter. Block size 8 (7 draft tokens)
incoai/Qwen3.8-27B-DFlash2-GGUF (mirror z-lab/Qwen3.8-27B-DFlash2-GGUF) Qwen3.8-27B GGUF GGUF Check the card llama.cpp path
incoai/Muse-Glimmer-30B-DFlash2 (mirror z-lab/Muse-Glimmer-30B-DFlash2) meta-models/Muse-Glimmer-30B Safetensors TBD — check the card Launch drafter. Launch evals used block size 16
incoai/GLM-5.3-DFlash2 zai-org/GLM-5.3 Safetensors TBD — check the card Post-launch drafter
incoai/GLM-5.3-Flash-DFlash2 GLM-5.3-Flash Safetensors CC BY-NC-ND 4.0 (research and evaluation). Commercial licensing from Inco Post-launch drafter. No derivatives: do not ship requantized copies
incoai/Qwen3.8-27B-Splash, incoai/Qwen3.6-35B-A3B-Splash Qwen3.8-27B, Qwen3.6-35B-A3B (4-bit) Splash package (17.4 GB, 20.9 GB) Check the card Bundles target + DFlash 2 draft. Loads only in Splash

Licenses differ per checkpoint

The code (z-lab/dflash, the dflash package) is MIT. Drafter weights are not covered by that license. Read each model card before commercial use.

DFlash (v1) Targets

The DFlash collection lists these targets (z-lab README, 2026-09-25), about 25 checkpoints:

Family Targets
Qwen Qwen3.6 (27B, 35B-A3B). Qwen3.5 (4B, 9B, 27B, 35B-A3B, 122B-A10B, 397B-A17B). Qwen3 (4B and 8B non-thinking, Coder-Next, Coder-30B-A3B)
Gemma Gemma 4 (12B, 31B, 26B-A4B)
MiniMax M2.5, M2.7
Kimi K2.5, K2.6, K2.7-Code
Others gpt-oss (20B, 120B), Llama-3.1-8B, GLM 5.1, Alpamayo 1.5/R1 10B

Vendor-published v1 drafters also exist, for example nvidia/Kimi-K2.7-Code-DFlash and modal-labs/Qwen3.5-397B-A17B-DFlash (mirrored as lmsys/ and z-lab/).

Drafter Config Keys

Engines rebuild the drafter from the dflash_config block in the drafter's config.json. The DFlash 2 keys are all optional. Their presence switches an engine to the DFlash 2 model class.

Key Applies to Meaning Example value
block_size DFlash, DFlash 2 Verify window the drafter was trained for (draft tokens = block size - 1) 8 (Qwen3.8-27B), 16 (NeMo example)
mask_token_id DFlash, DFlash 2 Token that fills block positions 1..K-1. Must match training 151669 (NeMo Qwen3 example)
target_layer_ids DFlash, DFlash 2 Target layers whose hidden states are injected For example [1, 9, 17, 25, 33]
conv_kernel_size DFlash 2 Convolution taps (2 = position + predecessor). Needs conv_group_size 2
conv_group_size DFlash 2 Channels that share one dynamic tap correction 16
selector_rank DFlash 2 Width of the selector's A/B token codebooks and context gate. Needs selector_top_k 256
selector_top_k DFlash 2 Candidates kept and scored per position 16

SGLang rejects a config where only one of a pair is set (conv_kernel_size/conv_group_size, selector_rank/selector_top_k). TensorRT-LLM reads mask_token_id and target_layer_ids from the checkpoint when unset and warns if max_draft_len + 1 differs from block_size.

Engine Flags

SGLang

Flag Meaning
--speculative-algorithm DFLASH Enables DFlash and DFlash 2 (built-ins: EAGLE, EAGLE3, NEXTN, STANDALONE, NGRAM, DFLASH, DSPARK)
--speculative-draft-model-path <repo> Drafter checkpoint
--speculative-dflash-block-size <n> DFLASH only. Alias of --speculative-num-draft-tokens
--speculative-num-draft-tokens <n> Verify window length for DFLASH
--speculative-draft-attention-backend <b> Drafter attention backend: flashinfer, fa3, fa4, triton, ascend, trtllm_mha (DFLASH rejects trtllm_mha when its per-layer needs are not met)
--speculative-dflash-draft-window-size <n> Sliding window for the drafter's local KV cache
SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 Env var used in the LMSYS Qwen3.5-397B launch command

vLLM

--speculative-config key Meaning
method "dflash" for DFlash and DFlash 2 (vLLM can also infer it when the drafter name contains "dflash")
model Drafter checkpoint
num_speculative_tokens K draft tokens. DFlash uses one bonus query plus K mask queries, so K = block size - 1

llama.cpp

Flag Meaning
--spec-type draft-dflash DFlash drafting. DFlash 2 GGUFs are detected automatically
-hfd / --spec-draft-hf <user>/<model>[:quant] Drafter from a Hugging Face repo
-md <path> Local drafter GGUF
--spec-draft-n-max <n> Tokens to draft per step (env LLAMA_ARG_SPEC_DRAFT_N_MAX)

TensorRT-LLM (DFlashDecodingConfig)

Field Meaning
max_draft_len Draft tokens per step. Set to block_size - 1 of the checkpoint
speculative_model Path or HF id of the drafter
mask_token_id, target_layer_ids Read from the drafter config when None
attention_backend Cross-attention backend: VANILLA (default, runs anywhere), TRTLLM (SM100/SM103 only), FA4 (SM90 only)
skip_ctx_buffer_budget_check Skip the conservative config-time memory check for the context K/V buffers
decoding_type: DFlash YAML form for trtllm-serve / trtllm-bench

dflash CLI (0.1.0)

Command Backends Key options
dflash generate <backend> transformers, mlx, openai --model, --draft, --reasoning, --temperature, --top-p, --top-k, --block-size, --draft-bits, --base-url
dflash benchmark <backend> transformers, mlx, openai --dataset (gsm8k, math500, humaneval, mbpp, mt-bench), --num-prompts, --concurrency, --max-samples

Install extras: pip install dflash (client for an OpenAI-compatible server) or pip install "dflash[local]" (MLX on Apple Silicon, Transformers on Linux). Backend coverage: Transformers runs DFlash 2 for Muse-Glimmer-30B and DFlash for Qwen3 and Llama-3.1-8B. MLX runs DFlash 2 for Qwen3.8-27B and DFlash for Qwen3, Qwen3.5, Qwen3.6, and Gemma 4.

Reasoning controls: Qwen3.8 uses reasoning_effort (low, medium, xhigh, default xhigh). Muse Glimmer uses reasoning_strength (low, medium, high, xhigh, default high).

NeMo AutoModel Training Config

Values from examples/speculative/dflash/qwen3_dflash2.yaml (recipe TrainDFlash2Recipe, entry point nemo_automodel.recipes.llm.train_dflash2).

Field Example Meaning
target_model_name_or_path Qwen/Qwen3-8B Frozen target
draft_num_hidden_layers 5 Drafter depth
block_size 16 Tokens drafted per block
num_anchors 256 Blocks sampled per sequence per step
loss_decay_gamma 7.0 Position decay of the block loss
mask_token_id 151669 Required. A reserved, unused token, never pad
attention_backend flex_attention Or sdpa as the portable fallback
conv_kernel_size / conv_group_size 2 / 16 DFlash 2 convolution
selector_rank / selector_top_k / selector_loss_weight 256 / 16 / 1.0 DFlash 2 selector. Loss = base loss + weight x selector loss
seq_length, num_epochs, lr 3072, 6, 6.0e-4 Training schedule in the example

Supported target architectures for DFlash 2 training: Qwen3ForCausalLM, Qwen3MoeForCausalLM, and the Qwen3.5 family (Qwen3_5*ForCausalLM, Qwen3_5*ForConditionalGeneration, which is what Qwen3.8-27B ships as). Kimi K3 is registered for DFlash (v1) training only. DFlash 2 logs train/base_accept_len, train/candidate_recall, and train/selector_loss in addition to the DFlash metrics.

Benchmarks

H200 (SGLang, Qwen3.8-27B, block size 8 = 7 draft tokens)

Source: the incoai/Qwen3.8-27B-DFlash2 model card. The GSM8K concurrency-1 figure (236.1 tok/s, 3.43x) is also in the SGLang PR #35371 description and the v0.5.19 release notes. FlashAttention 3 for target and draft. Model-default sampling (temperature 1.0, top-p 0.95, top-k 20), xhigh reasoning effort, 4096 max new tokens. Throughput = output tokens / wall time, speedup vs autoregressive:

Concurrency GSM8K MATH-500 HumanEval MBPP MT-Bench
1 236.1 tok/s (3.43x) 230.7 (3.34x) 214.6 (3.11x) 226.9 (3.29x) 184.0 (2.67x)
8 1,328.7 (2.84x) 1,368.3 (2.85x) 1,291.5 (2.67x) 1,328.0 (2.78x) 1,090.2 (2.27x)
32 1,922.5 (1.45x) 1,951.8 (1.30x) 1,799.0 (1.16x) 1,886.8 (1.25x) 1,525.3 (1.01x)

Native MTP on the same rig: 2.59x on GSM8K at concurrency 1, and below 1x on several tasks at concurrency 32 (0.77-0.94x). All speculative methods proposed 7 draft tokens per step in this comparison. SGLang v0.5.19 notes add one higher-load point: DFlash 2 is about 24% faster than DFlash at concurrency 64.

Acceptance Length (Inco AI blog)

Per-request mean acceptance length. Lossless rejection sampling, default sampling per model, block size 8 for Qwen3.8-27B and 16 for Muse Glimmer:

Target MTP DFlash DSpark DFlash 2
Qwen3.5-4B (mean of 5 tasks) 4.54 4.92 5.49 5.97
Qwen3.8-27B (vs native MTP) 4.28 — 3.62 4.80
Muse-Glimmer-30B (vs official DFlash) — 4.44 4.48 5.70

Module costs on a 5-layer Qwen3-4B-class drafter: path selector +2.0M params and +0.6% cycle latency. Convolutions +16.5M params (+3%) and +0.7% cycle latency. Both together +1.3%. DSpark-style correction heads: +77.8M params and +9.6%.

Engine PR Measurements

Source Rig Result
vLLM PR #52816 Qwen3.8-27B, GSM8K Acceptance 5.34 (DFlash 2) vs 4.27 (DSpark), +25.2%. 224.6 tok/s (3.51x) at concurrency 1. 2,759.4 tok/s (2.20x) at concurrency 32. Selector about 0.84% of step time at batch 1
SGLang PR #35371 Qwen3.8-27B 236.1 tok/s (3.43x) on GSM8K at concurrency 1. Convolution + selector cost 1.5-2.7% of step time, falling as batch size grows
llama.cpp PR #27342 Qwen3.8-27B Q4_K_M, Apple M5 Pro 18.89 tok/s decode, 1.81x, acceptance 5.03
ollama PR #17865 Qwen3.8-27B, Apple M5 Pro (MLX) About 2.45x vs autoregressive, 16.2% ahead of built-in MTP (author's sweep)
Splash README (vendor claim) Qwen3.8-27B 4-bit, 48 GB M5 Pro 2x the decode speed of the next-fastest engine. 282 ms first token with 32K context cached. Inco also claims 144 tok/s on an M5 Max

PR figures are the authors' own measurements, summarized from the PR descriptions.

NVIDIA Blackwell (TensorRT-LLM, DFlash v1 drafter)

gpt-oss-120b on 8x DGX B300 (Blackwell Ultra), SPEED-Bench coding: >15x higher throughput at 500-600 tok/s/user interactivity vs autoregressive, 1.5x higher than EAGLE-3 at the same point. Batch size 1 more than doubles interactivity. Interactivity speedups at matched concurrency: gpt-oss-120b, DFlash 2.3x avg vs EAGLE-3 1.7x. Llama-3.1-8B, DFlash 2.8x vs 2.2x. Single-GPU: Gemma-4-31B on vLLM/B300 up to 5.8x (MATH-500). Qwen3-8B on SGLang/B200 5.1x (MATH-500).

Google TPU v5p (JAX, vLLM tpu-inference, DFlash v1)

Average 3.13x tokens/s across datasets (peak about 6x on math. MATH-500 8.02 → 1.40 ms/token at K=16, greedy. MBPP 9.81 → 3.48 ms/token). Head-to-head with out-of-the-box Llama-3.1-8B checkpoints: DFlash 2.29x vs EAGLE-3 1.30x end-to-end serving speedup. Scaling: K=16 already captures >90% of the theoretical maximum. K from 16 to 128 adds less than one accepted token per step. Improving per-position acceptance is 2-3x more valuable than growing K.

SGLang Spec V2 and Qwen3.5-397B (LMSYS, DFlash v1)

  • Spec V2 overlap scheduler with DFlash: 11.4 → 15.3 ktok/s (>33%), Qwen3-8B on one B200 at concurrency 32.
  • Qwen3.5-397B-A17B (BF16) on 8x B200, HumanEval, greedy, thinking on: >4.3x the baseline throughput and 1.5x the MTP throughput at concurrency 1 (DFlash block size 16, MTP 7 steps). Higher throughput than native MTP at every tested concurrency from 1 to 32.

Ablations (LMSYS, Qwen3-4B, 5-layer drafters, acceptance / speedup)

Configuration GSM8K HumanEval MT-Bench
EAGLE-3 (5-layer) 4.2 / 2.1x 4.3 / 2.2x 3.1 / 1.4x
DFlash (both techniques) 4.2 / 3.3x 4.0 / 3.2x 3.0 / 2.2x
DFlash diffusion only 3.5 / 2.9x 3.5 / 2.9x 2.6 / 2.0x
DFlash injection only 4.8 / 2.4x 4.6 / 2.3x 3.4 / 1.5x

Production Data Points

  • CoreWeave's Kimi K2.7 Code endpoint pairs a custom NVFP4 quantization with a DFlash (v1) drafter on vLLM by default. It was the fastest provider for that model on Artificial Analysis at publication (281.9 output tok/s).
  • Xiaomi's MiMo v2.5-Pro-UltraSpeed uses DFlash to reach over 1,000 output tok/s (cited by LMSYS).
  • The Inco blog reported more than 3.5M Hugging Face downloads across DFlash models (Aug 2026).

Benchmark Caveats

  • The "15x" from NVIDIA is throughput at matched interactivity on an 8-GPU Pareto curve, not a batch-1 latency speedup. The 2.7-3.4x from the H200 model card describes a different operating point.
  • The two Qwen3.8-27B engine measurements disagree at concurrency 32: 1.16-1.45x on SGLang (model card) vs 2.20x on vLLM (PR #52816, GSM8K). Rigs and sampling settings differ. Benchmark your own stack.
  • Acceptance on Qwen3.8-27B differs by source: 4.80 (blog, default sampling) vs 5.34 (vLLM PR, GSM8K). The DSpark baseline also moves (3.62 vs 4.27).
  • The blog's +1.3% overhead is per draft-verify cycle on a 5-layer Qwen3-4B-class drafter. SGLang PR #35371 measures 1.5-2.7% of serving step time on Qwen3.8-27B.
  • Secondary summaries of the Google TPU post attribute the 3.13x average to Llama-3.1-8B-Instruct. The MATH-500 ms/token figures were recorded here as Qwen3-4B. Check the post before quoting either model.
  • Several numbers above come from vendor or author measurements. No independent third-party DFlash 2 benchmark was found as of 2026-09-25.

Sources