Skip to content

Operations

Context

Deployment recipes for every engine that runs DFlash 2 today (SGLang, vLLM, llama.cpp, ollama, oMLX, TensorRT-LLM), the dflash benchmark client, block-size tuning rules, drafter training via NVIDIA NeMo, and known operational pitfalls.

Deployment

pip install "sglang[all] @ git+https://github.com/sgl-project/sglang.git#subdirectory=python"

python -m sglang.launch_server \
  --model-path Qwen/Qwen3.8-27B \
  --speculative-algorithm DFLASH \
  --speculative-draft-model-path incoai/Qwen3.8-27B-DFlash2 \
  --speculative-num-draft-tokens 8

For large MoE targets (Qwen3.5-397B-A17B pattern from the LMSYS/Modal/Z Lab launch):

export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1

python -m sglang.launch_server \
  --model-path Qwen/Qwen3.5-397B-A17B \
  --trust-remote-code \
  --speculative-algorithm DFLASH \
  --speculative-draft-model-path modal-labs/Qwen3.5-397B-A17B-DFlash \
  --speculative-dflash-block-size 8 \
  --speculative-draft-attention-backend fa4 \
  --attention-backend trtllm_mha \
  --tp-size 8 \
  --max-running-requests 32 \
  --cuda-graph-max-bs-decode 32 \
  --mem-fraction-static 0.8

Migrating from EAGLE: change the speculative algorithm to DFLASH and point at a matching drafter — no application changes.

vLLM (via Speculators; install pinned to integration PR until merged)

pip install -U "vllm @ git+https://github.com/vllm-project/vllm.git@refs/pull/52816/head"

vllm serve Qwen/Qwen3.8-27B \
  --speculative-config '{
    "method": "dflash",
    "model": "incoai/Qwen3.8-27B-DFlash2",
    "num_speculative_tokens": 7
  }'

llama.cpp (local, single-request)

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
git fetch origin pull/27342/head:pr-27342
git switch pr-27342

# NVIDIA CUDA
cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=ON
cmake --build build -j

# Apple Silicon
cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_METAL=ON
cmake --build build -j

./build/bin/llama-server \
  -hf ggml-org/Qwen3.8-27B-GGUF:Q4_K_M \
  -hfd incoai/Qwen3.8-27B-DFlash2-GGUF:Q4_K_M \
  --spec-type draft-dflash \
  --spec-draft-n-max 7

ollama (build from DFlash 2 branch)

git clone https://github.com/ollama/ollama.git
cd ollama
git fetch origin pull/17865/head:dflash2
git switch dflash2

cmake -B build .
cmake --build build --parallel 8

TARGET="$(hf download mlx-community/Qwen3.8-27B-4bit)"
DRAFT="$(hf download incoai/Qwen3.8-27B-DFlash2)"
printf "FROM %s\nDRAFT %s\n" "$TARGET" "$DRAFT" > Modelfile

./ollama create qwen38-dflash2 --experimental --draft-quantize int4
./ollama serve
./ollama run qwen38-dflash2 --think high

oMLX (Apple Silicon GUI server)

  1. Install the prebuilt build: oMLX 0.6.2-zlab-dflash2 (arm64, signed).
  2. In the Model Downloader (127.0.0.1:8891), download mlx-community/Qwen3.8-27B-4bit and incoai/Qwen3.8-27B-DFlash2.
  3. Edit the target model in the Model Manager: enable DFlash, set draft model to incoai/Qwen3.8-27B-DFlash2, enable draft quantization, runtime block size 5, verify mode dflash.
  4. Load the target model and serve.

TensorRT-LLM

Supported on NVIDIA Blackwell (NVIDIA published the gpt-oss-120b 8xB300 Pareto results with it). Per-engine launch flags are not publicly documented yet — TBD, track the NVIDIA post and the TensorRT-LLM repo.

Commands & Recipes: the dflash Client

The repo's CLI is a lightweight client/benchmark harness — all speculation happens server-side. Requires Python 3.10+.

pip install dflash            # client only (talks to a running server)
pip install "dflash[local]"   # + local inference (MLX on Apple Silicon, Transformers on Linux)

Generate with the Transformers backend (DFlash 2 on Muse-Glimmer-30B):

dflash generate transformers \
    --model meta-models/Muse-Glimmer-30B \
    --draft z-lab/Muse-Glimmer-30B-DFlash2 \
    --reasoning high --temperature 1 --top-p 0.95 --top-k 64 \
    "How many positive whole-number divisors does 196 have?"

Generate locally on Apple Silicon (MLX, both models 4-bit — note block size 5, see tuning below):

dflash generate mlx \
    --model mlx-community/Qwen3.8-27B-4bit \
    --draft z-lab/Qwen3.8-27B-DFlash2 \
    --draft-bits 4 --block-size 5 --reasoning xhigh \
    "How many positive whole-number divisors does 196 have?"

Generate against any OpenAI-compatible SGLang/vLLM server:

dflash generate openai \
    --base-url http://127.0.0.1:8000 --model Qwen/Qwen3.8-27B \
    "How many positive whole-number divisors does 196 have?"

Benchmark (datasets: gsm8k, math500, humaneval, mbpp, mt-bench; cached via HF Datasets):

dflash benchmark openai \
    --base-url http://127.0.0.1:8000 --model Qwen/Qwen3.8-27B \
    --dataset gsm8k --num-prompts 128 --concurrency 1 --reasoning xhigh \
    --temperature 1 --top-p 0.95 --top-k 20

dflash benchmark mlx \
    --model mlx-community/Qwen3.8-27B-4bit --draft z-lab/Qwen3.8-27B-DFlash2 \
    --dataset gsm8k --max-samples 128 --reasoning xhigh --block-size 5 --draft-bits 4

Performance Tuning

Knob Guidance
Block size 8 is the Qwen3.8-27B default (--speculative-num-draft-tokens 8 / num_speculative_tokens: 7 draft tokens); Muse Glimmer launch evals used 16; TPU measurements put the sweet spot at K=16 (>90% of theoretical max; 16→128 adds <1 token/step)
MLX quantized Use block size <= 5 — MLX's quantized matmul kernel loses efficiency at larger verify widths (both target and draft 4-bit)
Draft attention backend --speculative-draft-attention-backend fa4 on SGLang for the drafter's non-causal block attention
Spec engine Prefer Spec V2 (overlap scheduler): >33% throughput gain over the V1 path at concurrency 32 (SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 in current releases)
Reasoning effort Qwen3.8: reasoning_effort low/medium/xhigh (default xhigh); Muse: reasoning_strength low/medium/high (default high) — higher effort raises acceptance by making output more predictable
Concurrency expectations Plan for ~3x at batch 1 compressing toward ~1x at concurrency 32 (full table: Architecture — Benchmarks); don't pay speculation overhead for pure-throughput batch workloads
Checkpoint pairing The drafter must match the target exactly (incoai/Qwen3.8-27B-DFlash2 for Qwen/Qwen3.8-27B); a mismatched pair degrades or fails — drafters are per-target, not portable

Training a Drafter for Your Own Model

No official training code ships in z-lab/dflash (Issue #1). The documented path is NVIDIA's NeMo AutoModel recipe, which trains DFlash, DFlash 2, Domino, and JetSpec variants with the same scaffolding (frozen target, online hidden-state capture, DDP across GPUs):

torchrun --standalone --nproc_per_node=2 \
  -m nemo_automodel.recipes.llm.train_dflash2 \
  -c examples/speculative/dflash/qwen3_dflash2.yaml

Data rules that matter (from the recipe docs):

  • Use chat-format prompts with responses regenerated by the target model — training is teacher-forced, so regenerating first avoids a train/inference distribution mismatch.
  • mask_token_id is required and must be a reserved, rarely-used token. Never reuse pad (often aliased to eos) — it conflates the mask signal with content and quietly erodes acceptance. The inference runtime must fill block slots with the same id.
  • Objective: cross-entropy on block positions 1..K-1 weighted by position decay w_k = exp(-(k-1)/loss_decay_gamma); paper values track block size (gamma 7 for K=16, 5 for 10, 4 for 8). DFlash 2 adds a selector loss over top-k candidates weighted by selector_loss_weight.

DFlash 2-specific config fields: conv_kernel_size / conv_group_size (convolution taps and channels-per-correction), selector_rank / selector_top_k / selector_loss_weight (selector width, candidates per position, objective weight). Published drafters also diverge from their target via draft_sliding_window (1024) and a different attention shape (draft_num_attention_heads and friends). Supported targets: Qwen3/Qwen3.5/Qwen3.8 dense and MoE, incl. multimodal variants. Training logs train/accept_len — watch it, not just loss.

Troubleshooting

  • CUDA graph crash under load — Issue #146 reports DFlash speculation crashing CUDA graph capture/execution under load; a correctness-class bug. Keep an eye on the issue tracker before production rollout; pin to a known-good engine build.
  • No speedup at high concurrency — expected behavior, not a bug; see Concurrency expectations above.
  • llama.cpp concurrency — speculative decoding in llama.cpp prevents concurrency scaling; treat that path as single-user local inference only.
  • vLLM from a PR ref — the documented install pins refs/pull/52816/head; re-pin deliberately when the PR merges. Same for SGLang (PR #35371 reference), llama.cpp (PR #27342), ollama (PR #17865).
  • Acceptance length regression after retraining — check the mask_token_id aliasing rule and that responses were regenerated by the frozen target before training.

Cost Analysis

DFlash 2 decodes at close to 3x the speed of autoregressive decoding at roughly one third of the compute per token with identical output. Concretely (H200, Qwen3.8-27B, batch 1): ~69 tok/s autoregressive vs ~236 tok/s with DFlash 2 on GSM8K — the same GPU serves ~3.4x more interactive users at the same latency, or cuts per-token cost to roughly a third. The drafter itself adds ~2B parameters (BF16) of memory plus the ~1.3% cycle overhead.

Sources