Skip to content

Architecture

Context

How DFlash 2 works: the one-pass block-diffusion draft inherited from DFlash (Z Lab, ICML 2026), the KV-injection conditioning that keeps a 2B-parameter drafter accurate, and the two DFlash 2 additions — the candidate path selector and the two-tap dynamic convolution — each fixing one measured loss mode of parallel drafting.

Speculative Decoding Context

A speculation cycle has two phases. A small drafter proposes a block of future tokens. The frozen target LLM verifies the whole block in one forward pass and accepts the longest valid prefix (speculative decoding fundamentals cover the mechanism). Rejection sampling makes the scheme lossless. Speedup reduces to acceptance length x (verification cost / (draft cost + verification cost)). The two questions: how many tokens survive each cycle, and how cheap the guessing was.

Autoregressive drafters (EAGLE-3, native MTP) need K sequential forward passes to propose K tokens, so their draft cost grows linearly with block size and they run shallow drafters. The TPU v5p measurements from UCSD added the hardware punchline, K-flat verification. On datacenter accelerators, verifying 1024 drafted tokens costs nearly the same as verifying 16. Weight loading dominates, not attention math. Draft quality, not width, is the binding constraint — which is exactly the constraint DFlash 2 attacks.

Component Breakdown

Component Role Key Facts
Target LLM (frozen) Verifier. Also the context expert For example, Qwen3.8-27B, Muse-Glimmer-30B. Shares embed_tokens and lm_head with the drafter
DFlash drafter backbone One-pass block denoiser ~2B params (Qwen3.8-27B drafter), 5 layers, sliding-window attention (1024-token window), own attention shape independent of target
Block attention mask Non-causal in-block attention Position 0 holds the real anchor token. Positions 1..K-1 start as MASK and are denoised in parallel
KV injection path Conditioning on target context Target hidden states pass through the KV projection of the drafter into the KV cache of every draft layer — not just the input embedding (the EAGLE-3 way)
Candidate path selector (DFlash 2) Coherence without autoregression — fixes incoherent independent picks (detail below) Top-16 candidates per position, low-rank bilinear pair scoring. +2.0M params, +0.6% cycle latency
Two-tap dynamic convolution (DFlash 2) Suffix-decay fix (detail below) Two taps around every attention and MLP sublayer, mixing each position with its predecessor. +16.5M params (+3%), +0.7% cycle latency
DFlashWorker / DFlashDraftModel SGLang runtime Draft worker drives the scheduler and wraps the target worker for verification passes (PR #22077, then Spec V2 in PR #23000)
Speculators library vLLM runtime Connects the drafter to target hidden states inside the vLLM path. Swap EAGLE-3 for DFlash by config only
TPU proposer (tpu-inference) JAX runtime Dual-cache: target keeps paged KV (Pallas kernels), drafter uses static on-device JAX arrays (PRs #1868-1870)

Draft-Verify Cycle

flowchart TB
    subgraph CTX["Context Capture (prefill / last verify)"]
        T0["Target LLM layers<br/>(frozen weights)"]
        HP["Target hidden states<br/>of context tokens"]
        T0 --> HP
    end

    subgraph DRAFT["Drafter — single forward pass"]
        KV["KV injection:<br/>draft KV projection of hidden states<br/>into every draft layer"]
        BB["5-layer block-diffusion backbone<br/>sliding window 1024, non-causal block mask"]
        CONV["Two-tap dynamic convolution<br/>before/after each attn + MLP sublayer"]
        HEAD["Shared frozen lm_head<br/>logits + top-16 candidates per position"]
        KV --> BB --> CONV --> HEAD
    end

    subgraph SEL["Path Selection (sequential, tiny)"]
        SC["Pairwise scorer<br/>U_t(b) + bilinear match A(a), B(b)<br/>gated by context H(h_t)"]
        WALK["Greedy walk or sampling<br/>from last verified token"]
        SC --> WALK
    end

    subgraph VER["Verification"]
        TV["Target LLM forward pass<br/>over draft block + next token"]
        RS["Rejection sampling<br/>(lossless, target distribution)"]
        TV --> RS
    end

    HP --> KV
    HEAD --> SC
    WALK -->|"draft block (K tokens)"| TV
    RS -->|"accepted prefix + bonus token"| CTX
    RS -->|"committed output"| OUT["Detokenizer / client"]

One Cycle, Step by Step

sequenceDiagram
    autonumber
    participant S as SGLang Scheduler
    participant DW as DFlashWorker
    participant DR as DFlash Drafter (2B)
    participant PS as Path Selector (+2M)
    participant T as Target LLM (frozen)

    S->>DW: forward_batch_generation(batch)
    DW->>DR: inject target hidden states (KV projection)
    DR-->>DW: whole block predicted in ONE pass<br/>top-16 candidates per position
    DW->>PS: candidate lists + last verified token
    PS-->>DW: one coherent path (K tokens)
    DW->>T: verify draft block in parallel
    T-->>DW: target logits for every position
    DW->>DW: rejection sampling (lossless)
    DW-->>S: accepted prefix + 1 bonus token

The counterintuitive SGLang detail (useful when reading traces): the draft worker is the one that talks to the scheduler — it wraps and calls the target worker when drafts are ready, not the other way around.

How It Works: The Two DFlash 2 Fixes

Problem 1 — Independent picks are incoherent (selection headroom)

Predicting every position independently means each pick is locally plausible but jointly incoherent — two neighbors can both pick "decoding", and the stutter dies at verification. The evidence that this is a selection problem, not a prediction problem, sits in the candidate lists of DFlash itself. The data below uses the 5-layer Qwen3-4B drafter on GSM8K, conditioned on all earlier positions being right:

Metric Pos 0 1 2 3 4 5 6 Acceptance length
Recall@1 85.4% 80.3% 79.4% 78.3% 77.5% 75.9% 72.9% 4.27
Recall@16 99.5% 97.3% 94.8% 92.6% 90.8% 89.4% 87.8% 6.79

An oracle picking from the top-16 reaches 6.79 vs 4.27 — 2.5 tokens of pure selection headroom. The path selector of DFlash 2 harvests part of it: keep the top-16 candidates per position and score every adjacent pair,

S_t(a, b) = U_t(b) + < A(a) ⊙ H(h_t), B(b) >

where U_t(b) is the logit of the drafter for candidate b, A and B are compact 256-dimensional token embeddings, and H(h_t) is a context gate deciding which parts of the predecessor-match count — in essence a low-rank bilinear attention over adjacent candidates. All pairs score in one parallel shot (no extra backbone or LM-head pass). Only the final walk over precomputed scores is sequential. Greedy follows the best successor. Sampling draws from the same scores. Rejection sampling restores the exact target distribution.

Results vs the DSpark-style sequential correction (5-layer Qwen3-4B, GSM8K. Overheads relative to plain DFlash):

Method Added params Latency Acceptance @ T=0 @ T=1
DFlash — — 4.27 3.78
+ DSpark correction +77.8M +9.6% 4.49 4.08
+ path selector (DFlash 2) +2.0M +0.6% 4.61 4.25

Roughly 40x fewer parameters and 16x lower latency overhead than the correction-head approach, with higher acceptance. Choosing is cheaper than predicting.

Problem 2 — Suffix decay is local (backbone, not selection)

Even the oracle decays down the block (99.5% → 87.8% across positions in the recall table): the candidates themselves run out. Depth fixes it indiscriminately. 3-, 5-, and 15-layer drafters are identical at position 0 and fan apart down the block. But ten extra layers cost +15.2% cycle latency and erase the efficiency edge of DFlash.

The attention of DFlash shows where the capacity is needed. It has two jobs — read the context before the block, and model dependencies inside it — but the attention share of the block falls from 30% in layer 1 to 8% in layer 5, with the remainder concentrating in a shrinking handful of heads. DFlash 2 splits the jobs: a dedicated module takes the within-block work. The design puts a two-tap dynamic depthwise convolution before and after each attention and feed-forward sublayer:

Conv_k(x)_t = k_t,0 ⊙ x_t + k_t,1 ⊙ x_{t-1}

Each coefficient combines a learned base kernel with a small correction computed from the current hidden state (one correction shared per 16 channels). The first position reads the representation of the last verified token. Every later position reads its predecessor. Information crosses the block while all positions still compute in parallel — and the module is block-local and stateless, so attention, the LM head, and verification are untouched.

Outcome: the convolutions buy what ten extra layers bought (overheads in the tables in this section) — five-layer DFlash + convolutions nearly matches 15-layer DFlash on suffix decay. Average within-block attention across layers 4-5 falls from 9.4% to 0.5%: the module has absorbed the local work, and attention returns to reading context. Suffix decay is mostly a local problem.

Combined System

Selector + convolutions together add 1.3% to the draft-verify cycle latency. Per-request mean acceptance length (lossless rejection sampling. Default sampling per model, block size 8 for Qwen3.8-27B, 16 for Muse Glimmer):

Target MTP DFlash DSpark DFlash 2
Qwen3.5-4B (mean of 5 tasks) 4.54 4.92 5.49 5.97
Qwen3.8-27B (vs native MTP) 4.28 — 3.62 4.80
Muse-Glimmer-30B (vs official DFlash) — 4.44 4.48 5.70

On Qwen3.5-4B that is +1.05 tokens per pass over DFlash (+21%) and +0.48 over DSpark. On both launch models DFlash 2 averages more than a full token ahead of DSpark. On MATH-500 the gain is visible position by position: DFlash 2 holds ~86% conditional acceptance to the last position while every baseline ends the block 6-9 points lower.

Internals Worth Knowing (Serving Path)

  • Immediate materialization of the draft KV projection. Target latents are projected by the drafter ahead of the draft forward pass rather than stored. This preserves KV space and radix-cache prefix sharing. SGLang implements this with a layer-batched linear projection plus a fused Triton kernel for norm+RoPE post-processing.
  • Overlap scheduling (SGLang Spec V2). Host-side cleanup of batch N-1 and KV allocation for batch N overlap with GPU work. This cuts host-device synchronization. Combining DFlash with Spec V2 improved throughput >33% (11.4 → 15.3 ktok/s, Qwen3-8B on B200 at concurrency 32).
  • TPU dual-cache. The non-causal block diffusion of DFlash is incompatible with paged attention, so the tpu-inference port runs the target on paged KV (Pallas kernels) and the drafter on static on-device JAX arrays. A metadata rework fixed "sequence length inflation", where draft state drifted from the accepted-token count of the target.

Benchmarks

H200 (SGLang, Qwen3.8-27B, block size 8 = 7 draft tokens)

FlashAttention 3 for target and draft. Model-default sampling (temperature 1.0, top-p 0.95, top-k 20), xhigh reasoning effort, 4096 max new tokens. Throughput = output tokens / wall time, speedup vs autoregressive:

Concurrency GSM8K MATH-500 HumanEval MBPP MT-Bench
1 236.1 tok/s (3.43x) 230.7 (3.34x) 214.6 (3.11x) 226.9 (3.29x) 184.0 (2.67x)
8 1,328.7 (2.84x) 1,368.3 (2.85x) 1,291.5 (2.67x) 1,328.0 (2.78x) 1,090.2 (2.27x)
32 1,922.5 (1.45x) 1,951.8 (1.30x) 1,799.0 (1.16x) 1,886.8 (1.25x) 1,525.3 (1.01x)

Honest scaling picture: native MTP at concurrency 32 falls below 1x on several tasks (0.77-0.94x). DFlash 2 degrades most gracefully but still compresses toward parity. All speculative methods proposed 7 draft tokens per step in this comparison.

NVIDIA Blackwell (TensorRT-LLM, DFlash v1 drafter)

gpt-oss-120b on 8x DGX B300 (Blackwell Ultra), SPEED-Bench: >15x higher throughput at 500-600 tok/s/user interactivity vs autoregressive, 1.5x higher than EAGLE-3 at the same point. Batch size 1 more than doubles interactivity. Interactivity speedups at matched concurrency — gpt-oss-120b: DFlash 2.3x avg vs EAGLE-3 1.7x. Llama-3.1-8B: DFlash 2.8x vs 2.2x. Single-GPU: Gemma-4-31B on vLLM/B300 up to 5.8x (MATH-500). Qwen3-8B on SGLang/B200 5.1x (MATH-500).

Google TPU v5p (JAX, vLLM tpu-inference)

Average 3.13x tokens/s across datasets (peak ~6x on math. MATH-500 8.02 → 1.40 ms/token, Qwen3-4B, K=16, greedy). Head-to-head, Llama-3.1-8B out-of-the-box checkpoints: DFlash 2.29x vs EAGLE-3 1.30x end-to-end serving speedup. Scaling theory: K=16 already captures >90% of the theoretical maximum. K from 16 to 128 adds less than one accepted token per step. Improving per-position acceptance is 2-3x more valuable than growing K.

Ablations (LMSYS, Qwen3-4B-class drafters, acceptance / speedup)

Configuration GSM8K HumanEval MT-Bench
EAGLE-3 (5-layer) 4.2 / 2.1x 4.3 / 2.2x 3.1 / 1.4x
DFlash (both techniques) 4.2 / 3.3x 4.0 / 3.2x 3.0 / 2.2x
DFlash diffusion only 3.5 / 2.9x 3.5 / 2.9x 2.6 / 2.0x
DFlash injection only 4.8 / 2.4x 4.6 / 2.3x 3.4 / 1.5x

Diffusion drafting buys speed at equal acceptance. KV injection buys acceptance that survives draft depth. Together they dominate.

Source Discrepancies

  • The "15x" from NVIDIA is throughput at matched interactivity on an 8-GPU Pareto curve, not a batch-1 latency speedup. The 2.7-3.4x from the blog (H200, batch 1) and the number from NVIDIA describe different operating points. Provenance caveats (release-date and attribution discrepancies) are annotated in the index Sources.

Sources

Security

Context

DFlash 2 is a decoding-time optimization, not a network service — its security surface is different from that of an inference engine. The angles that matter: losslessness as an output-integrity property, the drafter-checkpoint supply chain, privacy characteristics of the draft/verify data flow, and the (small) set of ways the optimization itself can go wrong operationally.

Output Integrity: Losslessness as a Guarantee

The core security-relevant property of DFlash 2 is that it is provably output-preserving:

  • Greedy decoding produces token-for-token the same output as the target model alone.
  • Sampled decoding draws from the exact distribution of the target model via rejection sampling over draft proposals.
  • The selector and convolutions only change which candidates get proposed — verification and rejection sampling are untouched, so the committed distribution is unchanged regardless of drafter quality.

This matters for regulated or audit-sensitive deployments: adding DFlash 2 does not change model behavior, alignment characteristics, or output policy. An output audit run on autoregressive decoding remains valid under DFlash 2.

What can break the guarantee in practice is runtime correctness, not the algorithm:

  • z-lab/dflash Issue #146 reports CUDA-graph crashes under load — a correctness-class bug. A crash is visible, but any engine bug that silently mis-schedules speculation (for example, wrong accepted-prefix accounting) can corrupt output.
  • The "sequence length inflation" bug of the TPU port (draft state drifting from the accepted-token count of the target) is the canonical example of this bug class: fixed by synchronizing the proposer strictly with the true accepted token count.
  • Operational rule: diff-test after every engine or drafter upgrade (see Monitoring and Audit Hooks) — losslessness makes the check exact.

Supply Chain: Drafter Checkpoints and Build Provenance

DFlash 2 adds a second artifact to the model supply chain — the drafter — loaded alongside the target model into the serving process with code-execution potential (custom architecture code, trust-remote-code in SGLang configs).

Artifact Source Risk Posture Mitigation
Official drafters incoai/* and z-lab/* on Hugging Face (mirrored pairs, for example Qwen3.8-27B-DFlash2) Moderate — new org, fast-moving project Pin exact revisions. Prefer the z-lab/ mirror when paring incoai vs z-lab. Scan safetensors before load
Community drafters for example RadixArk/Qwen3.8-27B-DSpark, DaoCloud/Muse-Glimmer-30B-DSpark, vendor drafter orgs (nvidia, RedHatAI, modal-labs, XiaomiMiMo, poolside, meta-models) Higher — third-party training provenance unknown Treat as untrusted model code. Review model cards for training data claims. Quarantine in a staging server first
GGUF drafts incoai/Qwen3.8-27B-DFlash2-GGUF (llama.cpp path) Moderate — conversion adds a transform step Verify conversion provenance. Compare greedy output against the safetensors path once
Engine builds vLLM from PR head ref (#52816), llama.cpp PR #27342, ollama branch PR #17865, oMLX signed dmg High while pinned to PR refs — code not yet through mainline review Pin exact commit SHAs. Move to merged/mainline releases as soon as available. Verify the oMLX signature
dflash pip package PyPI (dflash, MIT) Low — client/benchmark harness only Standard pip hash-pinning practice

Additional notes:

  • Drafter-target mismatch is a quality/availability issue, not an integrity one — a wrong drafter degrades speed and can crash the server, but lossless verification still filters any bad tokens. Do not rely on that as a safety net for code-level compromise, which verification cannot catch.
  • No known CVEs or published compromises of DFlash artifacts as of 2026-08-28 — this reflects the age of the project (v1 Feb 2026), not audit depth. TBD — re-check the issue tracker and advisories before production use.

Privacy and Data Flow

  • All computation stays in the serving process. Drafting, selection, and verification operate on the same prompts and KV state as ordinary decoding. No DFlash-specific data leaves the server. The dflash CLI documents no telemetry (its only outbound channel is a feedback form) — verify against the installed version if this matters to you.
  • No persistent cross-request state. Draft-side KV state is transient per cycle (Architecture — Internals). Nothing drafter-side persists beyond the request. Standard engine KV-cache isolation and prefix-sharing rules still govern cross-request leakage — DFlash does not weaken them by design, but engine-level cache-isolation bugs can now span both models.
  • The API perimeter is unchanged. Clients talk to the standard OpenAI-compatible endpoint. Apply the usual controls: API-key auth, TLS, network segmentation of the GPU serving VLAN, per-tenant rate limits.
  • Prompt content shapes acceptance, not exposure. Higher task predictability (math/code) raises acceptance. Nothing about the drafter makes prompt data more observable to other tenants.

Access Control and Hardening

  • Model-loading is the privileged operation. Whoever can point a SGLang/vLLM server at a drafter repo executes new model code with trust-remote-code. Restrict server launch and model-manager access (relevant for GUI servers like oMLX on shared machines — its admin dashboard binds to 127.0.0.1:8891 by default, keep it that way on multi-user hosts).
  • Resource controls bound speculation overhead. mem-fraction-static, max-running-requests, and cuda-graph-max-bs-decode cap the memory and scheduling footprint of the drafter. The drafter adds ~2B params (BF16) of resident memory.
  • Denial of budget, not denial of service. At high concurrency, speculation overhead compresses gains toward 1x — a misconfigured rollout can silently waste drafter memory and cycle time. Watch acceptance length (see Monitoring and Audit Hooks).

Threat Model Summary

Threat Vector Impact Likelihood Control
Malicious drafter checkpoint HF repo / mirror swap Code execution in server process Low today, grows with ecosystem Pin revisions, scan artifacts, staging server, prefer official mirrors
Tampered engine build PR-ref installs Arbitrary code execution Low Pin SHAs, move to mainline releases, verify signatures (oMLX dmg)
Silent output corruption Engine speculation bug (for example, #146-class, TPU seq-inflation-class) Integrity loss on committed tokens Low Greedy diff-testing vs autoregressive mode. Track the issue tracker. Acceptance-length monitoring
Cross-request data leakage Engine KV/radix-cache bug spanning target and draft KV Confidentiality breach Low (design preserves isolation) Standard engine isolation updates. Keep the engine patched
Budget waste (spec overhead) High-concurrency rollout without benchmarking Cost/throughput regression Medium Concurrency-tier benchmarking before rollout. Disable speculation for throughput-shaped traffic
Model unavailability Checkpoint-locked ecosystem. No training code for private fine-tunes Cannot accelerate custom models Certain today (Issue #1) NeMo training recipe. Z Lab/Modal engagement for custom drafters
Mask-token aliasing (training) Reusing pad/eos as mask_token_id in a self-trained drafter Quiet acceptance-length erosion (quality, not integrity) Medium Reserve a dedicated rarely-used token. Match the id at inference. Monitor train/accept_len

Monitoring and Audit Hooks

  • Track acceptance length per request — the natural health metric (mean committed tokens per verification step). Sudden drops indicate a wrong block size, a mismatched drafter, quantization drift, or a runtime bug. The dflash benchmark harness reports it directly.
  • Keep a lossless diff in the release checklist — run a fixed prompt set greedily through autoregressive and DFlash modes after every engine/drafter change. Outputs must be byte-identical, which makes output-integrity regression testing exact rather than statistical.
  • Record artifact provenance at deploy time — log the exact HF revision SHAs of target and drafter, the engine build (PR ref or release), and the oMLX/pkg versions alongside service version so any supply-chain incident maps to a precise artifact set.
  • Watch the upstream issue tracker — correctness-class reports (for example, CUDA-graph crashes) land as GitHub issues first. There is no separate security-advisory channel for the drafter ecosystem as of 2026-08-28.

Sources