The headline in NVIDIA's post is 185.1%. The number worth keeping is 42.5%. Both come from the same three optimizations applied to Ling-3.0-flash on a single DGX Spark, and the gap between them is the most instructive part of the write-up: a synthetic 32K-in, 2K-out workload with greedy decoding lifts decode throughput from 43.94 to 102.88 tokens per second, while all 1,319 GSM8K questions at temperature 0.6 lift it from 37.81 to 53.89. Synthetic benchmarks are known to flatter speculative decoding; the useful question is by how much, and why.
The setup is a 124-billion-parameter mixture-of-experts model from Inclusion AI with 5.1 billion parameters active per token: 42 layers, 35 of them Kimi Delta Attention linear-attention layers and seven MLA layers, with each MoE layer routing to eight of 512 routed experts plus one shared expert. Decode ran at batch size 1, where no batching amortizes a weight read and the arithmetic intensity of each step sits near one multiply-add per byte moved. Profiling put the time in two places: the routed-expert computation and the output-layer projection.
The second is easy to reason about in isolation. The LM head projects a 2,560-wide hidden state to a 157,184-token vocabulary, roughly 402 million weights, stored in BF16 because the checkpoint lists it in modules_to_not_convert. Every generated token reads all of it: about 805 MB per token for one matrix-vector product. FP8 halves that to roughly 400 MB and bought 11.4% on top of the kernel change. It is a bookkeeping win other deployments are likely leaving on the table, disabled by default because the cookbook baseline turns on --enable-fp32-lm-head.
The kernel change is more interesting as practice. Humming replaces FlashInfer's CUTLASS MoE path and uses NVRTC to compile CUBINs at runtime for the actual M/N/K dimensions and dtype, caching them in ~/.humming/cache. The model's moe_intermediate_size is 768, narrow enough that kernels tuned for larger tiles waste work; shape-specialized codegen recovers 9.3%, taking time per output token from 27.71 ms to 25.35 ms. Compile time improves too, seconds against tens of minutes for the 97-target CUTLASS build, which moves a build step into production startup: a cache keyed on architecture, dtype and shape is now part of the deployment contract.
Speculative decoding is where the architecture starts to matter. Ling-3.0-flash ships a NEXTN prediction layer, so the draft costs no extra training and no extra download. The configuration runs three draft steps against a four-token verification window. NVIDIA measured a verification-cost ratio of about 2.26 and attributes it not to attention but to expert activation: a normal decode step reads eight experts for one token, while a verification pass over four positions can touch up to 32. In a dense model, verifying four tokens costs barely more than decoding one, because the extra tokens ride along in the same batched GEMM. In a sparse one, verification widens the union of activated experts, so each position adds weight traffic instead of reusing it. Speculation at batch 1 with 512 experts buys fewer rounds at a higher price per round, and that price scales with top-k.
The ratio is also why the two acceptance numbers matter. On the synthetic workload the draft landed 3.90 accepted tokens per round; on GSM8K, 2.68. The post names three reasons, all fair: synthetic outputs are repetitive, so drafts are easy to accept; the synthetic run used greedy decoding where the real run used temperature 0.6, top_p 0.95 and top_k 29; and the synthetic run fixed 2,048 output tokens while the GSM8K median is about 310. The third is underrated. Kernel and quantization wins are per-step savings, so they amortize over output length; at a median of 310 tokens there is less room for them to accumulate.
The accuracy line deserves more scrutiny than it usually gets. GSM8K moved from 96.89% to 96.82%, under a tenth of a point, about one question in 1,319. The post says accuracy was essentially flat, and by its own data that is true. But a benchmark at 96.9% has almost no headroom to detect what an FP8 vocabulary projection could damage: the failure mode is rare tokens, long-tail vocabulary and calibration-sensitive sampling, none of which a near-saturated math set will localize. Top-k agreement against the BF16 head would cost minutes and answer the question the accuracy column only appears to answer.
Two reporting gaps are worth raising as requests. The first is bandwidth. If each decode step streams 2.5 to 3 GB of weights, that being 5.1 billion active parameters at roughly 4 bits plus the BF16 head, then 27.71 ms per token implies an effective weight-traffic rate near 100 to 130 GB/s. Whether that is close to the platform's ceiling is not stated, and without it a reader cannot tell whether the 9.3% kernel gain closed an efficiency gap or merely narrowed it. The second is the latency distribution: P50 of 6.11 s against a mean of 11.49 s and P95 of 38.60 s. The mean is a tail statistic in disguise, and the tail is what an agentic caller with a timeout experiences. Mean output length also grew from 567 to 607 tokens, unexplained; if the optimized run generated more, part of the comparison is normalization rather than speed.
The paragraph to read twice is the integration warning. Enabling FP8 head quantization and multi-token prediction together requires the draft to share the entire LM head module with the target, not just its weight tensor. Ling's original NEXTN implementation did not expose that interface, and the failure is silent: the draft receives FP8 weights, interprets them as BF16, accept length falls to 1.00, and the fully optimized configuration runs slower than the unoptimized baseline. Shapes match, nothing raises, the logits are merely wrong, and every round pays verification for a draft that never lands a token. This is the general hazard of composing optimizations that each own part of the same module: quantization state is metadata that has to travel with the object, and a tensor-level handoff drops it. A single dtype-mismatch assertion at startup would have caught it in one second.
What the post establishes is narrow and solid: on this model, this shape, this device, at batch size 1, three non-overlapping mechanisms produce a real 42.5% decode gain with no measurable accuracy change. What it does not establish is transfer. Raise the batch and decode becomes compute-bound, LM-head traffic stops being the tax, and speculation's cost model shifts again. No power or energy per token is reported, which is the axis that decides whether an edge deployment is viable. And 185.1% is a ceiling on a friendly workload, not a promise.
The transferable method is the one to copy: isolate each change on a synthetic workload to see the mechanism, confirm the stack on a full evaluation set to see the deployment number, report acceptance length and verification-cost ratio so readers can price the trade, and assert configuration state at startup rather than inferring it from a throughput curve. The distance between 185% and 42.5% is not a disappointment. It is the measurement doing its job.
Sources: NVIDIA Technical Blog, "3 Ways to Boost MoE Inference on NVIDIA DGX Spark", https://developer.nvidia.com/blog/3-ways-to-boost-moe-inference-on-nvidia-dgx-spark/