NVIDIA published on October 27 a batch-size-1 study of Ling-3.0-flash, a 124-billion-parameter mixture-of-experts model from Inclusion AI with 5.1 billion active parameters per token, running on a single DGX Spark. Profiling showed decoding was memory-bound, with the MoE computation and the output-layer projection dominating execution time.
The first change swaps FlashInfer's MXFP4 MoE backend for Humming kernels, which compile CUBINs at runtime for the actual matrix shapes and cache them. Time per output token fell from 27.71 ms to 25.35 ms, a 9.3% throughput gain. The second quantizes the language-model head online to FP8, halving that layer's weight reads and lifting cumulative throughput 21.7% over baseline. The third uses the model's built-in NEXTN layer for multi-token prediction with three draft steps and a four-token verification window, reaching 3.90 accepted tokens per round on a synthetic 32K-input, 2K-output workload.
NVIDIA is explicit that the synthetic result overstates the gain. Across all 1,319 GSM8K questions the three changes together raised decode throughput from 37.81 to 53.89 tokens per second, a 42.5% gain, cut mean latency 24.5% and left accuracy essentially flat at 96.89% to 96.82%, with accept length falling to 2.68. It also documents an integration trap: enabling FP8 head quantization and multi-token prediction together requires sharing the full output-layer module, otherwise the draft silently reads FP8 weights as BF16, accept length drops to 1.00 and the combined setup runs slower than baseline.
Our takeThe most valuable part is the negative result. A 185% synthetic gain shrinking to 42.5% on GSM8K, with accept length dropping from 3.90 to 2.68, is what speculative decoding looks like outside a benchmark harness. The fixed overhead saved by the first two optimizations also matters less when median outputs are 310 tokens. The silent FP8/BF16 fallback is the other lesson: a configuration that fails by getting slower, without an error, is what tooling should catch.