NVIDIA published a tutorial on 30 October showing how to run a language model on the DGX Spark, which is now sold in a 64GB configuration, using TensorRT Edge-LLM 0.11.1.
TensorRT Edge-LLM is a lightweight C++ inference runtime for embedded workloads. NVIDIA says the same runtime is used for development on Spark and for deployment on Jetson Thor, DRIVE AGX Thor and IGX Thor, so a prototype does not have to be rebuilt on a different stack when it moves to a device.
The trtedge command-line interface downloads the chosen model and builds TensorRT engines optimised for inference on Spark. In the walkthrough the model is Nemotron-3.5 Lightning, a text-only model; developers test prompts in a terminal, then serve the model through an API for application prototyping, and can enable DSpark speculative decoding to accelerate generation.
The prerequisites listed are a supported Spark software stack, CUDA 13, TensorRT 10.16.1.11 or newer within the 10.x series and Python 3.10 to 3.12. The examples were tested with TensorRT Edge-LLM 0.11.1 and TensorRT 10.16.1.11.
Our takeThe interesting design choice is deployment parity: one runtime from the desktop-sized Spark to Jetson, DRIVE and IGX means the engine configuration, not just the model weights, carries over. The tutorial still leaves the hard part out. Engine builds are sensitive to batch size, sequence length and precision, and a default build that fits 64GB may not fit a Thor module's memory or latency budget. Speculative decoding likewise depends on a draft model whose acceptance rate is workload-specific, so the speed-up should be measured per application rather than assumed.