NVIDIA's technical blog published the second part of a series on the Global Performance Tuner introduced in TensorRT 11.1. The design represents alternative scheduling strategies as explicit, searchable build routes: each compiler knob has a finite set of allowed values and a default, and a route is one concrete combination of those values. TensorRT exposes three APIs, one to apply a fixed route, one to read the current route and one to enumerate all available routes, and the trtexec tool expands a regex-like route expression into a list of concrete candidates. It then builds the model once per candidate, measures accuracy and end-to-end performance in each iteration, discards candidates that crash during the build or fail the accuracy requirement, and selects the fastest eligible engine. The release ships more than 200 tunable knobs, with the same implementation slated for Python in Torch-TensorRT 2.15. Across ten evaluated workloads, the best eligible route ran between 1.14x and 1.63x faster than the default build route.
Our takeThis frames compilation as a search problem with an accuracy constraint attached, which is a cleaner statement than hand-tuning heuristics, but the cost model is brute force: each candidate route is a full engine build plus a measured run, so the 1.14 to 1.63x band is paid for in build time that grows with the expression being explored. The 200-plus knobs also mean the search is only as good as the route expression a human writes. Reproducibility depends on whether the winning route is recorded next to the engine.