<p>NVIDIA has open-sourced NVIDIA Attribution, a workload-centric failure-attribution system it says has run in production across its AI infrastructure for years, analysing millions of workloads a week across hundreds of thousands of GPUs. The problem it targets: a single GPU or fabric fault surfaces as NCCL errors on other ranks, scheduler state changes and thousands of downstream messages, so the reported exception shows where a workload stopped rather than where it failed.</p><p>The software ships as two components. An Attribution Service is a self-hosted HTTP wrapper around a rules engine that takes normalised observations and returns one verdict per workload attempt. An Attribution Blueprint is a Python reference orchestrator that discovers workloads, resolves execution context, runs collectors, calls the service and writes a structured result. The Blueprint includes execution-context connectors for Slurm, Kueue and static inputs, log and metric connectors for Loki, local files, Kubernetes pod logs and PromQL, and output connectors for stdout and SQLite, plus a read-only dashboard.</p><p>Attribution runs in four stages: scope the workload attempt from scheduler identifiers or an execution window, collect observations from application logs, syslogs, GPU driver and Fabric Manager messages plus node and network metrics, extract and correlate events so deterministic rules separate downstream symptoms from the causing event, then return the verdict with terminal state, attributed domain, symptoms, affected nodes and collector coverage. In the worked example, three NCCL errors appear across nodes while system logs on one node carry XID 149 decoding to an NVLink signal-integrity issue; the rules keep that as the root cause and demote the NCCL reports to evidence. When several causes fit, the attributed domain is reported as ambiguous, and the diagnosis records that uncertainty rather than converting incomplete telemetry into a definite answer.</p>
Our takeDeterministic rules buy reproducibility at the price of coverage: a failure mode with no rule or parser attached is simply not attributed, and the service says so rather than guessing. Keeping demoted symptoms as inspectable evidence is the design detail worth copying, because it preserves the reasoning behind a verdict. The binding dependency is telemetry completeness, which the structured output records explicitly.