On 2 November NVIDIA published two companion posts and open-sourced two components under Apache 2.0: NVIDIA Attribution, a workload-scoped failure-attribution service it says has run in production across its AI infrastructure for years, and NVIDIA Recovery, a cluster-level loop that detects interruptions, decides what to do about them, excludes implicated nodes and restarts jobs from checkpoints. The engineering claim worth examining is not that training failures can be automated. It is the epistemic contract: one verdict per workload attempt, symptoms preserved rather than deleted, and uncertainty named instead of resolved by guesswork.
The problem is specific. In a multi-rank job a single GPU or fabric fault surfaces as NCCL errors on other ranks, as scheduler state changes and as thousands of downstream log lines. The reported exception marks where the workload stopped, not where it failed. In the worked example, three different NCCL errors appear across ranks while system logs on one node carry XID 149, whose error words decode to an NVLink signal-integrity issue. A human reading the job log sees three communication failures; the underlying event is one link on one node.
Attribution runs in four stages: scope a workload attempt from scheduler identifiers or an explicit execution window; collect observations from application logs, syslogs, driver and Fabric Manager messages, scheduler history and node and network metrics; extract and correlate events, using deterministic rules to weigh timing and interaction and separate downstream symptoms from the causing event; return a structured verdict with terminal state, attributed domain, symptoms, affected nodes and a record of collector coverage. The Attribution Service is a self-hosted HTTP wrapper around the rules engine. The Blueprint is a Python orchestrator with connectors for Slurm, Kueue, Loki, PromQL and SQLite, plus a read-only dashboard.
Two details deserve copying. Demotion is not deletion: when the rules settle on the XID as root cause, the NCCL reports stay in the record flagged as interesting evidence, so a reviewer can see why the verdict was reached instead of trusting a label. And when several causes fit, the attributed domain is reported as ambiguous, which gives downstream systems a way to escalate rather than an authoritative-looking answer derived from half the telemetry.
The weaknesses are where the evidence thins. The published worked example is synthetic. There is no labelled corpus of real failures, no confusion matrix by failure class, and no precision or recall figure. The deployment-scale claim, millions of workloads a week across hundreds of thousands of GPUs, describes exposure, not accuracy. Deterministic rules buy reproducibility at the cost of coverage: a failure mode with no extractor and no rule is simply not attributed. Recording collector coverage is the right mitigation, but silence and absence of failure look identical on a dashboard unless somebody reads that field.
Correctness is bounded by log fidelity. XID decoding depends on driver messages surviving into the collection path; Fabric Manager output differs across fabric generations; and every scheduler words its errors a little differently. The example rule in the post matches an srun string for an invalid Pyxis argument, exactly the kind of pattern that breaks quietly after a version bump. A pre-deployment step validates signals against the contract and catches label mismatches early, but drift afterwards is not continuously tested in anything described. Rules are compiled into the service at startup rather than uploaded through the API, which is defensible for auditability, and makes the rule catalog the real product boundary.
Recovery supplies the numbers. On one training cluster between 9 July and 9 October, covering nearly 200 jobs a day, the loop recorded roughly 8,550 anomaly detections with mean detection and recovery-action latencies of 69 and 94 seconds. Two details are more honest than usual: recovery-action latency ends at the automated action, with scheduling and checkpoint restoration determining when productive training resumes; and allocation restart, restored progress and completion are tracked as distinct events. What is missing is the shape of the distribution and the error rate. Means without percentiles hide the tail, which is where long jobs suffer. Over roughly 18,600 job-days, 8,550 detections is about 0.46 per job per day, so the loop intervenes often; the number that matters next is how often the intervention was wrong.
The scale argument is where the modelling gets loose. If each GPU survives a day with 99.9% probability, a 10,000-GPU job completes the day uninterrupted with probability 0.999^10,000, about 0.005%. That assumes independence, and datacentre failures are not independent: power events, a shared switch, cooling, a bad image applied to a whole allocation, a maintenance action, all hit many GPUs at once. Independence understates correlated events while overstating the value of excluding individual nodes, since excluding three nodes does not help if the cause is a fabric port or an image tag. The empirical anchor beside it, an 11-month study of 24,000 A100 GPUs at Meta in which hardware failures interrupted fewer than 1% of jobs that consumed 19% of GPU runtime, is the stronger half of the argument because it is measured rather than modelled.
The economics of a wrong verdict compound. Excluding one node is cheap once; excluding nodes systematically because a fabric problem is being read as a node fault shrinks usable capacity on every subsequent restart. The loop can requeue into the same failure and exclude another node. A watchdog covers stalled or failed decisions, but nothing described guards against confident wrong answers. That is why the monitor-first default matters more than it looks: new deployments publish advice, record what they would have done, and take no scheduler action until an operator is satisfied with the signal-to-noise ratio. IBM Research is reported to have taken that path. Shadow deployment is the correct posture for a controller whose mistakes are durable.
Goodput, meanwhile, is decided elsewhere. If reported latency stops at the action, then for long jobs the dominant term is checkpoint write and restore time plus queueing. A 69-second detector on top of a fifteen-minute checkpoint interval is a marginal improvement; the same detector on top of five-minute asynchronous checkpoints changes what jobs are feasible. The lever that determines the real cost of a failure remains the checkpoint policy of the job, which no controller can set for you.
What would make the claim testable is not more deployment statistics but a fault-injection benchmark: XID classes, ECC and row-remap events, NVLink and port flaps, node hangs, stragglers and bad images, injected on a known cluster with ground-truth labels, reporting per-class precision and recall, time-to-goodput distributions rather than means, and a false-exclusion rate. Pair that with a published map of rule coverage against an observed failure taxonomy, and a continuous check that parsers still match the logs they were written for. Until then, whether the loop helps depends on how closely a cluster's failure mix resembles the catalog NVIDIA wrote for its own fleet.
The parts of this design that are architectural rather than promotional are worth naming: symptoms preserved as evidence, ambiguity as a first-class output, detection separated from action, and observation as the first deployment mode. The missing piece is measurement of the verdict itself. An attribution system should be the one component in the stack able to say how often it is wrong.
Sources
NVIDIA Technical Blog, Diagnose Distributed AI Workload Failures with NVIDIA Attribution (2 November 2026): https://developer.nvidia.com/blog/diagnose-distributed-ai-workload-failures-with-nvidia-attribution/ · NVIDIA Technical Blog, Keep Distributed Training Running with Automated Workload Recovery (2 November 2026): https://developer.nvidia.com/blog/keep-distributed-training-running-with-automated-workload-recovery/ · Repositories: https://github.com/dsx-ai-factory/attribution and https://github.com/dsx-ai-factory/nvidia-recovery