Nvidia recovery loop logged 8,550 training anomalies and 69-second detection on one cluster

<p>NVIDIA Recovery is an open-source, workload-level service inside the company DSX OS software stack that automates recovery for training jobs across a cluster. It runs a four-stage loop: detection, using application logs and scheduler signals to recognise PyTorch and NCCL failures, CUDA errors and interconnect patterns, plus missing progress signals for hangs and sustained throughput degradation against a recent baseline for stragglers; attribution, using the same evidence and optionally NVIDIA Attribution to distinguish hardware faults, system-software issues and application bugs; isolation, where an Anomaly Remediator excludes implicated nodes from later restart attempts or stops a workload, tracks nodes across attempts and hands back to normal scheduling through a watchdog if a decision stalls; and restart or stop through the scheduler.</p><p>The service integrates with Slurm through the Slinky slurm-operator, with Kubernetes deployments using NVSentinel for GPU health monitoring and cordon-and-drain, and with NVRx for in-job hang detection and recovery. NVIDIA reports monitoring on one training cluster between July 9 and October 9 that covered nearly 200 jobs a day and recorded roughly 8,550 anomaly detections, with mean detection and recovery-action latencies of 69 and 94 seconds respectively. It notes that recovery-action latency ends at the automated action, with scheduling and checkpoint restoration determining when productive training actually resumes.</p><p>The post also argues the scale case: if each GPU runs a day without failure with 99.9% probability, a 10,000-GPU job completes that day uninterrupted with a probability of about 0.005%, and it cites an 11-month study of 24,000 A100 GPUs at Meta in which hardware failures interrupted fewer than 1% of jobs that nonetheless accounted for 19% of GPU runtime.</p>