Nvidia documented Mission Control 2.5, whose autonomous recovery engine detects, diagnoses and recovers from anomalies across the stack. Until this release the service was limited to high-performance computing workloads scheduled with Slurm; the post says it now extends to Kubernetes-based workloads managed with Nvidia Run:ai.
The stated goal is for AI factories to behave as self-healing systems: failures are detected, recovery decisions are made from workload context and failure type, and jobs resume automatically with as little wasted compute as possible. The post frames the problem in terms of mean time to repair and mean time between failures, arguing that manual intervention, static checkpointing and reactive monitoring do not scale once an operator runs dozens of concurrent multi-node jobs on shared infrastructure.
Nvidia describes the recovery loop as isolating anomalies, tagging affected nodes for maintenance, and moving them out of risky or stranded states and back into the schedulable pool so training can continue.
Our takeSelf-healing at this scale lives or dies on failure classification: a dying GPU, a bad NIC and a workload that corrupts its own checkpoint can present the same symptoms, while the right action differs — quarantine the node versus fix the job. The metric that matters is therefore time-to-repair broken out by failure class, plus how often the engine quarantines hardware that was actually healthy. Anyone adopting this should instrument both before letting it restart training jobs unattended.