AWS Machine Learning describes HyperPod checks for silent data corruption in distributed training

AWS Machine Learning describes a set of checks for detecting silent data corruption in PyTorch DistributedDataParallel training on Amazon SageMaker HyperPod. The approach adds three guards: non-finite gradient checks, ring replay audits, and weight fingerprint comparisons. When a check fires, the system maps the suspect rank to a node and lets HyperPod replace the bad hardware. The post says the method was tested with software-injected faults on a 4-instance ml.p5.48xlarge cluster with 32 NVIDIA H100 GPUs.