LAB LEDGER DESK
00:00:00 UTC2026-11-17
Official sources only·Nothing invented

AWS · Note ·

Detecting and recovering from silent data corruption on Amazon SageMaker HyperPod

Silent data corruption is a GPU fault that returns wrong but plausible numbers with no error, quietly poisoning distributed training. This post adds three…

Also filed

Also filed by AWS — Build a model improvement flywheel with Amazon SageMaker HyperPod Data Capture (2026-11-13).

What moved

Silent data corruption is a GPU fault that returns wrong but plausible numbers with no error, quietly poisoning distributed training. This post adds three checks to a PyTorch DDP loop to detect it, localize the faulty node on Amazon SageMaker HyperPod, and hand the node to automatic recovery for replacement.

Why it matters

Filed as a public note from AWS, 2026-11-11. The desk tags this brief Hardware / Enterprise.

On the record

  • Filed from the AWS official RSS on 2026-11-11.
  • Primary source host: aws.amazon.com.
  • Silent data corruption is a GPU fault that returns wrong but plausible numbers with no error, quietly poisoning distributed training.
Primary source
aws.amazon.com
Desk
Logged as brief 11216
Cite this briefDetecting and recovering from silent data corruption on Amazon SageMaker HyperPod (AWS, 2026-11-11) — https://labledgerdesk.pages.dev/b/2026/11/11/detecting-and-recovering-from-silent-data-corruption-on-amazon/