NCCL timeouts are lying to you

A collective timeout names the rank that noticed, not the rank that caused it. Those are almost never the same rank.

What the error looks like

Watchdog caught collective operation timeout:
WorkNCCL(SeqNum=48211, OpType=ALLREDUCE, Timeout=1800000ms)
Rank 3 detected a hang.

Why rank 3 is innocent

An all-reduce completes only when every rank has entered it. If rank 6 is stuck somewhere earlier in the step, every other rank sits in the collective waiting. The watchdog fires on whichever of the waiting ranks hits its timeout first, which is essentially arbitrary. Rank 3 is the messenger.

Finding the rank that actually stalled

• Compare the sequence number across ranks. The stalled rank is the one whose last completed collective has a lower SeqNum than everyone else.
• Log a heartbeat per rank per step. The one that stops logging first is the culprit, and it usually stops for an unrelated reason: a dataloader worker died, a checkpoint write blocked, an assert fired on one shard of the data.
• Only after that should you suspect the network.

Takeaway

Treat the reported rank as evidence about timing, not about cause. The useful question is never "why did rank 3 time out," it is "which rank never arrived."