NotesDraft
Engineering write-ups and working references. A bug that took three days, a benchmark that lied, a default that was wrong, and the cluster guides I keep open in a second tab. Written down so I stop rediscovering them.
Everything I actually run on George Mason's HPC cluster: partitions, GPU sharding, Slurm recipes, and storage rules. Checked against live cluster output, which in several places contradicts the official wiki.
Aug 20, 2026
Our throughput fell off a cliff at eight GPUs and it was not the network. It was the sharding boundary landing in the middle of a transformer block.
Aug 12, 2026
A collective timeout names the rank that noticed, not the rank that caused it. Here is how to find the one that actually stalled.
Jul 9, 2026
Our numbers were 30% better than they should have been. The dataloader was serving the same cached batch every step and nothing in the metrics said so.
May 18, 2026