Three months of reading on distributed training, condensed

Notes from roughly forty papers on distributed training, organised by how much the field actually agrees. Kept for my own reference, put here in case it saves someone else the same three months.

What the literature agrees on

• Communication, not computation, is the scaling bottleneck past a modest device count. Nobody argues this any more.
• Sharding optimizer state is the single largest memory win available, and it is close to free in accuracy terms.
• Gradient accumulation trades wall clock for memory predictably enough that it can be treated as a tuning knob rather than a design decision.
• Mixed precision is the default. The debate is over which format, not whether.

Where it quietly contradicts itself

• Whether parameter servers are obsolete. Most surveys treat them as historical. The papers that actually test heterogeneous or unreliable hardware keep finding they hold up. The disagreement is really about what cluster you assume.
• The crossover point where sharding starts paying off. Reported thresholds vary by more than an order of magnitude, and almost nobody reports the interconnect alongside the number, which is what makes the comparison meaningless.
• Whether scaling efficiency should be measured against one GPU or one node. Both are used, they are not comparable, and papers rarely say which they picked.

Numbers that stopped being true

Anything benchmarked before roughly 2023 is measuring a different machine. Interconnect bandwidth, kernel fusion, and the compiler stack all moved enough that the old crossover points do not transfer. I would treat pre 2023 throughput comparisons as evidence about method design and not as evidence about numbers.

Open questions I did not find answered

1. How much of the reported FSDP overhead is inherent to sharding, and how much is the wrap policy being wrong by default?
2. Is there a principled way to choose shard boundaries from the model graph, rather than from parameter counts?
3. What does any of this look like when the workers are genuinely unreliable rather than merely uneven?

References

Placeholder. A plain numbered list of the papers worth reading first goes here, each with a one line note on why it is on the list rather than a full citation. The point of this section is triage, not bibliography.