Making Large Model Training Cheaper
A survey in parts, covering the four families of technique that actually move the cost curve and how they interact when stacked.
2 parts
NotesDraft
Topic Surveys
What a whole subarea agrees on, where it quietly contradicts itself, and which results stopped being true. Written as I read into a topic, so these grow in parts.
Individual notes
Three months of reading on distributed training, condensed
What the DDP, FSDP, and parameter server literature actually agrees on, where it quietly contradicts itself, and which numbers stopped being true after 2023.
Aug 18, 2026