The four families of technique

Four families of technique claim to make large model training cheaper. They are rarely stated in the same units, which makes them impossible to compare and easy to stack badly.

The four families

• Sharding. Split parameters, gradients, and optimizer state across devices. Buys memory, costs communication.
• Quantization. Use fewer bits per number. Buys memory and bandwidth, costs accuracy in a way that depends on where you apply it.
• Sparsity. Skip work entirely, either structurally or dynamically. Buys arithmetic, costs hardware utilisation.
• Better optimizers. Reach the same loss in fewer steps. Buys wall clock, costs memory and tuning stability.

Putting them in one unit

The only comparison that survives contact with a real cluster is cost to reach a target loss, measured in GPU hours. Memory saved is not a benefit on its own. It is a benefit exactly to the extent that it lets you raise batch size, lower parallelism degree, or use fewer nodes.

Stated that way, several widely reported wins shrink considerably, because the memory they save was not the thing limiting the run.

How they interact

They are not additive. Sharding increases communication, and quantization reduces communication volume, so those two compose well. Sparsity and sharding compose badly, because irregular work makes the collective sizes uneven and the slowest shard sets the pace.

Part 2 takes quantization on its own, since it is the family where the reported numbers and the numbers you get diverge most.