NotesDraft/Paper Notes
What our DDP, FSDP, and parameter server paper actually found
We spent most of last year benchmarking three ways to train a large model across a GPU cluster. The paper is in NCAA 2026 and it has the tables. This is the part that does not fit in a paper: what we expected, what actually happened, and the one figure that made us rerun half the experiments.
The setup, in one paragraph
Three strategies. Distributed Data Parallel keeps a full copy of the model on every GPU and averages gradients after each step. Fully Sharded Data Parallel splits the parameters, gradients, and optimizer state across GPUs and gathers each layer only while it is being used. Parameter server keeps the authoritative weights on dedicated nodes and has workers push and pull updates. We ran all three across the same cluster, the same models, and the same data pipeline, then measured throughput, memory, and scaling efficiency as we added GPUs.
What we expected
FSDP to win. That was the whole hypothesis going in. It is the newer method, it is the one everyone reaches for when a model will not fit, and the memory argument for it is airtight: if you shard optimizer state you get a large fraction of your memory back, and memory is the thing that stops you.
We assumed the memory saving was free. It is not free. It is paid for in communication, and the exchange rate depends on your model's shape.
What actually happened
FSDP won decisively on memory and lost on throughput at small scale. Below a certain model size the all-gather traffic it needs to reconstruct each layer cost more than the memory it saved was worth. DDP, which is the dumbest of the three, was the fastest option for anything that comfortably fit in memory, and it was not close.
Placeholder for a results figure or table. A chart, an inline SVG, or a plain HTML table drops in here. Anything inside a block that starts with a tag is passed through the build untouched.
Figure 1. Caption sits under the block in muted text.
The parameter server results were the ones that surprised us. It is the oldest approach and the conventional wisdom is that it has been superseded. In our heterogeneous runs, where the workers were not identical, it held up better than either of the others, because it is the only one of the three that does not force every worker to wait for the slowest.
The plot that changed our methodology
Partway through we plotted throughput against GPU count for a fixed model and got a curve that bent in the wrong place. Scaling efficiency dropped hard at a specific worker count and then partially recovered. It was not the interconnect. It was that our sharding boundary was splitting a transformer block across two devices, so a single forward pass needed an extra round trip that the neighboring configurations did not.
Every configuration after that got its shard boundaries checked against the model architecture before the run rather than after. That one habit is probably worth more than any individual number in the paper.
What I would tell someone starting this
Pick the strategy from your bottleneck, not from the literature's default. If the model fits, use DDP and stop reading. If it does not fit, FSDP buys you the room but bring a communication budget. If your hardware is uneven, the old parameter server design solves a problem the newer methods do not even try to solve.