The Ultra-Scale Playbook vol-3: DeepSpeed ZeRO

25 Sep, 2025

Zero Redundancy Optimiser

TLDR: Instead of naive data parallelism,

partition the optimizer states, gradients, and parameters across the DP dimension,
while still allowing computation with the full parameter set. Can require more communnication between DP ranks (think: overlapping).

3 stages of ZeRO

ZeRO-1: only partition optimiser states
ZeRO-2: partition optimiser states + gradients
ZeRO-3: partition optimiser states + gradients + params

Given the model parameter count $N$ , mixed precision training with Adam dictates the following memory usage:

params bf16 = $2 N$
grads bf16 = $2 N$ , if we want to accumulate grad in fp32 then $4 N$
optimiser states fp32 = $4 N + 4 N$ , and parameters fp32 = $4 N$

For efficiency let's keep grad-accumulation in bf16 and so total memory usage becomes $2 N + 2 N + 12 N$ . Now given data parallel degree $N_{d}$ ,

Baseline: $2 N + 2 N + 12 N$ across each DP rank
ZeRO-1: $2 N + 2 N + 12 N / N_{d}$
ZeRO-2: $2 N + (2 N + 12 N) / N_{d}$
ZeRO-3: $(2 N + 2 N + 12 N) / N_{d}$

ZeRO-1: only shard optimiser states

Each DP rank forward passes with the same full set of bf16 params but on a different microbatch of data.
Each DP rank back-props with the same full set of bf16 grads but on a different microbatch of data.
Since each DP rank stores only $1 / N_{d}$ of the optimiser states, and we want to back-prop on all the data via a mean/sum reduction, we perform a reduce_scatter on the gradients, i.e., $g r a d_{k}^{j} \leftarrow \sum_{l} g r a d_{l}^{j}$ . This accumulates the gradient of each $N / N_{d}$ -th parameter chunk on all the data $\equiv \nabla_{w} ℓ (X) = \sum_{i = 1}^{n} \nabla_{w} ℓ (X_{i})$ .
Now that we have each $N / N_{d}$ -th parameter chunk accumulated across the data, we perform an optimiser step using only $1 / N_{d}$ states $⟹$ $1 / N_{d}$ fp32 updated params, converted to $1 / N_{d}$ bf16 params.
At this point, each DP rank has $1 / N_{d}$ updated bf16 params. It now receives the other $(N_{d} - 1) / N_{d}$ fraction of params from the other DP ranks via an all_gather operation.
Now each DP rank has the same full set of updated bf16 params, ready for the next step of training.
Final memory footprint: $2 N + 2 N + 12 N / N_{d}$ .

💡 To update each chunk during reduce_scatter, only that chunk (across different microbatches) is needed per machine.

What motivates ZeRO-2: Why not accumulate a chunk on all data during back-prop, and then only store the grads required for the optimiser step? That eliminates the need to always store all the grads.

ZeRO-2: shard optimiser states + grad

Only perform reduce_scatter during back-prop. Now only $1 / N_{d}$ -th of the gradients are needed in memory, freeing up memory and giving us a much better memory footprint of $2 N + (2 N + 12 N) / N_{d}$ .

What motivates ZeRO-3: Distributing the params across DP ranks can make forward pass possible by doing an all_gather for each microbatch per DP rank. Think of it this way: we temporarily "stitch" all the shards of a layer together, forward pass a microbatch through it, then flush the gathered shards to keep only $1 / N_{d}$ params in memory.

ZeRO-3: shard everything

Forward pass: distribute the bf16 param set across DP ranks $⟹$ $2 N / N_{d}$ memory on each rank. Given a microbatch, use all_gather to stitch a sharded layer together, apply the layer on the microbatch, and flush the other gathered param shards from memory.
Back-prop: use all_gather to stitch sharded params together, compute gradient for gathered params per microbatch, and then reduce_scatter to accumulate all microbatches into each chunk.
Final memory footprint: $(2 N + 2 N + 12 N) / N_{d}$ .

💡 ZeRO-3 requires $2 \cdot num\_layers - 1$ additional calls to all_gather w.r.t ZeRO-2, and each comes with a small base latency. Also, we gather all the shards once in the forward pass and once during back-prop, incurring a communication tax of $N$ each time. Adding another communication tax of $N$ from the reduce_scatter called during back-prop, our total communication cost is $3 N$ compared to $2 N$ in ZeRO-2.

💡 While this may seem like a lot of overhead due to communication, in practice we use prefetching to compensate for this: simply all_gather weights for the next layer when we forward pass through the current layer. Similarly, all_gather weights of the previous layer while back-propping through the current layer.

When will this fail: when our DP dimension exceeds 512, the communication overhead will become too large due to ring latency, and our overlap will fail. Need to think of something else at those scales.