Persistent model states plus selected activation memory.
Per-device memory map
Each GPU bar separates parameters, gradients, optimizer states, and activations. Striped regions are sharded across the selected worker group.
Stage 0 replicates every model state.
Replicated state memory divided by sharded state memory.
Approximate collective traffic per worker for model states.
Whether the estimate stays within selected GPU capacity.
Data parallel workers need not replicate every training byte.
Adam stores more than parameters.
BF16 parameters and gradients use two bytes each. FP32 master weights plus first and second moments add twelve more bytes, making optimizer states the largest persistent component.
replicated state = 2P + 2P + 12P = 16P bytesEach stage shards another state.
Stage 1 partitions optimizer states. Stage 2 also partitions gradients. Stage 3 partitions parameters and gathers them only when a layer executes.
stage 1: O/N, stage 2: O/N + G/N, stage 3: O/N + G/N + P/NStage 1 attacks the largest block.
Optimizer moments and master weights dominate Adam memory, so sharding them produces a large first reduction without parameter gathers.
Stage 3 trades memory for traffic.
Parameter shards minimize persistent memory, but each layer needs parameter all-gathers and backward reduce-scatter communication.
Offload moves the bottleneck.
CPU offload frees GPU optimizer memory but introduces host capacity and interconnect bandwidth requirements. It does not make the state disappear.