START — every GPU holds a full tensor, split into 4 chunks GPU 0 a0 a1 a2 a3 GPU 1 b0 b1 b2 b3 GPU 2 c0 c1 c2 c3 GPU 3 d0 d1 d2 d3 START — each GPU holds one different slice, all correct GPU 0 p0 GPU 1 p1 GPU 2 p2 GPU 3 p3 AFTER reduce → one GPU has the sum GPU 0 Σ0 Σ1 Σ2 Σ3 GPU 1 GPU 2 GPU 3 AFTER all-reduce → every GPU has the sum GPU 0 Σ0 Σ1 Σ2 Σ3 GPU 1 Σ0 Σ1 Σ2 Σ3 GPU 2 Σ0 Σ1 Σ2 Σ3 GPU 3 Σ0 Σ1 Σ2 Σ3 AFTER reduce-scatter → each GPU has one slice of the sum GPU 0 Σ0 GPU 1 Σ1 GPU 2 Σ2 GPU 3 Σ3 AFTER all-gather → every GPU has all slices, concatenated GPU 0 p0 p1 p2 p3 GPU 1 p0 p1 p2 p3 GPU 2 p0 p1 p2 p3 GPU 3 p0 p1 p2 p3 all four contribute · only GPU 0 keeps the result reduce-scatter → the whole sum exists once, spread across the GPUs GPU 0 Σ0 GPU 1 Σ1 GPU 2 Σ2 GPU 3 Σ3 then all-gather → which is exactly the all-reduce result GPU 0 Σ0 Σ1 Σ2 Σ3 GPU 1 Σ0 Σ1 Σ2 Σ3 GPU 2 Σ0 Σ1 Σ2 Σ3 GPU 3 Σ0 Σ1 Σ2 Σ3 all-reduce = reduce-scatter + all-gather AFTER PASS 1 — each GPU added what its left neighbour sent GPU 0 a0 a1 a2 d3+a3 GPU 1 a0+b0 b1 b2 b3 GPU 2 c0 b1+c1 c2 c3 GPU 3 d0 d1 c2+d2 d3 AFTER 3 PASSES — reduce-scatter done, each GPU owns one finished chunk GPU 0 Σ1 GPU 1 Σ2 GPU 2 Σ3 GPU 3 Σ0 AFTER 3 MORE PASSES — all-gather done, everyone has everything GPU 0 Σ0 Σ1 Σ2 Σ3 GPU 1 Σ0 Σ1 Σ2 Σ3 GPU 2 Σ0 Σ1 Σ2 Σ3 GPU 3 Σ0 Σ1 Σ2 Σ3 per GPU: 2 × (N−1) chunks × (D / N) bytes = 2D · (N−1)/N N = 2 → 1.00 D N = 4 → 1.50 D N = 8 → 1.75 D N = 64 → 1.97 D N → ∞ → 2D bounded by 2D no matter how many GPUs you add