Fully Sharded Data Parallel (FSDP)

A technique that shards model parameters, gradients, and optimizer states across GPUs to save memory