Distributed Training
Multi-node pretraining and fine-tuning that launches cleanly, stays in lockstep and converges. I work across the parallelism stack: data, tensor, pipeline, sequence and expert parallel, and the collectives that hold them together. Rendezvous and process-group init, world-size and rank derivation, MoE router numerics, mixed-precision drift, checkpoint resharding across topologies.
What it buys you: A 64-GPU run that gets past init on the first try, survives a node loss, and keeps a finite loss at step 40,000.
- FSDP2 / ZeRO-3
- Megatron-Core TP/PP/CP
- Expert parallelism
- NCCL collectives
- FP8 delayed scaling
- Distributed checkpointing
