Presentation
MLP-Offload: Multi-Level, Multi-Path Offloading for LLM Pre-Training To Break the GPU Memory Wall
DescriptionLarge language models (LLMs) have been rapidly adopted across all domains, supporting divergent use cases with remarkable accuracy. However, training these massive models requires scaling across multiple GPUs. Given the expensive and limited GPU resources, advanced redundancy elimination and parallelization techniques are employed to maximize training throughput. Furthermore, to run LLMs larger than the aggregated memory of multiple GPUs, host memory or disk offloading techniques are leveraged. Despite advanced asynchronous multi-tier read/write strategies, such offloading strategies result in significant I/O overheads in the critical path of training. To this end, we propose MLP-Offload, a novel multi-level, multi-path offloading engine specifically designed for optimizing LLM training on resource-constrained setups by mitigating I/O bottlenecks. We design and implement MLP-Offload to offload the optimizer states across multiple tiers in a cache-efficient and concurrency-controlled fashion to mitigate I/O bottlenecks. Evaluations on models up to 280B parameters show that MLP-Offload achieves 2.5x faster iterations.
Event Type
Paper
TimeWednesday, 19 November 20253:30pm - 3:52pm CST
Location261-262-265-266
HPC for Machine Learning