Presentation
LowDiff: Efficient Frequent Checkpointing via Low-Cost Differential for High-Performance Distributed Training Systems
DescriptionDistributed training of large deep-learning models often leads to failures, so checkpointing is commonly employed for recovery. State-of-the-art studies focus on frequent checkpointing for fast recovery from failures. However, it generates numerous checkpoints, incurring substantial costs and thus degrading training performance. Recently, differential checkpointing has been proposed to reduce costs, but it is limited to recommendation systems, so its application to general distributed training systems remains unexplored.
This paper proposes LowDiff, an efficient frequent-checkpointing framework that reuses compressed gradients (commonly used in distributed training), serving as differential checkpoints to reduce cost. Furthermore, LowDiff incorporates a batched gradient write optimization to efficiently persist these differentials to storage. It also dynamically tunes both the checkpoint frequency and the batching size to maximize the performance. Experiments on various workloads show that LowDiff can achieve checkpointing frequency up to per iteration with less than 3.1% overhead on training time.
This paper proposes LowDiff, an efficient frequent-checkpointing framework that reuses compressed gradients (commonly used in distributed training), serving as differential checkpoints to reduce cost. Furthermore, LowDiff incorporates a batched gradient write optimization to efficiently persist these differentials to storage. It also dynamically tunes both the checkpoint frequency and the batching size to maximize the performance. Experiments on various workloads show that LowDiff can achieve checkpointing frequency up to per iteration with less than 3.1% overhead on training time.
Event Type
Paper
TimeWednesday, 19 November 20252:37pm - 3:00pm CST
Location263-264
HPC for Machine Learning