Presentation
Invited Talk 1: Memorization vs Reasoning in MoEs and Estimating Memory Consumption in Distributed Training
DescriptionMixture of Experts (MoE) allows one to increase the model capacity with minimal training/inference cost. Recent LLMs such as Qwen3-235B-A22B, gpt-oss-120B, Kimi-K2, GLM-4.5, DeepSeek-R1 are very sparse MoEs, though there are some subtle differences in the details of the architecture. The first part of this talk will focus on our recent efforts to measure the effect of sparsity on memorization tasks and reasoning tasks. We initially find that increasing the total parameters without increasing the active parameters increases the performance on memorization tasks but shows an inverse scaling on reasoning tasks. However, when the dataset is carefully constructed we show that the inverse scaling on reasoning tasks disappears. The second part of this talk will describe a tool to estimate the memory consumption during distributed training, and its effectiveness when trying to maximize per GPU Flop/s on a given system.
Presenter
Event Type
Workshop
TimeSunday, 16 November 20259:10am - 10:00am CST
Location264
Similar Presentations