Presentation
Demystifying the Resilience of Large Language Model Inference: An End-to-End Perspective
DescriptionDeep neural networks are known to be resilient to random bitwise faults in their parameters. However, this resilience has primarily been established through evaluations on classification models. The extent to which this claim holds for large language models remains under-explored.
In this work, we conduct an extensive measurement study on the impact of random bitwise faults in commercial-scale language model inference. We first expose that these language models are not truly resilient to random bit-flips. While aggregate metrics such as accuracy may suggest resilience, an in-depth inspection of the generated outputs shows significant degradation in text quality. Our analysis also shows that tasks requiring more complex reasoning suffer more from performance and quality degradation. Moreover, we extend our resilience analysis to models with augmented reasoning capabilities, such as Chain of Thought or Mixture of Experts architectures.
In this work, we conduct an extensive measurement study on the impact of random bitwise faults in commercial-scale language model inference. We first expose that these language models are not truly resilient to random bit-flips. While aggregate metrics such as accuracy may suggest resilience, an in-depth inspection of the generated outputs shows significant degradation in text quality. Our analysis also shows that tasks requiring more complex reasoning suffer more from performance and quality degradation. Moreover, we extend our resilience analysis to models with augmented reasoning capabilities, such as Chain of Thought or Mixture of Experts architectures.
Event Type
Paper
TimeWednesday, 19 November 20253:30pm - 3:52pm CST
Location263-264
Performance Measurement, Modeling, & Tools