BEGIN:VCALENDAR
VERSION:2.0
PRODID:Linklings LLC
BEGIN:VTIMEZONE
TZID:America/Chicago
X-LIC-LOCATION:America/Chicago
BEGIN:DAYLIGHT
TZOFFSETFROM:-0600
TZOFFSETTO:-0500
TZNAME:CDT
DTSTART:19700308T020000
RRULE:FREQ=YEARLY;BYMONTH=3;BYDAY=2SU
END:DAYLIGHT
BEGIN:STANDARD
TZOFFSETFROM:-0500
TZOFFSETTO:-0600
TZNAME:CST
DTSTART:19701101T020000
RRULE:FREQ=YEARLY;BYMONTH=11;BYDAY=1SU
END:STANDARD
END:VTIMEZONE
BEGIN:VEVENT
DTSTAMP:20260202T201805Z
LOCATION:241
DTSTART;TZID=America/Chicago:20251116T094000
DTEND;TZID=America/Chicago:20251116T100000
UID:submissions.supercomputing.org_SC25_sess209_ws_fgai124@linklings.com
SUMMARY:Batch Tiling on Attention: Efficient Mixture of Experts Training o
 n Wafer-Scale Processors
DESCRIPTION:Daria Soboleva, Etienne Goffinet, Hui Zeng, Sangamesh Ragate, 
 Elif Albuz, and Natalia Vassilieva (Cerebras)\n\nMixture of Experts (MoE) 
 models face computational bottlenecks on wafer-scale processors due to con
 flicting batch size requirements: attention mechanisms need smaller batche
 s for memory constraints, while routable MLP layers require larger batches
  for optimal compute density.\nWe introduce \textbf{Batch Tiling on Attent
 ion (BTA)}, which decouples batch processing across MoE computation stages
  by applying dynamic tiling on attention's batch dimension. Our method pro
 cesses attention operations at reduced batch size $B$ through tiled comput
 ation, then concatenates outputs to form larger batch size $\widetilde{B} 
 = G \cdot B$ for MLP operations, where $G$ is a positive integer. This add
 resses attention memory limitations while maximizing hardware utilization 
 in expert layers.\nWe demonstrate BTA's effectiveness on Cerebras wafer-sc
 ale engines using Qwen3-like models, achieving up to 5$\times$ performance
  improvements at higher sparsity levels compared to conventional uniform b
 atching. Unlike existing GPU-focused solutions like FlashAttention and exp
 ert parallelism, BTA specifically targets wafer-scale processors' unique c
 omputational characteristics.\n\nRecording: Livestreamed, Recorded\n\nRegi
 stration Category: Technical Program Reg Pass, Workshop Reg Pass\n\nSessio
 n Chairs: Charlie Catlett (Argonne National Laboratory (ANL), University o
 f Chicago); Javier Aula-Blasco (Barcelona Supercomputing Center (BSC)); Va
 lerie Taylor (Argonne National Laboratory (ANL), University of Chicago); L
 aura Morselli (CINECA); and Rio Yokota (Institute of Science Tokyo)\n\n
END:VEVENT
END:VCALENDAR