BEGIN:VCALENDAR
VERSION:2.0
PRODID:Linklings LLC
BEGIN:VTIMEZONE
TZID:America/Chicago
X-LIC-LOCATION:America/Chicago
BEGIN:DAYLIGHT
TZOFFSETFROM:-0600
TZOFFSETTO:-0500
TZNAME:CDT
DTSTART:19700308T020000
RRULE:FREQ=YEARLY;BYMONTH=3;BYDAY=2SU
END:DAYLIGHT
BEGIN:STANDARD
TZOFFSETFROM:-0500
TZOFFSETTO:-0600
TZNAME:CST
DTSTART:19701101T020000
RRULE:FREQ=YEARLY;BYMONTH=11;BYDAY=1SU
END:STANDARD
END:VTIMEZONE
BEGIN:VEVENT
DTSTAMP:20260202T201805Z
LOCATION:241
DTSTART;TZID=America/Chicago:20251116T120000
DTEND;TZID=America/Chicago:20251116T123000
UID:submissions.supercomputing.org_SC25_sess209_ws_fgai114@linklings.com
SUMMARY:Frameworks for Large Language Model Serving in HPC Environments
DESCRIPTION:Rohan Marwaha (University of Illinois at Urbana-Champaign); Qi
 nren Zhou (University of Illinois at Urbana-Champaign, National Center for
  Supercomputing Applications (NCSA)); Kastan Day and Asmita Dabholkar (Uni
 versity of Illinois at Urbana-Champaign); and Volodymyr Kindratenko (Unive
 rsity of Illinois at Urbana-Champaign, National Center for Supercomputing 
 Applications (NCSA))\n\nWe introduce open-source frameworks for deploying 
 and running large language models (LLMs) within high-performance computing
  (HPC) environments. One such framework targets high-\nthroughput batch in
 ference, enabling users to submit LLM requests in an OpenAI-compatible for
 mat as traditional HPC jobs. Another framework is based on Ray Serve and i
 t provides dynamic, on-demand allocation of HPC resources for interactive 
 LLM serving via APIs, supporting applications such as chatbots and AI agen
 ts. The third framework is a production-grade, always-\non platform for re
 al-time interaction, that relies on a dedicated GPU server for model infer
 ence. These frameworks are designed to abstract away underlying computer s
 ystem complexities, allowing researchers to request and utilize GPU resour
 ces for model inference without manual environment setup. We describe thes
 e systems and report LLM-specific performance metrics. Results demonstrate
  that the proposed frameworks enable scalable and resource-efficient LLM s
 erving across both batch and interactive workloads in support of diverse u
 ser needs.\n\nRecording: Livestreamed, Recorded\n\nRegistration Category: 
 Technical Program Reg Pass, Workshop Reg Pass\n\nSession Chairs: Charlie C
 atlett (Argonne National Laboratory (ANL), University of Chicago); Javier 
 Aula-Blasco (Barcelona Supercomputing Center (BSC)); Valerie Taylor (Argon
 ne National Laboratory (ANL), University of Chicago); Laura Morselli (CINE
 CA); and Rio Yokota (Institute of Science Tokyo)\n\n
END:VEVENT
END:VCALENDAR