BEGIN:VCALENDAR
VERSION:2.0
PRODID:Linklings LLC
BEGIN:VTIMEZONE
TZID:America/Chicago
X-LIC-LOCATION:America/Chicago
BEGIN:DAYLIGHT
TZOFFSETFROM:-0600
TZOFFSETTO:-0500
TZNAME:CDT
DTSTART:19700308T020000
RRULE:FREQ=YEARLY;BYMONTH=3;BYDAY=2SU
END:DAYLIGHT
BEGIN:STANDARD
TZOFFSETFROM:-0500
TZOFFSETTO:-0600
TZNAME:CST
DTSTART:19701101T020000
RRULE:FREQ=YEARLY;BYMONTH=11;BYDAY=1SU
END:STANDARD
END:VTIMEZONE
BEGIN:VEVENT
DTSTAMP:20260202T201803Z
LOCATION:241
DTSTART;TZID=America/Chicago:20251116T103000
DTEND;TZID=America/Chicago:20251116T110000
UID:submissions.supercomputing.org_SC25_sess209_ws_fgai127@linklings.com
SUMMARY:Automated MCQA Benchmarking at Scale: Evaluating Reasoning Traces 
 as Retrieval Sources for Domain Adaptation of Small Language Models
DESCRIPTION:Ozan Gokdemir (University of Chicago, Argonne National Laborat
 ory (ANL)) and Neil Getty, Robert Underwood, Sandeep Madireddy, Franck Cap
 pello, Arvind Ramanathan, Ian Foster, and Rick Stevens (Argonne National L
 aboratory (ANL))\n\nAs scientific knowledge grows at an unprecedented pace
 , evaluation benchmarks must evolve to reflect new discoveries and ensure 
 language models are tested on current, diverse literature. We propose a sc
 alable, modular framework for generating multiple-choice question-answerin
 g (MCQA) benchmarks directly from large corpora of scientific papers. Our 
 pipeline automates every stage of MCQA creation, including PDF parsing, se
 mantic chunking, question generation, and model evaluation. As a case stud
 y, we generate more than 16,000 MCQs from 22,000 open-access articles in r
 adiation and cancer biology. We then evaluate a suite of small language mo
 dels (1.1B–14B parameters) on these questions, comparing baseline accuracy
  with retrieval-augmented generation (RAG) from paper-derived semantic chu
 nks and from reasoning traces distilled from GPT-4.1. We find that reasoni
 ng-trace retrieval consistently improves performance on both synthetic and
  expert-annotated benchmarks, enabling several small models to surpass GPT
 -4 on the 2023 Astro Radiation and Cancer Biology exam.\n\nRecording: Live
 streamed, Recorded\n\nRegistration Category: Technical Program Reg Pass, W
 orkshop Reg Pass\n\nSession Chairs: Charlie Catlett (Argonne National Labo
 ratory (ANL), University of Chicago); Javier Aula-Blasco (Barcelona Superc
 omputing Center (BSC)); Valerie Taylor (Argonne National Laboratory (ANL),
  University of Chicago); Laura Morselli (CINECA); and Rio Yokota (Institut
 e of Science Tokyo)\n\n
END:VEVENT
END:VCALENDAR