Presentation
Lightning Talk: Adaptive In-Situ Sampling Framework for Accelerated Online Change Point Detection in Scientific Workflows
DescriptionHigh-performance computing (HPC) systems at the
Exascale generate monitoring and simulation data at rates that
exceed the capacity of traditional offline analysis, especially for
change point detection (CPD) in large-scale scientific workflows.
We present an adaptive in-situ sampling framework that combines
truncated singular value decomposition (SVD)-based manifold
learning with the Kernel Cumulative Sum (KCUSUM) method
for statistical CPD. Implemented with MPI4py for scalable
interprocess communication and ADIOS2 for high-throughput
streaming I/O, the framework dynamically adjusts sampling rates
in real time based on anomaly likelihood, enabling selective data
retention without sacrificing scientific information.
Evaluations on synthetic datasets and large-scale molecular
dynamics (MD) simulations from NWChem demonstrate up to
59% memory reduction, sub-second detection latency for critical
events, and near-perfect detection accuracy, all while eliminating
the need for storing full trajectories. These preliminary results
highlight the framework’s potential to deliver resource-efficient,
real-time anomaly detection in data-intensive Exascale scientific
computing environments,
Exascale generate monitoring and simulation data at rates that
exceed the capacity of traditional offline analysis, especially for
change point detection (CPD) in large-scale scientific workflows.
We present an adaptive in-situ sampling framework that combines
truncated singular value decomposition (SVD)-based manifold
learning with the Kernel Cumulative Sum (KCUSUM) method
for statistical CPD. Implemented with MPI4py for scalable
interprocess communication and ADIOS2 for high-throughput
streaming I/O, the framework dynamically adjusts sampling rates
in real time based on anomaly likelihood, enabling selective data
retention without sacrificing scientific information.
Evaluations on synthetic datasets and large-scale molecular
dynamics (MD) simulations from NWChem demonstrate up to
59% memory reduction, sub-second detection latency for critical
events, and near-perfect detection accuracy, all while eliminating
the need for storing full trajectories. These preliminary results
highlight the framework’s potential to deliver resource-efficient,
real-time anomaly detection in data-intensive Exascale scientific
computing environments,
Event Type
Workshop
TimeSunday, 16 November 202511:10am - 11:20am CST
Location242
AI, Machine Learning, & Deep Learning
Clouds & Distributed Computing
Performance Evaluation, Scalability, & Portability
Scientific & Information Visualization