Presentation
A Data-Size Adaptive Approach to I/O of Poorly Load Balanced In Situ Data Extracts
DescriptionEfficient parallel I/O is essential for large-scale scientific workflows, particularly in in situ visualization pipelines where output sizes and data distribution can vary significantly over time.
We present a new data-size based aggregation strategy for the ADIOS2 BP5 engine, designed to improve parallel I/O performance under load imbalance. Whereas existing aggregation strategies for ADIOS group writers based on compute node assignment, our approach dynamically balances subfile sizes according to the amount of data each process will write. We evaluate this strategy using a synthetic workload under low and severe load imbalance. Results show that the data-size based aggregation matches or outperforms existing strategies. These findings highlight the potential of adaptive aggregation strategies to improve I/O performance for imbalanced scientific workloads.
We present a new data-size based aggregation strategy for the ADIOS2 BP5 engine, designed to improve parallel I/O performance under load imbalance. Whereas existing aggregation strategies for ADIOS group writers based on compute node assignment, our approach dynamically balances subfile sizes according to the amount of data each process will write. We evaluate this strategy using a synthetic workload under low and severe load imbalance. Results show that the data-size based aggregation matches or outperforms existing strategies. These findings highlight the potential of adaptive aggregation strategies to improve I/O performance for imbalanced scientific workloads.
Event Type
Workshop
TimeSunday, 16 November 202511:20am - 11:45am CST
Location242
AI, Machine Learning, & Deep Learning
Clouds & Distributed Computing
Performance Evaluation, Scalability, & Portability
Scientific & Information Visualization