Presentation
Productive Scalable Distributed Task Scheduling Using an MPI-based Backend for Dagger
DescriptionDistributed computing frameworks are vital for managing complex workloads in high-performance computing and scientific research. Julia's Dagger.jl supports task-based parallelism using TCP communication, suitable for cloud and local environments. However, TCP limits performance on modern HPC systems with low-latency, high-bandwidth interconnects. We introduce MPIAcceleration, an MPI-based backend replacing TCP with MPI-aware task placement and data movement. We benchmarked MPIAcceleration against TCP using parallel Cholesky decomposition on the Aurora exascale supercomputer at Argonne National Laboratory. Results show MPI successfully enables Dagger on HPC interconnects, significantly outperforming TCP on Aurora by overcoming the latency limitations inherent in standard TCP.
Event Type
Research and ACM SRC Posters
TimeTuesday, 18 November 20258:00am - 5:00pm CST
LocationSecond Floor Atrium
Archive
view