Presentation
SIGN IN TO VIEW THIS PRESENTATION Sign In
Seamless End-to-End Containerized HPC Environments
DescriptionHigh-performance computing environments face increasing challenges from diverse scientific workflows, imposing conflicting demands for stability, customization, and reproducibility that traditional monolithic software stacks cannot accommodate.
We present a comprehensive approach to seamless end-to-end containerized HPC environments which decomposes the technical challenge into five manageable areas:
specification and construction of environments, session provisioning, scheduler integration, system integration, and security.
We develop and evaluate prototypes across these five technical areas, demonstrating practical feasibility through Spack-based environment construction with CI/CD pipelines, transparent session access via PAM and Kubernetes, and flexible job execution using Slurm's native container support.
% Our system integration framework supports multiple MPI strategies from host library injection to fully containerized stacks, accommodating diverse performance and portability requirements.
Through this work, we demonstrate that comprehensive containerization of HPC environments can be achieved using open standards, providing enhanced reproducibility and flexibility without sacrificing user experience.
We present a comprehensive approach to seamless end-to-end containerized HPC environments which decomposes the technical challenge into five manageable areas:
specification and construction of environments, session provisioning, scheduler integration, system integration, and security.
We develop and evaluate prototypes across these five technical areas, demonstrating practical feasibility through Spack-based environment construction with CI/CD pipelines, transparent session access via PAM and Kubernetes, and flexible job execution using Slurm's native container support.
% Our system integration framework supports multiple MPI strategies from host library injection to fully containerized stacks, accommodating diverse performance and portability requirements.
Through this work, we demonstrate that comprehensive containerization of HPC environments can be achieved using open standards, providing enhanced reproducibility and flexibility without sacrificing user experience.
