Presentation
SIGN IN TO VIEW THIS PRESENTATION Sign In
Quantifying AWS S3 I/O Performance Boundaries Using the Roofline Model
DescriptionHigh-performance computing facilities increasingly form hybrid environments that integrate cloud services.
To avoid cumbersome network transfers when sharing data, a new class of storage gateways map a subset of facility storage to a cloud counterpart and automatically manage data mirroring.
However, the performance characteristics of accessing AWS's S3 from HPC systems using different methods and patterns remains poorly understood.
This paper presents a roofline-based analysis of three S3 integration approaches: NFS-mounted AWS Storage Gateway, data migration through Storage Gateway, and direct S3 API transfers. We extend I/O roofline modeling to characterize operational intensity and bandwidth ceilings across varying data sizes and access patterns.
Our experimental evaluation demonstrates significant performance differences between access methods, with POSIX I/O on NFS Storage Gateway achieving up to 6.4× higher bandwidth than other approaches for large transfers. The roofline analysis reveals distinct characteristics for each method, enabling informed selection of S3 integration strategies.
To avoid cumbersome network transfers when sharing data, a new class of storage gateways map a subset of facility storage to a cloud counterpart and automatically manage data mirroring.
However, the performance characteristics of accessing AWS's S3 from HPC systems using different methods and patterns remains poorly understood.
This paper presents a roofline-based analysis of three S3 integration approaches: NFS-mounted AWS Storage Gateway, data migration through Storage Gateway, and direct S3 API transfers. We extend I/O roofline modeling to characterize operational intensity and bandwidth ceilings across varying data sizes and access patterns.
Our experimental evaluation demonstrates significant performance differences between access methods, with POSIX I/O on NFS Storage Gateway achieving up to 6.4× higher bandwidth than other approaches for large transfers. The roofline analysis reveals distinct characteristics for each method, enabling informed selection of S3 integration strategies.
Author/Presenters
Event Type
Workshop
TimeMonday, 17 November 20252:30pm - 3:00pm CST
Location230
Data Analytics
High Performance I/O, Storage, Archive, & File Systems
Storage
Livestreamed
Recorded
TP
W