Presentation
SIGN IN TO VIEW THIS PRESENTATION Sign In
DagOnStore: Reliable Data Management for Workflows on the Computing Continuum with DynoStore and DAGonStar
DescriptionThe computing continuum has emerged as a promising paradigm for decentralized data processing. This approach brings computation closer to data sources, reducing latency and enabling faster insights. However, managing such distributed systems introduces new challenges, particularly in ensuring the availability and reliability of data across heterogeneous and failure-prone environments. In this paper, we focus on addressing these challenges by introducing DagOnStore as a novel component of the DAGonStar workflow engine, integrating it with the DynoStore wide-area storage system to provide resilient and location-transparent data access. DagOnStore implements reliability and availability schemes based on erasure codes and utilization-aware load-balancing to guarantee that input and output data remain accessible and consistent, even in the presence of storage node failures or disconnections. We validate our approach through different tests, demonstrating that DagOnStore enables scalable and fault-tolerant workflow execution across the computing continuum with minimal user intervention.
