Presentation
TianheEngine: Hierarchy-Aware Adaptive Partitioning System for Trillion-Scale Graph Processing
DescriptionGraph partitioning is essential for effectively processing large-scale graphs in distributed computing systems. However, traditional graph partitioning strategies frequently lead to elevated communication costs, particularly within distributed computing systems that utilize thousands of computing nodes. This is because prior partitioning methods fail to consider the variations in communication costs across the communication hierarchies. We propose TianheEngine for leveraging the communication hierarchy among distributed computing systems containing thousands of computing nodes. TianheEngine introduces an adaptive, communication hierarchy-aware methodology to partition and distribute large graphs across computing nodes. It exploits the communication hierarchy of the underlying distributed computing system and the sparsity characteristics of the input graphs to improve communication efficiency. We evaluated TianheEngine on fundamental graph operations using both synthetic and real-world datasets. Experimental results show that TianheEngine is superior to state-of-the-art graph partitioning methods and parallel graph systems and outperforms top-ranked systems on the latest Graph 500 list.
Event Type
Paper
TimeThursday, 20 November 202511:38am - 12:00pm CST
Location275
System Software and Cloud Computing