OPTIQ: A Data Movement Optimization Framework for Data-centric Applications on Supercomputers

Data-centric applications on supercomputers need to reliably and rapidly compute and move large amounts of data through interconnect networks. The same trend is also observed in commodity clusters. Thus, optimizing data movement is essential at extreme scales in order to effectively utilize the systems. However, most of the optimization works are carried out separately at different layers in the supercomputers. In this dissertation, I present a holistic approach that takes system’s network routing, interconnect network topology and application’s communication patterns into optimizing that results in better performance over current data movement mechanisms. My approach includes heuristic algorithms and optimization models with solvers to improve and optimize data movement. The approach is realized in a Data Movement Optimization Framework (OPTIQ) that provides an application programming interface (API) requiring minimal changes in applications for integration. The OPTIQ framework is also extensible, allowing further development and expansion on algorithms for recommending multiple paths for data movement, different ways to schedule data transfer, and various mechanisms to transfer the data. It can also be extended to other systems.