HeteroYarn: Hadoop於異質平台之資源管理自我調校系統

Abstract

Apache Hadoop的蓬勃發展讓許多巨量資料的問題得以解決，但因為Hadoop運算平台的效能瓶頸使得Hadoop應用程式效能不彰常令人詬病。如要將Hadoop應用程式移植到如圖形處理器上達到效能增加的目的，往往需要程式開發者花費大量的心力將原本的應用程式移植到這些圖形處理器上，不僅是困難度增加，同時也容易因為的應用程式與系統資源管理調教不當，而無法達到預期的效果。 本篇論文提出一個結合Hadoop YARN以及Aparapi程式庫的系統，來解決 於異質平台的資源管理問題。我們提供了一個應用程式接口讓使用者容易在異質平台上開發他們的程式。系統藉由剖析圖形處理器以及中央處理器上工作的執行時間，讓Hadoop應用程式達到執行時間的最佳化。我們同時提出了數種方法來讓多個應用程式能公平地分享叢集的計算資源。在文末的實驗中，會看到在我們的系統下一個應用程式所得到的效能提昇。我們也分析了多個應用程式在不同的公平資源分享方法下，對程式效能所帶來的影響。

The booming of Apache Hadoop solves many kinds of big data problems, but the poor performance of Hadoop applications due to the bottlenecks of computing is always reviled. Porting Hadoop applications to accelerators, such as GPUs, is a solution to speedup the performance. However, programmers may take great effort to redesign applications for GPUs, and have troubles with managing the CPU and GPU resources. It is not feasible to let users handle the above difficulties. In this thesis, We proposed a framework which combines Hadoop YARN and Aparapi library for computing resources management in heterogeneous platforms. We provided an API to help users in porting their MapReduce applications onto heterogeneous platforms. Our work uses an optimized strategy to minimize the execution of a Hadoop application by profiling the execution time of tasks on CPUs and GPUs. We also proposed several methods to fairly share the CPU and GPU resources among running applications in the cluster. In the experiments, we show the speedup of an application, and analyze the effects to performance by different methods for resources fair sharing among multiple applications.