Hadoop系統參數優化

Abstract

在當前big data的時代，Hadoop系統對於分析和應用大數據有著至關重要的作用，我們既希望能夠把Hadoop系統參數能夠調節到最佳的狀態又希望能夠在不花費更多在硬體的更新上。因此我的碩論的主題選擇在Hadoop系統參數的優化，在這裡主要針對希望優化的效能是在於減少單一任務的執行時間。我採用的是三段式模型： （1）是在眾多參數中找到對於系統影響最大的參數，根據map和reduce分開觀察並選出20個參數作為我們主要要調節的參數； （2）是建立系統時間的預測模型，根據這20個參數去搜集更多的任務執行的時間和相對應的參數作為我們建立模型的基礎，運用機器學習的方法去做建模並且選擇出最適合的三層式模型； （3）是建立系統的優化模型，每次優化機會在設定的參數範圍內隨機選取出來參數，並且把它放到之前建好的預測的模型去預測其執行的時間，經過我設定好的優化模型最終會找到一個執行時間最短的參數組合。我總共選擇了4個程式，經過以上的方法組合去驗證。

Hadoop system is very popular recent year, which is a software framework with distributed processing large-scale data-sets by using a cluster of machines with MapReduce programming model. However, there are still two essential challenges for Hadoop users to manage the Hadoop system. (1) To tune the parameters appropriately; (2) To deal with dozens of configuration parameters which are involved to its performance. This paper will focus on optimizing the Hadoop MapReduce job performance. Our approach has two key model: Prediction and Optimization. The Prediction model is to estimate execution time of a MapReduce job and the Optimization model is to search the approximately optimal configuration parameters by invoking the prediction part repeatedly. By using an analytical method to choose approximately optimal configuration parameters to improve users’ job performance . Besides the configuration parameter tuning, the relevance of each parameters and the evaluation of our methods will also be discussed in this paper. Our paper may provide users a better method to improve the Hadoop system performance and save the hardware resource.