請用此 Handle URI 來引用此文件:
http://tdr.lib.ntu.edu.tw/jspui/handle/123456789/56509
標題: | 以機器學習改善Hadoop系統優化 Configuration Tuning on Hadoop System Based on Machine Learning |
作者: | Chi-Ou Chen 陳紀甌 |
指導教授: | 蘇雅韻(Ya-Yunn Su) |
關鍵字: | 巨量資料,分散式系統,機器學習,全局優化,隨機抽樣, big data,distributed system,machine learning,global optimization,random sampling, |
出版年 : | 2014 |
學位: | 碩士 |
摘要: | 隨著巨量資料分析的興起, 支持此類大規模資料處理的系統, 如分散
式系統也越受到關注, 在管理建立在日益龐大機器叢集的系統, 系統管 理者必須花更多心力管理。除了使系統能夠穩定地支援各式各樣的資 料分析應用, 也需要對系統作優化, 讓效能夠有效的提昇, 提高系統的使 率及降低運行這些資料分析應用的時間。然而, 對大規模機器叢集而 言, 系統參數調校是複雜的, 管理者除了要處理各個機器之間互動的問 題, 也必須針對不同應用, 了解其運算特性, 進而調校系統參數。而現行 系統參數調校的方法有可用性不高, 以及可調校的參數受到限制等缺 點。本研究基於這些現行的的方法, 以機器學習來改善上述的這些問 題, 打破這些限制使系統效能更進一步提昇 Big Data has emerged in recent year. Systems which is able to support such large-scale data analysis are received more attentions. The distributed system like Hadoop is most used for the analysis. However, it will be increasingly difficult for system administrators to manage the whole system when the cluster of the system scales out. System administrator should maintain the system to execute applications stably. Besides, they need to optimize the system to improve the performance, increase the system utilization and reduce the latency of application executing. And the configuration problem is the most important issue of system optimization. Configuration parameter tuning is related lots of complicated issues. It needs to understand the interaction between physical machines and the behavior of each applications. The current method, rule-based and cost-based optimization, have drawbacks like unfeasibility and limitation of configuration parameter space. Our work exploit machine learning to solve the problem to improve the performance. |
URI: | http://tdr.lib.ntu.edu.tw/jspui/handle/123456789/56509 |
全文授權: | 有償授權 |
顯示於系所單位: | 資訊工程學系 |
文件中的檔案:
檔案 | 大小 | 格式 | |
---|---|---|---|
ntu-103-1.pdf 目前未授權公開取用 | 1.65 MB | Adobe PDF |
系統中的文件,除了特別指名其著作權條款之外,均受到著作權保護,並且保留所有的權利。