請用此 Handle URI 來引用此文件：
http://tdr.lib.ntu.edu.tw/jspui/handle/123456789/55664完整後設資料紀錄
| DC 欄位 | 值 | 語言 |
|---|---|---|
| dc.contributor.advisor | 洪士灝(Shih-Hao Hung) | |
| dc.contributor.author | Jia-Kuan Su | en |
| dc.contributor.author | 蘇嘉冠 | zh_TW |
| dc.date.accessioned | 2021-06-16T04:15:51Z | - |
| dc.date.available | 2016-08-25 | |
| dc.date.copyright | 2014-08-25 | |
| dc.date.issued | 2014 | |
| dc.date.submitted | 2014-08-20 | |
| dc.identifier.citation | [1] “Apache Hadoop,” http://hadoop.apache.org/.
[2] S. Owen, R. Anil, T. Dunning, and E. Friedman, Mahout in action. Manning, 2011. [3] A. Thusoo, J. S. Sarma, N. Jain, Z. Shao, P. Chakka, S. Anthony, H. Liu, P. Wyckoff, and R. Murthy, “Hive: a warehousing solution over a map-reduce framework,” Proceedings of the VLDB Endowment, vol. 2, no. 2, pp. 1626–1629, 2009. [4] J. Dean and S. Ghemawat, “MapReduce: simplified data processing on large clusters,” Communications of the ACM, vol. 51, no. 1, pp. 107–113, 2008. [5] V. K. Vavilapalli, A. C. Murthy, C. Douglas, S. Agarwal, M. Konar, R. Evans, T. Graves, J. Lowe, H. Shah, S. Seth et al., “Apache hadoop yarn: Yet another resource negotiator,” in Proceedings of the 4th annual Symposium on Cloud Computing. ACM, 2013, p. 5. [6] “Aparapi,” http://code.google.com/p/aparapi/. [7] K. Shvachko, H. Kuang, S. Radia, and R. Chansler, “The hadoop distributed file system,” in Mass Storage Systems and Technologies (MSST), 2010 IEEE 26th Symposium on. IEEE, 2010, pp. 1–10. [8] “Apache Tez,” http://hortonworks.com/hadoop/tez/. [9] M. Isard, M. Budiu, Y. Yu, A. Birrell, and D. Fetterly, “Dryad: distributed data-parallel programs from sequential building blocks,” in ACM SIGOPS Operating Systems Review, vol. 41, no. 3. ACM, 2007, pp. 59–72. [10] M. Zaharia, M. Chowdhury, M. J. Franklin, S. Shenker, and I. Stoica, “Spark: cluster computing with working sets,” in Proceedings of the 2nd USENIX conference on Hot topics in cloud computing, 2010, pp. 10–10. [11] “OpenCL: The open standard for parallel programming of heterogeneous systems,” https://www.khronos.org/opencl/. [12] B. He, W. Fang, Q. Luo, N. K. Govindaraju, and T. Wang, “Mars: a MapReduce framework on graphics processors,” in Proceedings of the 17th international conference on Parallel architectures and compilation techniques. ACM, 2008, pp. 260–269. [13] W. Fang, B. He, Q. Luo, and N. K. Govindaraju, “Mars: Accelerating mapreduce with graphics processors,” Parallel and Distributed Systems, IEEE Transactions on, vol. 22, no. 4, pp. 608–620, 2011. [14] J. A. Stuart and J. D. Owens, “Multi-GPU MapReduce on GPU clusters,” in Parallel & Distributed Processing Symposium (IPDPS), 2011 IEEE International. IEEE, 2011, pp. 1068–1079. [15] M. Xin and H. Li, “An implementation of GPU accelerated MapReduce: using Hadoop with OpenCL for data-and compute-intensive jobs,” in Service Sciences (IJCSS), 2012 International Joint Conference on. IEEE, 2012, pp. 6–11. [16] M. Grossman, M. Breternitz Jr, and V. Sarkar, “HadoopCL: MapReduce on Distributed Heterogeneous Platforms through Seamless Integration of Hadoop and OpenCL.” in IPDPS Workshops, 2013, pp. 1918–1927. [17] M. Zaharia, A. Konwinski, A. D. Joseph, R. H. Katz, and I. Stoica, “Improving MapReduce Performance in Heterogeneous Environments.” in OSDI, vol. 8, no. 4, 2008, p. 7. [18] K. Shirahata, H. Sato, and S. Matsuoka, “Hybrid map task scheduling for GPU-based heterogeneous clusters,” in Cloud Computing Technology and Science (CloudCom), 2010 IEEE Second International Conference on. IEEE, 2010, pp. 733–740. [19] G. Giunta, R. Montella, G. Agrillo, and G. Coviello, “A GPGPU transparent virtualization component for high performance computing clouds,” in Euro-Par 2010-Parallel Processing. Springer, 2010, pp. 379–391. [20] “HSA Foundation,” http://www.hsafoundation.com/. [21] “Hadoop Performance Monitoring Tool,” http://code.google.com/p/hadoop-toolkit/wiki/HadoopPerformanceMonitoring. | |
| dc.identifier.uri | http://tdr.lib.ntu.edu.tw/jspui/handle/123456789/55664 | - |
| dc.description.abstract | Apache Hadoop的蓬勃發展讓許多巨量資料的問題得以解決，但因為Hadoop運算平台的效能瓶頸使得Hadoop應用程式效能不彰常令人詬病。如要將Hadoop應用程式移植到如圖形處理器上達到效能增加的目的，往往需要程式開發者花費大量的心力將原本的應用程式移植到這些圖形處理器上，不僅是困難度增加，同時也容易因為的應用程式與系統資源管理調教不當，而無法達到預期的效果。
本篇論文提出一個結合Hadoop YARN以及Aparapi程式庫的系統，來解決 於異質平台的資源管理問題。我們提供了一個應用程式接口讓使用者容易在異質平台上開發他們的程式。系統藉由剖析圖形處理器以及中央處理器上工作的執行時間，讓Hadoop應用程式達到執行時間的最佳化。我們同時提出了數種方法來讓多個應用程式能公平地分享叢集的計算資源。在文末的實驗中，會看到在我們的系統下一個應用程式所得到的效能提昇。我們也分析了多個應用程式在不同的公平資源分享方法下，對程式效能所帶來的影響。 | zh_TW |
| dc.description.abstract | The booming of Apache Hadoop solves many kinds of big data problems, but the poor performance of Hadoop applications due to the bottlenecks of computing is always reviled. Porting Hadoop applications to accelerators, such as GPUs, is a solution to speedup the performance. However, programmers may take great effort to redesign applications for GPUs, and have troubles with managing the CPU and GPU resources. It is not feasible to let users handle the above difficulties.
In this thesis, We proposed a framework which combines Hadoop YARN and Aparapi library for computing resources management in heterogeneous platforms. We provided an API to help users in porting their MapReduce applications onto heterogeneous platforms. Our work uses an optimized strategy to minimize the execution of a Hadoop application by profiling the execution time of tasks on CPUs and GPUs. We also proposed several methods to fairly share the CPU and GPU resources among running applications in the cluster. In the experiments, we show the speedup of an application, and analyze the effects to performance by different methods for resources fair sharing among multiple applications. | en |
| dc.description.provenance | Made available in DSpace on 2021-06-16T04:15:51Z (GMT). No. of bitstreams: 1 ntu-103-R01922081-1.pdf: 4493131 bytes, checksum: 65ec75be260e9cb977cad0aa2eb49345 (MD5) Previous issue date: 2014 | en |
| dc.description.tableofcontents | 致謝...i
中文摘要...ii Abstract...iii 1 Introduction...1 1.1 Thesis Organization...3 2 Background study and Related Works...4 2.1 Hadoop...4 2.1.1 Hadoop MapReduce...5 2.1.2 Hadoop YARN...6 2.2 OpenCL and Aparapi...7 2.3 Related Works...9 3 Framework of HeteroYarn....11 3.1 Programming of MapReduce Applications in HeteroYarn...11 3.2 Overview of HeteroYarn...13 3.2.1 Container of HeteroYarn...14 3.2.2 Node Manager of HeteroYarn...14 3.2.3 HeteroYarn Application Master...15 3.2.4 Resource Manager of HeteroYarn...16 3.3 Resource Request Decision Maker...16 3.3.1 FlowChart of Resource Request Decision Maker...17 3.3.2 Re-calculateτ...19 3.3.3 Maximum Utilization Request Plan...19 3.3.4 Scheduling of Reduce Tasks...22 3.4 HeteroYarn Fair Scheduler...23 3.4.1 Fair Sharing of CPU Resources...23 3.4.2 Fair Sharing of GPU Resources...24 4 Evaluation...29 4.1 Experimental Setup...29 4.2 Results of Resource Request Decision Maker...30 4.2.1 Benchmark: Gaussian Blur...30 4.2.2 Speedup...31 4.3 Results of HeteroYarn Fair Scheduler...34 4.3.1 Benchmarks...35 4.3.2 Waiting Time of a Task...35 4.3.3 GPU Utilization...36 4.3.4 Execution Time of Jobs...37 5 Conclusion and Future Work...39 5.1 Conclusion...39 5.2 Future Work...39 Bibliography...42 | |
| dc.language.iso | en | |
| dc.subject | 資源管理 | zh_TW |
| dc.subject | 巨量資料 | zh_TW |
| dc.subject | 異質平台 | zh_TW |
| dc.subject | big data | en |
| dc.subject | heterogeneous platforms | en |
| dc.subject | resource management | en |
| dc.title | HeteroYarn: Hadoop於異質平台之資源管理自我調校系統 | zh_TW |
| dc.title | HeteroYarn: A Self-Tuning Resource Management System for Hadoop on Heterogeneous Platforms | en |
| dc.type | Thesis | |
| dc.date.schoolyear | 102-2 | |
| dc.description.degree | 碩士 | |
| dc.contributor.oralexamcommittee | 劉邦鋒(Pangfeng Liu),施吉昇(Chi-Sheng Shih),廖世偉(SW Liao) | |
| dc.subject.keyword | 巨量資料,異質平台,資源管理, | zh_TW |
| dc.subject.keyword | big data,heterogeneous platforms,resource management, | en |
| dc.relation.page | 44 | |
| dc.rights.note | 有償授權 | |
| dc.date.accepted | 2014-08-20 | |
| dc.contributor.author-college | 電機資訊學院 | zh_TW |
| dc.contributor.author-dept | 資訊工程學研究所 | zh_TW |
| 顯示於系所單位： | 資訊工程學系 | |
文件中的檔案：
| 檔案 | 大小 | 格式 | |
|---|---|---|---|
| ntu-103-1.pdf 未授權公開取用 | 4.39 MB | Adobe PDF |
系統中的文件，除了特別指名其著作權條款之外，均受到著作權保護，並且保留所有的權利。