請用此 Handle URI 來引用此文件：
http://tdr.lib.ntu.edu.tw/jspui/handle/123456789/95633完整後設資料紀錄
| DC 欄位 | 值 | 語言 |
|---|---|---|
| dc.contributor.advisor | 王勝德 | zh_TW |
| dc.contributor.advisor | Sheng-De Wang | en |
| dc.contributor.author | 林竑逸 | zh_TW |
| dc.contributor.author | Hung-I Lin | en |
| dc.date.accessioned | 2024-09-15T16:13:19Z | - |
| dc.date.available | 2024-09-16 | - |
| dc.date.copyright | 2024-09-14 | - |
| dc.date.issued | 2024 | - |
| dc.date.submitted | 2024-08-12 | - |
| dc.identifier.citation | [1] Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey. Journal of Machine Learning Research, 20(55):1–21, 2019.
[2] Pengzhen Ren, Yun Xiao, Xiaojun Chang, Po-yao Huang, Zhihui Li, Xiaojiang Chen, and Xin Wang. A comprehensive survey of neural architecture search: Challenges and solutions. ACM Computing Surveys, 54(4):1–34, 2021. [3] Barret Zoph and Quoc Le. Neural architecture search with reinforcement learning. In Proceedings of International Conference on Learning Representations, 2017. [4] Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V. Le. Learning transferable architectures for scalable image recognition. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8697–8710, 2018. [5] Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V. Le. Regularized evolution for image classifier architecture search. Proceedings of AAAI Conference on Artificial Intelligence, 33(01):4780–4789, Jul. 2019. [6] Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: Differentiable architecture search. In Proceedings of International Conference on Learning Representations, 2019. [7] Xin Chen, Lingxi Xie, Jun Wu, and Qi Tian. Progressive differentiable architecture search: Bridging the depth gap between search and evaluation. In Proceedings of IEEE/CVF International Conference on Computer Vision, pages 1294–1303, 2019. [8] Tai-Che Feng and Sheng-De Wang. VP-DARTS: Validated pruning differentiable architecture search. In Proceedings of International Conference on Agents and Artificial Intelligence, pages 47–57, 2024. [9] Joe Mellor, Jack Turner, Amos Storkey, and Elliot J Crowley. Neural architecture search without training. In Proceedings of International Conference on Machine Learning, pages 7588–7598, 2021. [10] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In Proceedings of International Conference on Learning Representations, 2015. [11] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 770–778, 2016. [12] Colin White, Mahmoud Safari, Rhea Sukthanker, Binxin Ru, Thomas Elsken, Arber Zela, Debadeepta Dey, and Frank Hutter. Neural architecture search: Insights from 1000 papers. arXiv:2301.08727, 2023. [13] George Kyriakides and Konstantinos Margaritis. An introduction to neural architecture search for convolutional networks. arXiv:2005.11074, 2020. [14] Bowen Baker, Otkrist Gupta, Nikhil Naik, and Ramesh Raskar. Designing neural network architectures using reinforcement learning. In Proceedings of International Conference on Learning Representations, 2017. [15] Lingxi Xie and Alan Yuille. Genetic cnn. In Proceedings of IEEE/CVF International Conference on Computer Vision, pages 1388–1397, 2017. [16] Esteban Real, Sherry Moore, Andrew Selle, Saurabh Saxena, Yutaka Leon Suematsu, Jie Tan, Quoc V. Le, and Alexey Kurakin. Large-scale evolution of image classifiers. In Proceedings of International Conference on Machine Learning, pages 2902–2911, 2017. [17] Hieu Pham, Melody Guan, Barret Zoph, Quoc Le, and Jeff Dean. Efficient neural architecture search via parameters sharing. In Proceedings of International Conference on Machine Learning, pages 4095–4104, 2018. [18] Gabriel Bender, Pieter-Jan Kindermans, Barret Zoph, Vijay Vasudevan, and Quoc Le. Understanding and simplifying one-shot architecture search. In Proceedings of International Conference on Machine Learning, pages 550–559, 2018. [19] Shreyas Saxena and Jakob Verbeek. Convolutional neural fabrics. In Proceedings of Advances in Neural Information Processing Systems, 2016. [20] Lingxi Xie, Xin Chen, Kaifeng Bi, Longhui Wei, Yuhui Xu, Lanfei Wang, Zhengsu Chen, An Xiao, Jianlong Chang, Xiaopeng Zhang, and Qi Tian. Weight-sharing neural architecture search: A battle to shrink the optimization gap. ACM Computing Surveys, 54(9):1–37, 2021. [21] Arber Zela, Thomas Elsken, Tonmoy Saikia, Yassine Marrakchi, Thomas Brox, and Frank Hutter. Understanding and robustifying differentiable architecture search. In Proceedings of International Conference on Learning Representations, 2020. [22] Ruochen Wang, Minhao Cheng, Xiangning Chen, Xiaocheng Tang, and Cho-Jui Hsieh. Rethinking architecture selection in differentiable NAS. In Proceedings of International Conference on Learning Representations, 2021. [23] Hanwen Liang, Shifeng Zhang, Jiacheng Sun, Xingqiu He, Weiran Huang, Kechen Zhuang, and Zhenguo Li. Darts+: Improved differentiable architecture search with early stopping. arXiv:1909.06035, 2019. [24] Xiangxiang Chu, Xiaoxing Wang, Bo Zhang, Shun Lu, Xiaolin Wei, and Junchi Yan. DARTS-: Robustly stepping out of performance collapse without indicators. In Proceedings of International Conference on Learning Representations, 2021. [25] Peng Ye, Baopu Li, Yikang Li, Tao Chen, Jiayuan Fan, and Wanli Ouyang. 𝛽-DARTS: Beta-decay regularization for differentiable architecture search. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10864–10873, 2022. [26] Xiangxiang Chu, Tianbao Zhou, Bo Zhang, and Jixiang Li. Fair DARTS: Eliminating unfair advantages in differentiable architecture search. In Proceedings of European Conference on Computer Vision, pages 465–480. Springer, 2020. [27] Yuhui Xu, Lingxi Xie, Xiaopeng Zhang, Xin Chen, Guo-Jun Qi, Qi Tian, and Hongkai Xiong. PC-DARTS: Partial channel connections for memory-efficient architecture search. In Proceedings of International Conference on Learning Representations, 2020. [28] Xiangning Chen, Ruochen Wang, Minhao Cheng, Xiaocheng Tang, and Cho-Jui Hsieh. DrNAS: Dirichlet neural architecture search. In Proceedings of International Conference on Learning Representations, 2021. [29] Guohao Li, Guocheng Qian, Itzel C. Delgadillo, Matthias Müller, Ali Thabet, and Bernard Ghanem. Sgas: Sequential greedy architecture search. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1617–1627, 2020. [30] Meng-Ting Wu, Hung-I Lin, and Chun-Wei Tsai. A training-free neural architecture search algorithm based on search economics. IEEE Transactions on Evolutionary Computation, 28(2):445–459, 2024. [31] Meng-Ting Wu, Hung-I Lin, and Chun-Wei Tsai. A training-free genetic neural architecture search. In Proceedings of ACM International Conference on Intelligent Computing and Its Emerging Applications, page 65–70, 2022. [32] Xin Chen, Lingxi Xie, Jun Wu, and Qi Tian. Progressive darts: Bridging the optimization gap for nas in the wild. International Journal of Computer Vision, 129:638– 655, 2021. [33] Xuanyi Dong and Yi Yang. Nas-bench-201: Extending the scope of reproducible neural architecture search. In Proceedings of International Conference on Learning Representations, 2020. [34] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009. [35] Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8:229–256, 1992. [36] Stefan Falkner, Aaron Klein, and Frank Hutter. BOHB: Robust and efficient hyperparameter optimization at scale. In Proceedings of International Conference on Machine Learning, pages 1437–1446, 2018. [37] Liam Li and Ameet Talwalkar. Random search and reproducibility for neural architecture search. In Proceedings of Uncertainty in Artificial Intelligence Conference, pages 367–377, 22-25 Jul 2020. [38] Xuanyi Dong and Yi Yang. Searching for a robust neural architecture in four gpu hours. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1761–1770, 2019. [39] Xuanyi Dong and Yi Yang. One-shot neural architecture search via self-evaluated template network. In Proceedings of IEEE/CVF International Conference on Computer Vision, pages 3680–3689, 2019. | - |
| dc.identifier.uri | http://tdr.lib.ntu.edu.tw/jspui/handle/123456789/95633 | - |
| dc.description.abstract | 神經網路架構搜尋因可自動化設計及高效搜索能力受到許多關注，可找出比傳統人工設計的網路更具高效能的架構，但其所需的龐大運算量與搜尋時間，使其仍具有改善空間。可微分式架構搜尋，透過將原本離散的搜索空間泛化成連續空間，使其可透過計算梯度調整架構參數以降低所需時間，但仍需要數小時至數天的搜尋時間。因此本研究提出了HPE-DARTS，一種透過混合硬剪枝與軟剪枝的搜尋策略，並引入代理評估的方式，來達到高效搜尋神經網路架構。
HPE-DARTS的搜尋方法中主要有三個要素，分別是暖身、軟剪枝、以及硬剪枝。在搜索起初時，透過一個暖身階段，訓練網路本體使網路權重在進行任何剪枝等行為前可先收斂且穩定。而軟剪枝的策略，則是透過測量每個操作的重要性後，將較不重要的操作，減弱其對網路輸出的貢獻度。而有關操作重要性的部分，則是使用所提出的NetPerfProxy來計算。相對於以往使用驗證資料集來計算各操作對於網路準確度的影響程度，使用NetPerfProxy可以大幅減少驗證所需時間並且不損失過多準確性。而硬剪枝則是在軟剪枝調整完架構參數後，將較不重要的操作直接移除，使後續搜尋能夠更高效的專注在尋找更重要的參數。 我們將方法實作在NAS-Bench-201與DARTS的搜索空間上，實驗結果顯示，相對於使用傳統DARTS類型的方法，HPE-DARTS能夠大幅度的減少所需搜尋時間的情況下，達到與之相匹的結果，並且引入NetPerfProxy後，可降低原本驗證所需的大量時間且提升搜尋結果，表明HPE-DARTS僅需少許時間，即可穩定搜尋出具有良好性能的網路架構。 | zh_TW |
| dc.description.abstract | Neural architecture search (NAS) has emerged as a powerful methodology for automating the design of deep neural networks. However, the computational cost in conventional NAS approaches, particularly those that rely on differentiable search methods like DARTS, often renders them impractical for resource-constrained environments. In response to these challenges, we introduce the Hybrid Pruning and Proxy Evaluation in Differentiable Architecture Search (HPE-DARTS), an innovative framework that integrates both soft and hard pruning techniques with a proxy evaluation strategy to enhance the efficiency and effectiveness of architecture search.
Our proposed method initiates with a warm-up phase to stabilize the network parameters before engaging in a cyclic process of soft and hard pruning. The soft pruning evaluates the importance of architectural components via the proposed NetPerfProxy without extensive validating evaluation, allowing for rapid iteration and refinement. Subsequently, hard pruning decisively eliminates the least valuable operations, systematically narrowing down the search space to the most promising architectures. This hybrid approach not only reduces the computational burden but also accelerates the convergence towards optimal network structures. Experimental results demonstrate that HPE-DARTS significantly reduces search time and provides competitive performance for the derived architectures compared to traditional DARTS. By adopting a NetPerfProxy, our method addresses the typical reliance on costly validation procedures, thereby enabling a more scalable and practical search process. | en |
| dc.description.provenance | Submitted by admin ntu (admin@lib.ntu.edu.tw) on 2024-09-15T16:13:19Z No. of bitstreams: 0 | en |
| dc.description.provenance | Made available in DSpace on 2024-09-15T16:13:19Z (GMT). No. of bitstreams: 0 | en |
| dc.description.tableofcontents | Acknowledgements i
摘要 ii Abstract iv Contents vi List of Figures viii List of Tables ix Chapter 1 Introduction 1 Chapter 2 Related Work 4 2.1 Neural architecture search 4 2.2 DARTS 5 2.3 Approach of Evaluation 8 Chapter 3 Approach 11 3.1 Preliminaries 11 3.2 Hybrid Pruning Mechanism 12 3.2.1 Warm-up Stage 14 3.2.2 Hybrid Pruning Stage: Soft Pruning Part 14 3.2.3 Hybrid Pruning Stage: Hard Pruning Part 15 3.3 Network Performance Proxy 17 Chapter 4 Experiment 19 4.1 Experiment Overview 19 4.1.1 Environment 20 4.2 Results On NAS-Bench-201 20 4.2.1 Search Space 20 4.2.2 Implementation Details 21 4.2.3 Search Results 22 4.3 Results On DARTS 23 4.3.1 Search Space 23 4.3.2 Implementation Details 23 4.3.3 Search Results 24 4.4 Ablation Study 27 4.4.1 Number of Warm-up Epoches 27 4.4.2 Number of Pruning Epoches 28 4.4.3 Number of stages and operations to hard pruning 30 4.4.4 Impact of NetPerfProxy Modifications 30 Chapter 5 Conclusion 32 References 34 | - |
| dc.language.iso | en | - |
| dc.subject | 可微分架構搜尋 | zh_TW |
| dc.subject | 深度學習 | zh_TW |
| dc.subject | 神經網路架構搜尋 | zh_TW |
| dc.subject | 混合剪枝 | zh_TW |
| dc.subject | 代理評估 | zh_TW |
| dc.subject | Proxy Evaluation | en |
| dc.subject | Deep Learning | en |
| dc.subject | Neural Architecture Search | en |
| dc.subject | Differentiable Architecture Search | en |
| dc.subject | Hybrid Pruning | en |
| dc.title | 應用於可微分式神經網路架構搜尋之混合高效剪枝策略 | zh_TW |
| dc.title | HPE-DARTS: Hybrid Pruning and Proxy Evaluation in Differentiable Architecture Search | en |
| dc.type | Thesis | - |
| dc.date.schoolyear | 112-2 | - |
| dc.description.degree | 碩士 | - |
| dc.contributor.oralexamcommittee | 雷欽隆;于天立;余承叡 | zh_TW |
| dc.contributor.oralexamcommittee | Chin-Laung Lei;Tian-Li Yu;Cheng-Ruei Yu | en |
| dc.subject.keyword | 深度學習,神經網路架構搜尋,可微分架構搜尋,混合剪枝,代理評估, | zh_TW |
| dc.subject.keyword | Deep Learning,Neural Architecture Search,Differentiable Architecture Search,Hybrid Pruning,Proxy Evaluation, | en |
| dc.relation.page | 39 | - |
| dc.identifier.doi | 10.6342/NTU202404167 | - |
| dc.rights.note | 同意授權(限校園內公開) | - |
| dc.date.accepted | 2024-08-13 | - |
| dc.contributor.author-college | 電機資訊學院 | - |
| dc.contributor.author-dept | 電機工程學系 | - |
| 顯示於系所單位： | 電機工程學系 | |
文件中的檔案：
| 檔案 | 大小 | 格式 | |
|---|---|---|---|
| ntu-112-2.pdf 授權僅限NTU校內IP使用（校園外請利用VPN校外連線服務） | 683.49 kB | Adobe PDF |
系統中的文件，除了特別指名其著作權條款之外，均受到著作權保護，並且保留所有的權利。