應用於可微分式神經網路架構搜尋之混合高效剪枝策略

Abstract

神經網路架構搜尋因可自動化設計及高效搜索能力受到許多關注，可找出比傳統人工設計的網路更具高效能的架構，但其所需的龐大運算量與搜尋時間，使其仍具有改善空間。可微分式架構搜尋，透過將原本離散的搜索空間泛化成連續空間，使其可透過計算梯度調整架構參數以降低所需時間，但仍需要數小時至數天的搜尋時間。因此本研究提出了HPE-DARTS，一種透過混合硬剪枝與軟剪枝的搜尋策略，並引入代理評估的方式，來達到高效搜尋神經網路架構。 HPE-DARTS的搜尋方法中主要有三個要素，分別是暖身、軟剪枝、以及硬剪枝。在搜索起初時，透過一個暖身階段，訓練網路本體使網路權重在進行任何剪枝等行為前可先收斂且穩定。而軟剪枝的策略，則是透過測量每個操作的重要性後，將較不重要的操作，減弱其對網路輸出的貢獻度。而有關操作重要性的部分，則是使用所提出的NetPerfProxy來計算。相對於以往使用驗證資料集來計算各操作對於網路準確度的影響程度，使用NetPerfProxy可以大幅減少驗證所需時間並且不損失過多準確性。而硬剪枝則是在軟剪枝調整完架構參數後，將較不重要的操作直接移除，使後續搜尋能夠更高效的專注在尋找更重要的參數。 我們將方法實作在NAS-Bench-201與DARTS的搜索空間上，實驗結果顯示，相對於使用傳統DARTS類型的方法，HPE-DARTS能夠大幅度的減少所需搜尋時間的情況下，達到與之相匹的結果，並且引入NetPerfProxy後，可降低原本驗證所需的大量時間且提升搜尋結果，表明HPE-DARTS僅需少許時間，即可穩定搜尋出具有良好性能的網路架構。

Neural architecture search (NAS) has emerged as a powerful methodology for automating the design of deep neural networks. However, the computational cost in conventional NAS approaches, particularly those that rely on differentiable search methods like DARTS, often renders them impractical for resource-constrained environments. In response to these challenges, we introduce the Hybrid Pruning and Proxy Evaluation in Differentiable Architecture Search (HPE-DARTS), an innovative framework that integrates both soft and hard pruning techniques with a proxy evaluation strategy to enhance the efficiency and effectiveness of architecture search. Our proposed method initiates with a warm-up phase to stabilize the network parameters before engaging in a cyclic process of soft and hard pruning. The soft pruning evaluates the importance of architectural components via the proposed NetPerfProxy without extensive validating evaluation, allowing for rapid iteration and refinement. Subsequently, hard pruning decisively eliminates the least valuable operations, systematically narrowing down the search space to the most promising architectures. This hybrid approach not only reduces the computational burden but also accelerates the convergence towards optimal network structures. Experimental results demonstrate that HPE-DARTS significantly reduces search time and provides competitive performance for the derived architectures compared to traditional DARTS. By adopting a NetPerfProxy, our method addresses the typical reliance on costly validation procedures, thereby enabling a more scalable and practical search process.