整合式代理學習器用於神經架構搜索

Abstract

零代價代理具備對於未經訓練的神經網路架構評斷優劣之能力以利神經網路搜尋，使其於最近數年備受關注。然而，目前並未有任一零代價代理可在不同架構搜尋空間、任務上皆達到足夠好的表現。為了完善發揮零代價代理的潛力及複數代理之間的互補關係，我們提出了—整合式代理學習器用於神經網路搜尋。我們的方法透過學習架構的自編碼器，將離散的架構搜尋任務轉移至連續、可微分的向量空間。我們利用零代價代理計算方便快速的優勢，另外以極低成本訓練一架構的零代價代理之預測器。在此之上，我們利用此預測器，在架構的潛空間中以「梯度上升法」進行架構搜尋，其效率更勝於傳統的隨機搜尋及演化搜尋法。 我們的方法具有計算代價低廉之優勢，而搜尋得到的架構之表現也十分具競爭力，其中在ImageNet分類任務中取得76.5%的精準度，是目前在DARTS架構搜尋空間中的最佳表現。

Recently, zero-cost proxies for neural architecture search (NAS) have attracted increasing attention and research since they allows us to discover top-performing neural networks through architecture scoring without requiring training a very large network (i.e., supernet) first. Thus, it can save significant computation resources and time to complete the search. However, to our knowledge, no single proxy works the best for different tasks and scenarios. To consolidate the strength of different proxies and to reduce search bias, we propose a novel Integrated Proxy Learner (IPL) to perform a highly efficient and effective gradient-based search over random-based and evolutionary-based methods. We first train an autoencoder for one-hot encoded neural architecture. Then, we append a multi-layer perceptron (MLP) to the encoder and train the MLP to predict different proxies scores simultaneously in multi-task learning paradigm using collected score data from multiple existing zero-cost proxies. At last, we can sample a new architecture by performing gradient ascent-based search with the learned encoder and the MLP scorer. We conduct extensive experiments on the search spaces of NAS-Bench-201, NAS-Bench-101 and DARTS in different datasets, achieving the SOTA performance (76.5% top-1 accuracy) on ImageNet. The results demonstrate the effectiveness of the proposed approach over other state-of-the-art NAS algorithms.