有限條件下最佳網路自動搜索應用於分類網路之研究

Abstract

近年來，深度學習在許多領域，如影像處理、自然語言處理等方面取得了優秀的成果。然而，要設計出優秀的神經網絡架構，高度地需要該領域的知識和反覆實驗，這使得許多研究人員對機器學習望而卻步。為了簡化這一過程，神經架構搜索（NAS）得流行起來。我們首先以 AlexNet 的網路架構來深入了解不同初始化權重的方法對於網路架構性能的影響，這讓我們對於定義搜索空間提供了更深入的調查，均勻分佈初始化方法配合適當的上下界能夠產生出優異的結果。之後，我們探討卷積核尺寸(kernel size)對於模型效能的影響， 3x3 卷積核尺寸效能比大卷積核尺寸的效能還佳，所以我們以 3x3 的卷積核尺寸作為下一階段實驗的基礎。最後我們定義一個廣的模型，透過 NAS 來找到他的最佳子網路，當作重新訓練之網路架構，最佳子網路均有著更佳的準確率，以及更少的參數量。總結而言，我們測試了多組資料集，以及多種不同的隨機種子，證明所提出的算法能夠自動搜索出在特定資料集中，比基準AlexNet 架構更輕量且準確的最佳架構。

In recent years, deep learning has achieved outstanding results in many fields, such as computer vision, natural language processing. However, the process of designing an exceptional neural network architecture highly requires the domain knowledge and repeated experiments, which keeps the many researchers away from the machine learning approach. In order to simplify this process, neural architecture search (NAS) has become popular recently. We first delve into the significance of initialization methods on the weights of AlexNet and the kernel size in the convolutional layer, which provides us a further insight to define our search space. The initialization method of uniform distribution with a proper upper bound and lower bound is able to produce exceptional results. Then we investigate the importance of kernel size. Notably, the adoption of a 3×3 kernel size operation as the sole operation within a cell proves to be highly prominent. Last but not least, our proposed algorithm can automatically search optimal architecture which is more light-weight and accurate on specific datasets from ImageNet than the baseline AlexNet architecture.