高效能二值化權重神經網路之設計與實現

Abstract

近年來，神經網路逐漸被開發用於各領域之任務，並且達到了與人類相當甚至有所超越的表現。這些成果使得基於神經網路的商品逐漸出現在我們的生活中，如何在計算資源有限的終端裝置進行神經網路推論運算顯得十分重要。因此，在維持模型表現下，以降低推論運算的計算成本成為了我們的研究重點。 本文採用了各種技巧，在不進行人工修改模型架構的前提下，基於量化感知訓練將權重進行二值化。在各種模型、應用中，我們的二值化訓練流程都能達到不錯的準確率。 在此之上，我們發現了二值化訓練中存在不穩定的震盪情況。針對這個問題，我們以Hopfield模型作為理論基礎，在二值化神經網路量化感知訓練中加入遲滯。使用我們以統計值作為閾值之遲滯二值化，訓練時的震盪情形得以解決。在各種模型與任務上都能有進一步的提升。 接著我們以TPC-NAS演算法進行神經網路架構搜尋。在數分鐘內的計算時間下進行自動架構修改，在相同的計算量限制下找到比人工設計的模型更好的架構，進一步提高二值化後模型之準確率。 最後我們以邏輯閘級模擬進行能效分析，相較於8W8A之精度，進行二值化後的1W4A能夠節省3.57倍的面積與3.07倍的功耗。並且在乘加運算上提出了基於OAI之近似方法，並以近似感知訓練初步證明其近似準確率。

In recent years, neural networks have gradually been applied to various tasks, achieving performances comparable to or surpassing those of humans. These advancements have led to the emergence of neural network-based products in our daily lives. The challenge of performing neural network inference computations on resource-constrained terminal devices has become crucial. Therefore, our research focuses on maintaining model performance while reducing the computational cost of inference. This thesis employs various techniques to binarize weights based on quantization-aware training without manually modifying the model architecture. The binarization training process demonstrates good accuracy across various models and applications. Furthermore, we identify oscillations during binarization training. To address this issue, we incorporate hysteresis into binarized quantization-aware training using the theoretical foundation of the Hopfield model. We propose using statistical values as hysteresis thresholds, which resolves oscillation issues during training and leads to further improvements in various models and tasks. Next, we utilize the TPC-NAS algorithm for neural network architecture search. Through automated architecture modifications within a few minutes of computational time, the algorithm identifies architectures superior to manually designed models under the same computational constraints, further enhancing the accuracy of binarized models. Finally, we conduct energy efficiency analysis using logic gate-level simulation. Compared to the precision of 8 bits with 8-bit activations, the binarized model with 1-bit weights and 4-bit activations achieves a 3.57 times reduction in area and a 3.07 times reduction in power consumption. Additionally, we propose an approximate method based on OAI for multiplicationaccumulate operations and preliminarily demonstrate its approximate accuracy through approximate-aware training.