分析神經網路推論於Nvidia Jetson AGX Xavier之能源並以自適應頻率調節優化能源效率

Abstract

近年來，人工智慧在科技發展中扮演重要的角色，而在這其中最為重要的是神經網路。許多網站以及機器都會倚賴神經網路來增進其功能，為人類帶更多的方便。然而，因為神經網路的深度，造成劇增的計算量以及過多的能量耗損。 在神經網路推論的階段，GPU計算以及DRAM資料搬移在Nvidia Jetson AGX Xavier上佔據了約莫67%的總能源耗損。因此，這篇論文將以調節GPU以及DRAM的頻率來達到能源效率。此篇論文中，我們使用多層感知器來當作我們「頻率預測器」的模型架構去預測最節能的頻率設定。再者，我們在推論目標神經網路前（offline），對其做層層分析。依據不同種的網路層，我們提供訓練完成的「頻率預測器」，頻率預測器會根據網路層不同的設定預測出最有能源效率的頻率。之後，我們將所有網路層中預測的結果，取最高的頻率當作上限值，取最低的當作下限值，最後作為目標神經網路推論時，GPU以及DRAM可以浮動的區間。 在Nvidia Jetson AGX Xavier上，此機制在總能量耗損中達到平均20.0%的減少，其中，GPU達到平均29.8%，DRAM達到平均23.7%的能量耗損減少。

Artificial intelligence has played an important role in technology development in recent years, and the essential items in this path are neural networks. Recently, more and more devices and websites have counted on neural networks to improve their functionalities, bringing much more convenience to people in this era. However, thanks to the deeper depth of the recent neural network, the model inference will have massive computations and excessive energy consumption. During the inference stage, GPU computation and DRAM memory access occupy approximately 67% of overall inference energy consumption on Nvidia Jetson AGX Xavier. Therefore, we focus on the frequency modulations of GPU and DRAM in this work. Moreover, in this work, we utilized the multi-layer perceptron models (MLP models) as the frequency setting predictors to predict the most energy-efficient frequency setting. First, we offline layer-wisely analyze the target model, providing frequency setting predictors to determine frequency settings that can achieve maximum energy efficiency. Afterward, among all predicted frequency settings, we select the maximum frequency setting as the upper bound and the minimum frequency setting as the lower bound. Eventually, we set this limitation as the range in which GPU and DRAM frequency can fluctuate during the target model inference stage. This work achieves an average 20.0% overall energy consumption reduction and an average 29.8% and 23.7% energy consumption reduction for GPU and DRAM, respectively, on Nvidia Jetson AGX Xavier during the target model inference stage.