應用於強化學習之高能效神經網路處理器晶片

Abstract

強化學習已被廣泛應用於各種領域，尤其在智慧機器人領域已展現優異成果。本論文提出了一強化學習處理器，藉由完整的估測功能與訓練推論的平行處理，達成高效強化學習運算。本論文藉由演算法和硬體的共同設計優化，使強化學習的計算複雜度降低90%。並利用強化學習資料在空間和時間上的相關性，提出一種資料編碼方法以降低位元遮罩的資料大小達65%。處理器的硬體架構完整支援資料稀疏性與估測，並同時維持高硬體利用率。該處理器並整合訓練與推論的平行處理，以進一步降低執行時間。總體而言，本論文提出的處理器使執行強化學習演算法的延遲時間減少了89%。該處理器以40nm CMOS製程設計與製造，晶片達到209TOPS/W 能量效率以及2341GOPS/mm^2的面積效率。與過往文獻中的最佳設計相比，本晶片的能源效率和面積效率分別提高了7.1倍與7.3倍。

Reinforcement learning has shown remarkable performance in the field of autonomous vehicles and intelligent robots. This paper presents a reinforcement learning processor with full speculation exploitation and parallel processing for inference and training. By the co-design of algorithm and hardware, the computational complexity is significantly reduced by 90%. A data encoding scheme leveraging both spatial and temporal data correlations is proposed to reduce the bitmask size by 65%. The architecture is designed to perform operations with full support sparsity and speculation while maintaining high hardware utilization. The chip is designed to support parallel processing of inference and training to reduce latency. Overall, the proposed processor achieves an 89% reduction in processing time. Fabricated in 40-nm CMOS, the proposed reinforcement learning processor delivers a 209TOPS/W energy efficiency and a 2341GOPS/mm2 area efficiency. This work achieves 7.1× and 7.3× higher energy efficiency and area efficiency, respectively, than the state of the arts.