基於高階合成之矩陣乘法硬體加速及其在視覺轉換器之應用

Abstract

Vision Transformer (ViT) 由於具備強大的全域特徵擷取與長距依存關係建模能力，在影像辨識任務中展現出卓越的效能。然而，其龐大的運算量遠高於傳統卷積神經網路，特別是在線性層 (linear layer) 中的大量矩陣乘法，使其在計算與記憶體資源受限的邊緣裝置上難以高效率部署。 為解決上述問題，本研究採用 高階合成 (High-Level Synthesis, HLS) 技術於 可程式化邏輯閘陣列 (Field-Programmable Gate Array, FPGA) 平台上實現矩陣乘法加速。主要挑戰在於同時兼顧運算效率、記憶體頻寬與硬體資源配置的平衡。為此，本研究提出一個具層次化結構的優化方法，分為三個層級：(1) 運算層級 (Computing-based) 採用區塊矩陣乘法 (Block Matrix Multiplication, BMM) 結構以提升資料重用並降低 DRAM 存取延遲；(2) 編譯指令層級 (Pragma-based) 透過 loop pipelining、loop unrolling 與 array reshaping 技術以提升平行運算效率；(3) 系統層級 (System-level) 結合 AXI-Stream 通訊協定 與 Direct Memory Access (DMA) 機制以減少 PS 與 PL 間的資料傳輸開銷。 在 Xilinx ZCU104 平台上的實作結果顯示，本研究提出之多層次優化框架能有效提升運算效率並維持硬體資源的使用平衡。透過 PYNQ 平台的軟硬體整合部署，完整驗證了本研究架構的可行性與加速效果，證明階層式的 HLS 優化設計能顯著提升 Vision Transformer 模型於 FPGA 邊緣運算環境中的效能與能源效率。

The vision Transformer (ViT) has demonstrated outstanding performance in visual recognition tasks due to its ability to global contextual relationships. However, its computational complexity is significantly higher than that of conventional convolutional neural networks, particularly in the fully connected linear layers. The extensive matrix multiplications within the Query–Key–Value (QKV) projections make it difficult to deploy ViT models efficiently on edge devices with limited computing and memory resources. To address this challenge, this study employs High-Level Synthesis (HLS) to accelerate matrix multiplication on FPGAs. The main design challenge lies in balancing hardware resource utilization, memory bandwidth, and computation latency. To overcome these issues, a hierarchical optimization framework is proposed, consisting of three levels: (1) computing-based optimization using Block Matrix Multiplication to enhance data reuse and reduce DRAM access; (2) pragma-based optimization leveraging loop pipelining, unrolling, and array reshaping to increase parallelism and throughput; and (3) system-level optimization integrating AXI-Stream and Direct Memory Access (DMA) to minimize data transfer overhead between the processing system and programmable logic. The proposed framework was implemented on a Xilinx ZCU104 platform and deployed through the PYNQ environment. Experimental results confirm that the multilevel optimization approach effectively enhances computational efficiency and resource utilization, providing a practical and scalable solution for accelerating vision Transformer models on FPGA-based edge devices.