針對通用圖形處理器伺服器的大規模模型高效訓練之計算排程最佳化

Abstract

在大型模型的訓練中，商品GPU伺服器（commodity GPU servers）通常依賴管線平行化（pipeline parallelism）將模型分散到多個 GPU 上，以降低單個GPU的記憶體峰值使用量。然而，當模型規模超過多個GPU的總記憶體容量時，模型的訓練仍會面對到記憶體容量限制的問題，此時使用主記憶體作為外部記憶體是一種有效的解決方案。將最先進的置換（swapping）技術應用於管線平行化時，即使透過這些技術，模型參數仍無法及時從主記憶體傳輸到GPU，進而導致延遲和管線氣泡（pipeline bubbles），嚴重影響訓練效能。為了解決此問題，本研究提出了一種新的機制，能夠將佇列中的計算動態地安排到置換操作產生的閒置中。透過重疊計算與資料傳輸，此機制有效減少計算資源閒置時間並提升整體資源利用率。實驗結果顯示，所提出的機制在訓練吞吐量方面可實現達1.18倍的提升。

Large-scale model training on commodity GPU servers relies on pipeline parallelism to distribute models across multiple GPUs and reduce peak memory usage. However, when the model size exceeds the combined memory capacity of the GPUs, leveraging host memory as external memory serves as an effective solution to overcome memory limitations. When applying state-of-the-art swapping techniques to pipeline parallelism, despite the assistant of these techniques, model parameters still failed to transfer from host memory to GPUs in time, results in I/O delays and pipeline bubbles, significantly degrading training performance. To alleviate this performance degradation, this research proposes a mechanism that dynamically schedules queued computations into the idle intervals introduced by swapping operations. By overlapping computation with data transfer, the mechanism reduces idling and enhances overall resource utilization. Experimental results demonstrate throughput improvements of up to 1.18× compared to baseline.