利用參數伺服器在深度學習中應用多樣化的通訊最佳化

Abstract

深度學習已經成為最有希望解決人工智慧問題的方法之一。有效率地訓練一個大規模深度學習模型非常具有挑戰性，一個廣泛使用的加速方法是利用集中式的參數伺服器將計算分散到多臺工作節點上。為了克服因工作節點與參數伺服器交換資料而造成的通訊成本，通常會採用三種最佳化方法：資料放置、一致性控制和壓縮。 在本文中，我們提出了模組化參數伺服器架構，其具有多個容易覆蓋的關鍵元件。這讓開發者可以輕鬆地將最佳化技術整合至訓練過程中，而不必在現有系統中使用特殊的方式實作。通過這個平臺，使用者能分析不同技術組合，並開發新的最佳化演算法。實驗結果顯示，和 Google 的分散式 Tensorflow 相比，藉由結合多種最佳化技巧，基於模組化參數伺服器的分散式訓練系統在運算上能夠達到接近線性的加速，並在減少一半訓練時間的同時保持收斂的準確度。

Deep learning has become one of the most promising approaches to solve the artificial intelligence problems. Training large-scale deep learning models efficiently is challenging. A widely used approach to accelerate the training process is by distributing the computation across multiple nodes with a centralized parameter server. To overcome the communication overhead caused by exchanging information between workers and the parameter server, three types of optimization methods are adopted -- data placement, consistency control, and compression. In this paper, we proposed modularized parameter server, an architecture composed of key components that can be overridden without much effort. This allows developers to easily incorporate optimization techniques in the training process instead of using ad-hoc ways in existing systems. With this platform, the users can analyze different combinations of techniques and develop new optimization algorithms. The experiment results show that, compared with Google's distributed Tensorflow, our distributed training system based on the proposed modularized parameter server can achieve near-linear speedup for computing and reduce half of the training time by combining multiple optimization techniques while maintaining the convergent accuracy.