用於聯邦式學習的高效通訊 Wyner-Ziv 壓縮方式

Abstract

聯邦式學習是一種分散式機器學習，其訓練資料因隱私性只由邊緣設備各自擁有，而參數服務器負責提供訓練模型以及協調參數更新，由於本地參數更新需經過有限容量的上行通道傳送到參數服務器，會佔據過多頻寬導致通訊效益下降，為了突破有限容量瓶頸，目前有許多研究對使用不同壓縮方式來達到更低的碼率。本文額外利用參數更新彼此在時間或空間上的相關性，在參數服務器中產生帶有本地參數更新訊息量的側訊息，此架構為只有解碼器具有測訊息的源編碼問題 (SCSI Problem)，使得壓縮碼率進一步減少。本文採用 TCQ 對一半的邊緣設備做壓縮並生成測訊息，另一半使用餘式 WZC 架構再透過測訊息進行解碼，由於參數更新的收斂會使得數值不斷改變，我們提出預先縮放的 WZC 架構並對源訊息和測訊息之間的差做預測，使得每次跌代不需要重新設計 code book，模擬結果顯示，使用 TCQ 加 WZC 相較於全部使用 TCQ 可以在更低的碼率下達到相同的影像辨識精確度。

Federated learning is a form of decentralized machine learning, where training data is held by edge devices due to privacy concerns, and the parameter server is only responsible for providing training models and coordinating parameter updates. Due to the limited capacity of the uplink channel required for transmitting locally updated parameters to the parameter server, excessive bandwidth occupation leads to a decrease in communication efficiency. To overcome this bottleneck of limited capacity, many studies have explored the use of different compression methods to achieve lower rate. This paper additionally utilizes the temporal and spatial correlations of parameter updates to generate side information in the parameter server. This architecture presents a source coding problem with only the decoder having side information (SCSI Problem), further reducing compression bit rates. The paper adopts TCQ to compress half of the edge devices and generate side information, while the remaining half uses residual WZC architecture for decoding through side information. As parameter updates converge and numerical values continuously change, we propose a pre-scaling WZC architecture and predict the difference between source and side information to avoid the need for redesigning codebooks at each iteration. Simulation results demonstrate that using TCQ with WZC achieves the same image recognition accuracy at lower rate compared to using TCQ alone.