基於多方安全計算實現長序列 Transformer 解碼器模型之高效率安全推論

Abstract

多方安全計算（Secure Multi-Party Computation, SMPC）可在不洩漏資料內容的情況下進行隱私保護推論，然而在 SMPC 設定下，針對 Transformer decoder 模型進行長序列生成的研究仍相當有限且缺乏系統性探討。本文提出 SecLoT，一個旨在提升安全 Transformer decoder 推論效率的框架，透過同時解決演算法層面與系統層面的限制，推進長序列生成在 SMPC 環境中的可行性。我們將固定矩陣大小的 causal mask self-attention 機制引入 SMPC 設定中，使自回歸解碼過程得以在不依賴安全 softmax 的情況下進行，並達成每個生成 token 幾乎固定的計算時間與通訊成本。此外，我們擴充 CrypTen 編譯器以支援更複雜的模型架構，提出無輸入範圍限制且高效率的除法與倒數平方根演算法，並設計一個具正確性保證的截斷協定，以消除既有實作中可能發生的隨機錯誤。在基於 decoder-only Transformer 的 MNIST 圖像生成實驗中，結果顯示 SecLoT 相較於以 softmax 為基礎的方法，能有效降低計算時間與通訊開銷，且此優勢隨著生成序列長度的增加而更加明顯。

Secure Multi-Party Computation (SMPC) enables privacy-preserving inference, yet long-sequence generation with Transformer decoder models has not been systematically explored under SMPC. In this work, we present SecLoT, a framework that advances secure Transformer decoder inference by addressing both algorithmic inefficiencies and system-level limitations. We adapt a fixed-size causal mask self-attention formulation to the SMPC setting, enabling autoregressive decoding without secure softmax and achieving nearly constant computation time and communication cost per generated token. In addition, we extend the CrypTen compiler to support more complex model architectures, propose domain-unrestricted and efficient algorithms for division and inverse square root, and introduce a correct truncation protocol that eliminates probabilistic errors in existing implementations. Experiments on decoder-only Transformer-based MNIST image generation demonstrate that SecLoT significantly reduces computation time and communication overhead compared to softmax-based baselines, particularly as sequence length increases.