具有核化斯坦因差異和分數匹配的變分推論

Abstract

近年來，近似難以處理的概率分佈已成為一個重要的研究課題。針對這一問題，目前主要有兩類技術方法：馬爾可夫鏈蒙特卡羅（MCMC）和變分推論（VI）。然而，MCMC 方法計算成本高，且對於大規模數據集並不實用。相比之下，VI 方法受到越來越多的關注。然而，傳統的變分方法通過最小化目標分佈與一個相對簡單的參數化分佈（變分族）之間的 Kullback–Leibler （KL）散度，會受到變分族限制和有過於簡化的問題。本研究提出了一種基於核化斯坦因差異的變分推論方法（KSD-VI），旨在緩解傳統 VI 的限制，更進一步結合分數匹配原則（KSDSM-VI）以解決過度簡化的問題。

Recently, approximating intractable probability distributions has become an important problem. Two main classes of techniques address this issue: Markov chain Monte Carlo (MCMC) and variational inference (VI). However, MCMC methods are computationally expensive and impractical for large datasets. In contrast, VI has received increasing interest. Traditional variational inference, minimizing the Kullback-Leibler divergence (KL divergence) between a relatively simple parametric family (variational family) and the target distribution, suffers from limitations on variational family and the risk of oversimplification. This research proposes a new variational inference method based on kernelized Stein discrepancy (KSD-VI) to overcome the restrictions of traditional VI. Furthermore, it integrates the score matching principle (KSDSM-VI) to address oversimplification.