用於學習分佈式可泛化知識之貝氏混合神經網路

Abstract

在機器學習的領域中，估計模型在未知資料上的效能一直是一個重要的挑戰。一個經常被採用的方法是，假設訓練和測試資料是取樣自同一機率分佈。然而在實際應用中，訓練和測試資料分布之間往往存在偏移使得假設不成立。在本文中，我們假設訓練數據來自多個不同的分佈，並且共享與任務相關的知識。我們進而提出了一種新型模型: 貝氏混合神經網路，用於學習對非因果關係的分佈偏移有韌性的共享知識。通過提出的變分推理方法，我們提出的神經網路可以很容易地被應用於多模態和不變式學習的問題中。在這兩種問題中，訓練和測試分佈不一定會被假定為相似的分佈。以多模態學習來說，我們的神經網路可以在沒有明確監督訊號的情況下，從資料中解構共享和模態特定的資訊。同樣地，在不變式學習中，我們提出的神經網路能夠以無監督的方式學會丟棄與目標無因果關係的特徵。與現有的解決方案相比，我們提出的深度學習模型在多模態和不變學習的任務上均實現了最好的性能和效率。

Estimating model performance on unseen data is a fundamental challenge in machine learning. A commonly adopted approach is to assume training and testing data are sampled from the same distribution; however, in real-world applications, a distribution shift between training and testing data often exists. In this paper, we assume training data are sampled from multiple and distinct distributions which share task-relevant knowledge. We then propose a novel model, Bayesian Mixture Neural Network (BMNN), for learning the shared knowledge that can be robust to non-causal distribution shift. With the proposed variational inference method, BMNN can be easily employed in multimodal and invariant learning problems, where the training and testing distributions are not necessarily assumed to be aligned. In multimodal learning, we show that BMNN can disentangle shared and modality-specific information without explicit supervision. Similarly, in invariant learning, BMNN learns to discard non-causal features in an unsupervised manner. Compared with existing solutions, BMNN achieves state-of-the-art performance and efficiency on both multimodal and invariant learning tasks.