互補標籤學習之代理損失架構研究

Abstract

在弱監督學習領域中，無偏風險估計量(unbiased risk estimator)在訓練資料與測試資料分佈不一致的情況下常被用於訓練分類器。然而，在複雜的模型中無偏風險估計量經常發生過擬合現象，例如深度學習的情境。本論文針對弱監督學習中的互補標籤學習問題(learning with complementary labels)進行探討，分析無偏風險估計量過擬合的成因。透過一系列實驗分析，我們發現即使無偏風險估計量能產生無偏梯度，但產生的梯度方向與目標梯度方向有顯著差距，且變異數值大。本論文提出一個新的代理損失架構(Surrogate Complementary Loss，簡稱SCL)，針對互補標籤學習問題使用最小似然估計設計損失函數，有效降低梯度變異並使梯度方向更接近目標梯度方向。實驗數據顯示SCL方法有效降低過擬合現象，並在多個資料集達成更高的分類準確度。

In weakly supervised learning, unbiased risk estimator (URE) is a powerful tool for training classifiers when training and test data are drawn from different distributions. Nevertheless, UREs lead to overfitting in many problem settings when the models are complex like deep networks. In this thesis, we investigate reasons for such overfitting by studying a weakly supervised problem called learning with complementary labels. We argue the quality of gradient estimation matters more in risk minimization. Theoretically, we show that a URE gives an unbiased gradient estimator (UGE). Practically, however, UGEs may suffer from huge variance, which causes empirical gradients to be usually far away from true gradients during minimization. To this end, we propose a novel surrogate complementary loss (SCL) framework that trades zero bias with reduced variance and makes empirical gradients more aligned with true gradients in the direction. Thanks to this characteristic, SCL successfully mitigates the overfitting issue and improves URE-based methods.