半監督式學習中利用四元損失函數進行一致性正規化

Abstract

在半監督式學習（Semi-supervised learning）中，一致性正規化（Consistency regularization）是一個有效且常見的方法。它的目的在於，讓一個機器學習模型，其輸出不會因為對模型進行微小擾動而大受影響。一致性正規化往往透過新增一項損失函數來實現，該項損失函數會試圖最小化兩個模型輸出之間的距離，其中一個輸出是模型未經過擾動的輸出，另一個輸出是模型經過擾動後的輸出。然而，在過去的損失函數中，並未考量到這兩個模型輸出的信心程度。在這篇論文裡，我們提出一個新的損失函數稱作四元損失函數（Quadruplet loss），此損失函數能同時考慮模型的一致性以及模型輸出的質量。具體而言，我們在損失函數中透過計算模型輸出的熵（Entropy），內建了一個考量兩個模型輸出質量的機制，該機制會動態調整損失函數使得其鼓勵模型做出低熵的輸出。因此，模型的參數可以被較可靠地更新。我們將此新的四元損失函數實作在Π-model和virtual adversarial training這兩個最先進的半監督式學習方法上，並使用兩個基準資料集CIFAR-10和SVHN作測試。結果顯示出四元損失函數確實在這兩個資料集上增進了這兩個半監督式學習的成效。除此之外，模型訓練的穩定度也有所提升，在驗證資料（Validation data）缺乏以致難以做模型選擇（Model selection）時，這項性質便相當重要。我們也發現使用四元損失函數能讓這兩個半監督式學習方法在訓練資料只有極少量有標籤的情況下，減緩成效下降的程度。

Consistency regularization is often employed to enforce that a small perturbation of a semi-supervised learning model does not seriously affect the output of the model. This is achieved by introducing an additional loss (or cost) function to minimize the distance between the outputs of the model before and after the perturbation. However, the confidence level of the two outputs is not considered for the minimization. In this paper, we propose a new loss function called quadruplet loss to take both model consistency and output quality into consideration at the same time. Specifically, an entropy measurement of the output quality is built into the proposed loss function, and an adjustment is made dynamically to encourage low-entropy outputs in each training iteration. As a result, the model parameters are reliably optimized. We test the quadruplet loss on two state-of-the-art semi-supervised learning methods, the Π-model and the virtual adversarial training, using CIFAR-10 and SVHN as benchmark datasets. The results show that the quadruplet loss indeed improves the performance of these two methods and the stability of model training, which is critical when model selection is infeasible because only a limited amount of validation data are available. Moreover, the quadruplet loss prevents models from dramatic performance degradation when the amount of labeled data is small.