關於 L2 正則化線性分類與迴歸的權重密度

Abstract

對於傳統的線性模型而言，廣泛使用L2正則項的模型通常被認為是稠密的。因此，關於在什麼時候L2正則化線性模型可以得到稀疏解的問題很少受到關注。在這篇論文中，我們嚴謹地證明了對於L2正則化的支持向量分類/迴歸問題，當資料的特徵向量具有稀疏性時，其理論上的最優解可以為稀疏的。令人意外的是，我們觀察到某些最佳化方法無法保留這項稀疏性，反而產生稠密的數值解，導致不必要的儲存空間。我們透過詳細的分析來解釋這個現象。特別值得注意的是，我們新奇地發現某些投影梯度法在解決對偶問題時，會自然地產生比其他最佳化方法更稀疏的數值解。透過使用這些適合的算法，可以將模型的儲存空間減少多達50%，對於大規模的工業應用來說具有相當顯著的效益。

For traditional linear models with the widely used L2-regularizer, it is often assumed that the resulting models are dense. As a result, little attention has been paid to when the optimal solution for an L2-regularized problem can actually be sparse. In this work, we rigorously prove that for L2-regularized support vector classification/regression, the theoretical optimum can indeed be sparse when the data have sparse feature values. Surprisingly, we observe that some optimization methods fail to preserve this sparsity and instead produce fully dense numerical solutions, leading to unnecessary storage overhead. We explain this phenomenon through detailed analysis. In particular, we novelly show that certain projected gradient methods for solving the dual problem naturally yields sparser numerical solutions compared to other optimization algorithms. By applying suitable algorithms that preserve numerical sparsity, the storage can be reduced by up to 50%, which is highly advantageous for large-scale industrial applications.