在結構剪枝中保留模型權重的方法

Abstract

深度神經網路在各種應用中展現了優異的成果，但其對空間與運算資源的需求，對於資源受限的設備構成了重大的挑戰。模型剪枝（model pruning）透過移除較不重要的權重來解決此問題，通常會再進行微調（fine-tuning）以恢復性能。儘管已有研究顯示，在剪枝後更新權重可以提升結果，但這些方法往往依賴特定的剪枝準則，限制了其通用性。本研究提出了一種名為 WeightCompression 的有效且具彈性的方式，在微調前，將被剪除的權重重新分配到各層中保留的權重上。這樣的重分配機制為微調提供了更佳的初始化，使得本方法僅需一次剪枝步驟，即可達到與傳統多次剪枝方法相當的效能。與現有方法相比，WeightCompression 可將剪枝流程加速 3.27 倍至 12.58 倍，同時維持競爭力的準確率。此外，該方法對剪枝準則具有高度的適應性，並可有效擴展至大型模型。實驗結果顯示，WeightCompression 是一個靈活且高效的剪枝後權重更新方法。

Deep neural networks have shown promising results across a wide range of applications, but the requirements on space and computation present significant challenges for deployment on resource-constrained devices. Model pruning addresses this issue by removing less important weights, typically followed by fine-tuning to recover the performance. While prior work has shown that updating weights after pruning can improve results, these approaches are often tied to specific pruning criteria, limiting their generality. This work proposes WeightCompression, an effective and flexible method that redistributes pruned weights into the remaining ones within each layer before fine-tuning. This redistribution provides a better initialization for fine-tuning, enabling the proposed method to match the performance of iterative pruning approaches using only a single pruning step. Compared to existing methods, WeightCompression accelerates the pruning process by 3.27x to 12.58x while maintaining competitive accuracy. Last but not the least, it is agnostic to the choice of pruning criterion and scales well to large models. The results suggest that WeightCompression can be a flexible and efficient framework for post-pruning weight updates.