擴散模型之連續客製化

Abstract

在實際應用中，如何有效率的不斷更新擴散模型是個難題。因此，我們提出了一個創新的學習策略『萃取概念神經元』（Concept Neuron Selection）。這個簡單且有效的方法，可以在連續學習的條件下對擴散模型實現客製化。CNS 在擴散模型中辨認和目標概念相關的神經元。為了同時避免災難性遺忘以及保留零樣本文字轉圖像的生成能力，CNS 以增量方式微調概念神經元，並共同保留先前學到的概念知識。在現實世界資料集的評估下顯示，CNS 在只需最少參數調整的情況下達到最高的效能，並且無論在單一或多概念客製化任務中皆優於以往方法。CNS同時實現無融合操作，有效降低持續客製化所需的記憶體儲存與處理時間。

Updating diffusion models in an incremental setting would be practical in real-world applications yet computationally challenging. We present a novel learning strategy of Concept Neuron Selection, a simple yet effective approach to perform personalization in a continual learning scheme. CNS uniquely identifies neurons in diffusion models that are closely related to the target concepts. In order to mitigate catastrophic forgetting problems while preserving zero-shot text-to-image generation ability, CNS finetunes concept neurons in an incremental manner and jointly preserves knowledge learned of previous concepts. Evaluation of real-world datasets demonstrates that CNS achieves state-of-the-art performance with minimal parameter adjustments, outperforming previous methods in both single and multi-concept personalization works. CNS also achieves fusion-free operation, reducing memory storage and processing time for continual personalization.