具有共享及獨特特徵和優先權經驗回放的多任務強化學習

Abstract

多任務強化學習已成為一個具有挑戰性的問題，旨在減少強化學習的計算成本並利用任務之間的共享特徵來提高個別任務的性能。然而，一個關鍵挑戰在於確定應該在任務之間共享哪些特徵以及如何保留區分每個任務的獨特特徵。這個挑戰常常導致任務性能不平衡的問題，其中某些任務可能主導學習過程，而其他 任務則被忽視。在本文中，我們提出了一種新方法，稱為共享­獨特特徵和任務感知的優先經驗重放，以提高訓練穩定性並有效利用共享和獨特特徵。我們引入了一種簡單而有效的任務特定嵌入方法，以保留每個任務的獨特特徵，以減輕任務性能不平衡的潛在問題。此外，我們將任務感知設置引入到優先經驗重放算法中，以適應多任務訓練並增強訓練的穩定性。我們的方法在 Meta­World 的資料測試集中實現了最先進的平均成功率，同時在所有任務上保持穩定的性能，避免了任務性能不平衡的問題。結果證明了我們的方法在應對多任務強化學習挑戰方面的有效性。

Multi-task reinforcement learning (MTRL) has emerged as a challenging problem to reduce the computational cost of reinforcement learning and leverage shared features among tasks to improve the performance of individual tasks. However, a key challenge lies in determining which features should be shared across tasks and how to preserve the unique features that differentiate each task. This challenge often leads to the problem of task performance imbalance, where certain tasks may dominate the learning process while others are neglected. In this paper, we propose a novel approach called shared-unique features along with task-aware prioritized experience replay to improve training stability and leverage shared and unique features effectively. We incorporate a simple yet effective task-specific embeddings to preserve the unique features of each task to mitigate the potential problem of task performance imbalance. Additionally, we introduce task-aware settings to the prioritized experience replay (PER) algorithm to accommodate multi-task training and enhancing training stability. Our approach achieves state-of-the-art average success rates on the Meta-World benchmark, while maintaining stable performance across all tasks, avoiding task performance imbalance issues. The results demonstrate the effectiveness of our method in addressing the challenges of MTRL.