使用自適應性離散動作空間的基於模型強化學習

Abstract

連續動作控制是強化學習中其中一個主要的研究議題。在連續控制任務中，智能體通過從連續動作空間中決定精確的最佳動作值以採取接下來的行動，這相對於具有離散動作空間的決策任務更爲複雜且具挑戰性。因此，連續動作空間離散化是減少應對連續控制任務複雜性的其中一種可行直觀方式。然而，固定的離散化連續動作空間可能會在不同的離散程度遇到不同的問題。本研究提出了一種適應性連續動作空間離散化方法，在初始階段離散化後的連續動作空間集會較小且間距較稀疏，在智能體訓練中期時，此離散化連續動作空間集合會進行擴展，透過增加集合內的元素來獲得更緊密的離散化連續動作空間集合。我們更近一步一致性和適應性離散化連續動作取樣方法應用於最先進的基於模型的強化學習（model-based reinforcement learning）演算法，並在多個連續控制任務上進行評估，並在大部分任務中和原先方法相比取得較優或相近的結果。除此之外，我們提出的方法在計算時間效率上也優於原始的連續動作取樣方法。

Continuous control has emerged as a prominent area of focus within reinforcement learning. The agent takes action by determining the action value from a continuous action space for continuous control tasks, which is more challenging than decision-making tasks with discrete action space. Hence, continuous action space discretization is an intuitive approach to reduce the complexity of dealing with continuous control tasks. However, consistent action space discretization may encounter different problems depending on different fixed granularity. The present study introduces an adaptive continuous action space discretization approach, initializing with coarse discretization and then expanding the discretized action space set with denser granularity. We also apply both consistent and adaptive discretization methods to the state-of-the-art model-based reinforcement learning algorithm and benchmark several continuous control tasks. Our method achieves better or comparable results over the original action sampling method with superior computation time efficiency.