使用穩健最佳化作為深度強化學習指引 – 以化學物料生產排程作為應用案例

Abstract

在本文中，我們提出了一種新的強化學習訓練框架，專門用於動態排程單階段多產品化學反應器。該方法透過最佳化方法的合作，將策略梯度收斂時所遇到的局部最優挑戰減輕，優化了強化學習模型目標值以及它的穩定性。除此之外，我們進一步透過將穩健最佳化作為指導引擎，提高模型對參數誤差的穩健性，促進了在需求與生產效率不穩定的情況下所能採用的較保守之決策風格。實驗結果證明了我們所提出方法的有效性，與簡單的演員-評論家方法相比，在同樣的參數設計下多個指標上都有所提高。 本研究的研究貢獻在深度強化學習與最佳化方法上的整合有所突破，也透過本框架提供新的強化學習訓練框架，可以應用於其他領域問題之中。

In this paper, we present a novel guiding framework for the training phase of Reinforcement Learning (RL) models, specifically tailored for dynamic scheduling of a single-stage multi-product chemical reactor. Our approach addresses the challenge of local optima in policy gradient methods by integrating optimization methods, enhancing both the objective value and stability of the RL model. We further enhance the robustness of our model against parameter distortions by incorporating robust optimization as a guiding engine, promoting a more conservative decision-making style. Our experimental results demonstrate the efficacy of our approach, with several metrics compared to simple Actor-Critic methods. Our work thus serves as an advancement in the integration of Deep Reinforcement Learning and optimization methods, hopefully opening new avenues for research and application in dynamic scheduling and beyond.