發展多智能體深度強化學習降雨逕流模式

Abstract

降雨逕流模式為現今水文運用中水工結構物設計及水文相關研究最為重要的工具之一。依據其運作方式可以將其分為物理模式、概念型模式以及經驗型模式三類。以物理模式而言，地理條件的假設以及大量的公式運算，往往限制其實用性。概念型模式則運用經驗公式取代部分過於複雜的運算過程，改善部分物理模式可能遇到之問題。而以經驗型而言，傾向於輸入及目標值間運用統計或機器學習方法尋找彼此的關聯性，然而其黑盒特性也經常受到質疑。為了合理運用各個模式間的優勢並且改善其缺點，有效結合物理模式及經驗型模式的概念是必須的。因此，本研究中提出多智能體深度強化學習(Multi-Agents system Deep Reinforcement Learning)模式，可有效模擬水文過程並且大幅度減少物理及經驗公式運用之模式。於此模式中，單一水單元之行為及各個水單元的互動行為，分別可以交由深度強化學習及多智能體系統處理。 為了驗證模式之準確度和可行性，本研究中以一設計流域，建置本研究中提出的多智能體深度強化學習模式並與物理解析模式及通用的套裝模式SOBEK兩模式比較。為符合真實流域特性，設計流域採用臺灣石門流域之參數作為設計標準；設計之降雨事件採用中央氣象局所訂下之大雨、豪雨、豪大雨及兩種颱風極端事件模擬。本模式可量化強化學習的學習次數對於流量歷線的影響，結果顯示，隨著學習次數增加，模擬之流量歷線會逐漸貼近物理解析模式之流量，並且於大雨、豪雨條件甚至是極端事件，均優於SOBEK模式。 本研究以最新的深度學習理論建置混合物理概念與機器學習方法之降雨逕流模式，可了解水單元於水文過程中之行為，並減少於模式中使用之物理及經驗公式，研究成果將成為深度學習及強化學習運用於水文領域之基石。

Rainfall-runoff models are commonly classified into three categories, namely: physical-based, conceptual and empirical models to implement in hydraulic structure designing and researches. Physical-based models are restricted by geographic information and over-complex operations. Empirical models adopt several empirical equations to construct the correlation of inputs and outputs at outflow control section, but empirical models sometimes are distrusted due to the black box characteristics. A reasonable way to retain the advantages and improve the drawbacks of both physical-based and empirical models is constructing a rainfall-runoff model with novel algorithms. Therefore, this study presents a novel Multi-Agents system Deep Reinforcement Learning (MAS-DRL) that is capable of understanding the hydrological process through water units and reducing physical operations. The behavior of a water unit and interaction between water units can be simulated by Deep Reinforcement Learning and the Multi-Agents system, respectively. Designed cases are conducted to clearly demonstrate the advantage of the MAS-DRL model. To approximate the actual basin, the topographies are settled based on the Shihmen Reservoir basin (Taiwan), and rainfall events are designed due to the precipitation standards that were settled by Central Weather Bureau, respectively. The Eagleson Dynamic Wave solution (Eagleson solution) and SOBEK are constructed as comparison models. The results reveal that with increasing rounds of training, the simulated discharge resulting from the MAS-DRL model gradually closes to the results simulated from Eagleson solution regardless of the rainfall event. In summary, this study proposes a novel MAS-DRL model, which simulates the behavior of a water unit and interaction between water units to understand the hydrological process and reproduces physical operation. The proposed modeling technique could be help to future use of DRL in hydrology.