基於規則與強化學習方法的城市污水系統即時控制研究

Abstract

都市排水的主要目標是排除雨水與污水，並透過地下管路輸送至適當地點處理或排放。都市排水管路設計上，雨水流經雨水下水道並排放至河道中，而污水流經污水下水道並輸送至污水處理廠處理後，排放至海中。本研究之研究區域臺北市即是使用此雨污分離的設計。然而，本應完全分離的雨污下水道系統，因管線老舊滲入或雨污管線錯接等問題，導致降雨期間污水管線內的水量暴增，應對此現象即須仰賴污水抽水站操作人員制定雨天與晴天的抽水策略並進一步監視與即時控制。 為了增加決策應變時間，並減少人為操作的誤差，引入物聯網技術、自動控制、即時監控與演算等需求逐漸提升，因此，本研究提出即時決策模式，可依即時觀測資料預測未來25小時的濕井水位和抽水機啟閉建議以提供抽水站人員參考。 即時決策模式先以暴雨管理模型模擬臺北市污水下水道系統，並以控制演算方法，提出最適合的操作策略。控制演算方法包括規則導向控制(Rule-based Control, RBC)與深度強化學習(Deep Reinforcement Learning, DRL)，其中前者為基於現行抽水站操作策略制定，後者是基於設施操作規則獎勵懲罰方法訓練而出。為量化兩種模式的控制能力，本研究制定了多項評鑑指標，涵蓋系統控制目標(水位控制)、設施運行限制(抽水機與閘門操作)以及電力消耗，以評估兩方法在現行抽水站目標與限制條件下的表現，並比較控制能力於不同目標的優劣。此外，本研究進一步探討輸入資料誤差對模擬的影響，即模擬不完美輸入情境，檢測兩方法在系統輸入數據或模擬誤差累積對結果的不確定性影響，以比較微小擾動影響下的穩定性。 晴天結果顯示水位控制與電力成本評鑑指標之間呈現權衡關係，DRL雖呈現較佳的節電效果，但是在水位控制上表現略差於RBC。雨天結果顯示由於水位控制與抽水機操作之間具權衡之關係，RBC因為受到規則嚴格控制，在多項目標分歧時以水位控制為優先，因此，RBC模式在水位控制上有較好的表現；而DRL模式由於為多目標之間的權衡，因此，即便多項目標分歧時仍盡可能地減少違反抽水機的操作規則，在遵守抽水機規則目標中即表現較好。在不完美輸入模擬中，由於RBC模式規則為二元式判斷且規則的高度複雜等原因，不僅相較DRL機率式模式更易犯下錯誤的判斷，在進行錯誤行為後的補救行為也較遲鈍，因此RBC模擬的變異較大，在污水量大時較易於違反水位控制的基礎限制。 本研究針對晴天與雨天各別提出RBC和DRL模式，生成操作策略並將其輸入至污水模擬系統。結果顯示在晴天模擬下，由於RBC具備較佳且穩定的水位控制能力，因此選擇RBC作為晴天的污水控制模式；而在雨天模擬下，DRL展現出更穩定的控制能力，因此選擇DRL作為雨天的污水控制模式。

The primary goal of urban drainage systems is to discharge stormwater and sewage by transporting them through underground pipelines to appropriate locations for treatment or discharge. In the design of urban drainage pipelines, stormwater flows through stormwater drainage pipelines and is discharged into rivers, while sewage flows through sewage pipelines, is transported to sewage treatment plants for treatment, and subsequently discharged into the river. The study area, Taipei City, employs this separate drainage system design for stormwater and sewage. However, due to issues such as aging infrastructure causing infiltration or misconnected pipelines, the sewage pipelines which supposedly separate from stormwater experience a significant increase in sewage volume during rainfall events. Addressing this issue requires reliance on sewage pumping station operators to formulate pumping strategies for both wet and dry days and enhance monitoring and real-time control. To increase decision-making response time and reduce human operational errors, the demand for integrating IoT (Internet of Things) technology, automatic control, real-time monitoring, and computational algorithms has been steadily rising. Therefore, this study proposes a real-time decision-making model capable of predicting wet well water levels and providing pump operation recommendations for the next 25 hours based on real-time observational data, offering valuable guidance for pumping station personnel. The real-time decision-making model first simulates sewer system of Taipei City using the Storm Water Management Model (SWMM) and applies control algorithms to propose the most optimal operations. The control algorithms include Rule-Based Control (RBC) and Deep Reinforcement Learning (DRL). RBC is formulated based on the current operational strategies of pumping stations, while DRL is trained using a reward-penalty mechanism derived from facility operation rules. To quantify the control capabilities of the two approaches, this study establishes multiple evaluation metrics, encompassing system control objectives (water level regulation), facility operational constraints (pump and gate operations), and electricity consumption. These metrics are used to assess the performance of both methods under the current goals and constraints of pumping stations and to compare their effectiveness across different objectives. Furthermore, this study investigates the impact of input data errors on the simulation by simulating imperfect input scenarios. This analysis examines how accumulated errors in system input data or simulation affect the results, comparing the stability of the two methods under small perturbations and uncertainties. The results for dry days show a trade-off between water level control and electricity cost metrics. While DRL demonstrates better performance in electricity cost metric, its water level control is slightly inferior to that of RBC. In wet days, the results indicate a trade-off between water level control and pump operation. RBC, due to its strict rule-based control, prioritizes water level regulation when multiple objectives diverge, leading to better performance in water level control. In contrast, DRL, as a multi-objective method, aims to minimize violations of pump operation rules even when multiple objectives diverge, thus performing better in adhering to pump operation rules. In the simulation with imperfect inputs, RBC, with its binary decision rules and high complexity, is more prone to making erroneous judgments compared to the probabilistic DRL model. Furthermore, RBC is slower to recover from errors in behavior, leading to larger variations in the simulation. When wastewater volume is high, RBC is more likely to violate the fundamental water level control constraints. This study proposes RBC and DRL models for dry and wet days, respectively, to generate operational strategies and input them into the sewer simulation system. The results show that in dry days, RBC is selected as the wastewater control mode due to its superior and more stable water level control capabilities. In the wet days, DRL is chosen as the wastewater control mode due to its more stable control performance.