基於不確定性與專家互動改進深度強化學習

Abstract

近期研究顯示專家的演示有助於強化式學習的過程，然而在大多數 的情況下，專家需要花費大量的時間才能搜集足夠多的演示資料，本 篇論文的目標即是希望透過讓人工智能主動提出需要演示的情境來減 少專家花費於演示的時間。我們提出了一套能夠隨著訓練過程調整並 且適用於各種不同類型任務的發問策略，根據最近觀察到的情境的不 確定性來決定需不需要對當前的情境發問。我們採用兩種不同方式來 評估對於當前情境的不確定性，第一種是根據自助式採樣 Q 學習於不 同採樣之間的發散程度，第二種則是根據雜訊 Q 學習中預測值的變異 程度。發問得到的演示資料會透過同時最佳化監督式和時序差分的損 失函數來學習。在四種不同任務的實驗結果顯示，我們提出的方法不 但能夠達到超越專家的表現，相較於其他被動式的方法，在相同數量 的演示資料時也有更高的學習效率。

Recent research has shown reinforcement learning (RL) can benefit from expert demonstration. However it usually takes considerable efforts to obtain enough demonstration. In this work, we aims to improve reinforcement learn- ing more efficiently regarding the demonstration efforts by allowing the RL agent query for demonstration actively during training. We propose Asking DQN, a novel online query strategy which can adapt to different distribution shift during training and effective on various kinds of tasks. The query strat- egy decides whether to ask according to the uncertainty of states it observed in recent steps. We estimate the uncertainty in two ways, including the di- vergence of bootstrapped DQN and the variance of noisy DQN. The acquired expert demonstration is then utilized by optimizing supervised max-margin loss in addition to temporal difference loss. The empirical result shows our proposed method is able to outperform the expert, and learn faster compared to other passive methods with the same amount of demonstration in four dif- ferent tasks.