利用遺傳程式設計從專家示範自動推論任務子結構

Abstract

為了處理層次性和組合性的決策問題，智能代理人需要任務結構和子任務規則的領域知識表示，以進行規劃和推理。先前的方法通常假設預定義的子任務存在，因為在缺乏領域知識的狀況下確定子任務具有困難性。因此，我們提出了一個框架，從專家示範中自動歸納推論子任務以解決複雜任務。該框架涵蓋了經典規劃、深度強化學習和演化計算，過程包括為歸納符號規則、從目標構建任務結構，以及基於任務結構提供內在獎勵。我們利用基因程式設計進行符號規則推論，在此過程中，規則模型的選擇反映了先驗領域知識的效果規則。我們在兩個環境中評估了該框架，包括 Minecraft 環境，並證明它提升了深度強化學習代理的學習效率。此外，我們還展示了該框架能通過組合任務結構和推論新規則，展現在任務和技能層面的通用性。本研究對於整合框架作為解決層次性現實世界問題的認知架構提供了深入的觀點。

To deal with hierarchical and compositional decision-making problems, intelligent agents necessitate domain knowledge representation on task structures and subtask rules for planning and reasoning. Previous approaches often rely on strong assumptions about pre-defined subtasks due to the difficulty of determining subtasks lacking domain knowledge. Therefore, we propose a framework that automatically induces subtasks from expert demonstrations to solve complex tasks. The framework encompasses planning, deep reinforcement learning (DRL), and evolutionary computation, and the procedure involves inducing symbolic rules, constructing task structures from goals, and providing intrinsic rewards based on task structures. We utilize genetic programming for symbolic rule induction, where the selection of the rule model reflects prior domain knowledge of effect rules. We evaluate the framework in two environments, including the Minecraft environment, and demonstrate that it improves the performance of DRL agents. In addition, we also demonstrate the generalizability in task and skill level by composing the task structure and inducing the new rules. This research contributes insights into integrated frameworks as a cognitive architecture to address hierarchical real-world problems.