即時性多模態大型語言模型代理人架構於機械手臂對話機器人一般性取放任務及自訂義按摩任務之應用

Abstract

本論文提出一個即時性，具語音控制能力之多模態大型語言模型（Multimodal Large Language Model, MLLM）框架，該框架結合大型語言模型（Large Language Models, LLMs）、視覺語言模型（Vision Language Models, VLMs）、語音轉文字與文字轉語音之語音模組，以及檢索增強生成（Retrieval-Augmented Generation, RAG），建構為一個統一的代理人式（agent-based）機械手臂控制系統。系統核心設計引入中央主對話代理（Central Main Chat-Agent）與路由代理（Router Agent），其中路由代理負責將任務分派至背景模組，確保前端語音互動過程不中斷，並實現明確的前端–後端分離。此設計保證了框架的通用性，使前端流程可廣泛應用於多種機械手臂任務；同時透過任務平行化與可自訂程式碼生成，並搭配專用的機械手臂連線函式庫，實現可擴充性。 在一般取放任務中，本框架展示了如何透過 LLM 推理直接控制機械手臂，並利用多模態模組的串接鏈結（concatenated chaining）來實現感知、推理與執行的連續流程。在機械按摩任務中，系統進一步展現了語音模組如何與中央主對話代理及路由代理協同運作，實現前端與後端分離的互動架構，確保流暢且即時的語音控制。本案例同時驗證了 MLLM 在處理更複雜與可自訂的人本服務型應用上的潛力。 傳統工業機械手臂雖在重複性、預編程任務中表現優異，但缺乏面對開放式互動或個人化服務的適應性。透過將多模態人工智慧直接嵌入控制迴路，本框架突破此限制，使系統能在結構化與服務導向環境中實現即時感知、推理與動作。整體系統以對話代理的形式運作，支援自然語言對話、語音互動及視覺理解。基於 LLM 的聊天模組負責協調任務，並呼叫專門模組執行檢索、推理與軌跡生成；RAG 流程確保回應的準確性與語境相關性，而記憶機制則維持跨多輪互動的一致性。在感知面向上，本框架結合 YOLO 物件偵測與 Transformer 式 VLM，實現即時物件辨識、分割與情境解讀。由自訂的 Python 整合器負責連接高階 AI 推理與低階機械手臂控制，確保流暢銜接。 實驗驗證結果顯示，本框架能有效將自然語言指令轉換為可執行的機械手臂軌跡，能即時因應視覺輸入進行調整，並維持與使用者的連貫互動。本研究的貢獻在於展示如何將 LLM 推理、多模態對齊與代理式協作整合進入機械手臂控制系統，實現從抽象推理到物理行動的橋接，並為可擴充、互動式與以使用者為中心的智慧機器人平台鋪路。

This thesis proposes a Realtime Voice-Controlled-Enabled Multimodal Large Language Model (MLLM) Framework that integrates large language models (LLMs), vision–language models (VLMs), speech-to-text and text-to-speech voice modules, and retrieval-augmented generation (RAG) into a unified agent-based control system for robotic arms. At its core, the system introduces a Central Main Chat-Agent coupled with a Router Agent that delegates tasks to background modules, ensuring uninterrupted voice interaction through a clear frontend–backend separation. This design guarantees generalizability, as the frontend pipeline can be seamlessly applied to a wide range of robotic tasks. Scalability is achieved by enabling task parallelism and customizable code generation through the router agent, while dedicated libraries ensure reliable connections to robotic hardware. In the general pick-and-place task, the framework demonstrates how LLM reasoning can directly control robots, while a concatenated chaining of multimodal modules enables sequential perception, reasoning, and execution for structured manipulation tasks. In the robot massage task, the system showcases how the voice module functions in conjunction with a central main chat-agent and router-agent, implementing a frontend–backend separation framework that ensures smooth, real-time interactions. This case further demonstrates the potential of MLLMs to handle more complex, customizable, and human-centered robotic applications. Traditional industrial robots excel in repetitive, pre-programmed routines but lack adaptability for open-ended interaction or personalized services. By embedding multimodal AI directly into the control loop, the proposed framework addresses these limitations, enabling real-time perception, reasoning, and action across both structured and service-oriented environments. The system functions as a conversational agent supporting natural language dialogue, voice interaction, and visual grounding. An LLM-based chatbot orchestrates operations by invoking specialized modules for retrieval, reasoning, and trajectory generation. RAG pipelines maintain factual accuracy and contextual relevance, while memory mechanisms ensure continuity across extended interactions. On the perception side, the framework integrates YOLO-based vision modules with Transformer-based VLMs, enabling real-time object detection, segmentation, and contextual interpretation. A custom Python integrator bridges high-level reasoning with low-level robotic execution, ensuring seamless control. Experimental validation demonstrates the framework’s capacity to translate natural language instructions into executable robotic trajectories, adapt dynamically to visual input, and sustain coherent user interactions. The contributions of this work lie in showing how LLM reasoning, multimodal grounding, and agent-based orchestration can be unified within a robotic control system. This integration illustrates the feasibility of bridging abstract AI reasoning with physical action, paving the way for scalable, interactive, and user-centered robotic platforms.