DEXTER-LLM

Dynamic and Explainable Coordination of Multi-Robot Systems in Unknown Environments via Large Language Models

1Peking University, 2Duke Kunshan University
Nerfies: Deformable Neural Radiance Fields

DEXTER-LLM introduces a dynamic task planning framework for heterogeneous multi-robot systems in unknown environments, integrating mission comprehension, online subtask generation via LLMs, optimal subtask scheduling, and multi-rate adaptation to enable real-time coordination and adaptability.

Abstract

Online coordination of multi-robot systems in open and unknown environments faces significant challenges, particularly when semantic features detected during operation dynamically trigger new tasks. Recent large language model (LLMs)-based approaches for scene reasoning and planning primarily focus on one-shot, end-to-end solutions in known environments, lacking both dynamic adaptation capabilities for online operation and explainability in the processes of planning. To address these issues, a novel framework (DEXTER-LLM) for dynamic task planning in unknown environments, integrates four modules: (i) a mission comprehension module that resolves partial ordering of tasks specified by natural languages or linear temporal logic formulas (LTL); (ii) an online subtask generator based on LLMs that improves the accuracy and explainability of task decomposition via multi-stage reasoning; (iii) an optimal subtask assigner and scheduler that allocates subtasks to robots via search-based optimization; and (iv) a dynamic adaptation and human-in-the-loop verification module that implements multi-rate, event-based updates for both subtasks and their assignments, to cope with new features and tasks detected online. The framework effectively combines LLMs’ open-world reasoning capabilities with the optimality of model-based assignment methods, simultaneously addressing the critical issue of online adaptability and explainability. Experimental evaluations demonstrate exceptional performances, with 100% success rates across all scenarios, 160 tasks and 480 subtasks completed on average (3 times the baselines), 62% less queries to LLMs during adaptation, and superior plan quality (2 times higher) for compound tasks.

Overview Framework

Overview Framework

Overview of the proposed framework, which includes four core modules: the abstraction of partially-ordered tasks from natural languages or LTL formulas (left); the generation of plausible strategies to decompose each task into executable subtasks via LLMs (middle); the subtask assignment and scheduling via model-based optimization (right); and the human-in-the-loop online adaptation and verification (bottom).

Online Adaptation

Sequence Chart

Multi-level online adaptation to 7 types of events, and the human in-the-loop verification after each module.

Experiment

Scenario-I: Chemical Plant

Scenario-II: Arctic Region