LLM-GROP: Visually Grounded Robot Task and Motion Planning with Large Language Models

Xiaohan Zhang^*1, Yan Ding^*1,5,6, Yohei Hayamizu^*1, Zainab Altaweel^*1, Yifeng Zhu², Yuke Zhu², Peter Stone^{2, 3}, Chris Paxton⁴, Shiqi Zhang¹

¹The State University of New York at Binghamton    ²The University of Texas at Austin ³Sony AI ⁴Hello Robot ⁵Shanghai AI Laboratory ⁶OneStar Robotics

📄 Paper   |   Published at The International Journal of Robotics Research (2025)

Abstract

Task planning and motion planning are two of the most important problems in robotics, where task planning methods help robots achieve high-level goals and motion planning methods maintain low-level feasibility. Task and motion planning (TAMP) methods interleave the two processes of task planning and motion planning to ensure goal achievement and motion feasibility. Within the TAMP context, we are concerned with the mobile manipulation (MoMa) of multiple objects, where it is necessary to interleave actions for navigation and manipulation. In particular, we aim to compute where and how each object should be placed given underspecified goals, such as “set up dinner table with a fork, knife and plate.” We leverage the rich common sense knowledge from large language models (LLMs), e.g., about how tableware is organized, to facilitate both task-level and motion-level planning. In addition, we use computer vision methods to learn a strategy for selecting base positions to facilitate MoMa behaviors, where the base position corresponds to the robot’s “footprint” and orientation in its operating space. Altogether, this article provides a principled TAMP framework for MoMa tasks that accounts for common sense about object rearrangement and is adaptive to novel situations that include many objects that need to be moved. We performed quantitative experiments in both real-world settings and simulated environments. We evaluated the success rate and efficiency in completing long-horizon object rearrangement tasks. While the robot completed 84.4% real-world object rearrangement trials, subjective human evaluations indicated that the robot’s performance is still lower than experienced human waiters.

Overview Figure

Real Robot Experiment

We begin with a real-world demonstration to validate the proposed approach. The mobile manipulator used in this demonstration features a wheeled base for navigation and a 6-DOF robotic arm for performing manipulation tasks. Experimental Setup: We designed our real-world experiment to demonstrate that LLM-GROP can effectively handle a variety of scenarios. Tasks 4, 7, and 9 from Table 1 were selected for these demonstrations. The environment consisted of three tables, with objects initially placed on the left and right tables, and the robot positioned randomly. The robot repeated each task 15 times: during the first five repetitions, no obstacles were present (Easy Environment); in the next five, a chair was positioned on the upper side of the table (Hard Environment - Chair on Top); and in the final five, a chair was positioned on the lower side of the table (Hard Environment - Chair on Bottom).