EPoG: Integrated Exploration and Sequential Manipulation on Scene Graph

TL;DR

A bilevel task planner that maintains an LLM-augmented belief graph over partially observed scenes, computing action sequences via graph edit operations and replanning on the fly. 91.3% success and 36.1% shorter travel distance across 46 ProcThor-10k household scenes, validated on a real mobile manipulator.

Video

Task 1 — Plan with unknown object state. The robot is asked to place a cabbage and a cup into a basket on a coffee table. The global planner updates its belief graph via an LLM to locate the missing cup and complete the task. Task 2 — Exception handling. The robot retrieves a tissue box from a closet and places it on a coaster. The local planner handles inaccessibility (opening a closed closet), blocking, and potential collisions (temporarily moving a cup and a tea can out of the way).

Abstract

In partially known environments, robots must combine exploration to gather information with task planning for efficient execution. To address this challenge, we propose EPoG, an Exploration-based sequential manipulation Planning framework on Scene Graphs. EPoG integrates a graph-based global planner with an LLM-based situated local planner, continuously updating a belief graph using observations and LLM predictions to represent known and unknown objects. Action sequences are generated by computing graph edit operations between the goal and belief graphs, ordered by temporal dependencies and movement costs. This approach seamlessly combines exploration and sequential manipulation planning. In ablation studies across 46 realistic household scenes and 5 long-horizon daily object transportation tasks, EPoG achieved a success rate of 91.3%, reducing travel distance by 36.1% on average. A physical mobile manipulator successfully executed complex tasks in unknown and dynamic environments, demonstrating EPoG's potential for real-world applications.

Introduction

Real-world deployment violates the two comfortable assumptions of existing scene-graph planners: complete prior knowledge, and a static environment. A robot has to interleave information gathering with task planning, which introduces three coupled challenges — locating task-relevant objects under partial observability, trading off exploration and manipulation cost, and reasoning under uncertainty to produce situated, executable plans.

An example illustrating the challenges of integrating exploration and sequential manipulation: (a) Robots must prioritize potential exploration locations and balance exploration with manipulation tasks to execute efficiently. (b) The robot needs to engage in situated planning to handle unexpected situations.

To address these challenges, we propose EPoG, a framework that integrates Exploration and sequential manipulation Planning on Scene Graphs. EPoG adopts a bilevel planning architecture for partially observed environments, leveraging pretrained LLMs for informed exploration and situated replanning.

Overview of the proposed EPoG framework. In the global planner, the belief graph is updated by estimating the locations of target objects present in the goal graph but missing in the initial graph, using new observations and LLM predictions. The action plan is then obtained by performing the topological sort on the graph edit operations between the belief and the goal graph. In the local planner, the LLM generates a situated action sequence to handle exceptions encountered during execution.

Preliminary

We adopt a graph-based scene representation that models 3D perception with predicate-like attributes, so symbolic action reasoning is grounded in geometric structure while remaining queryable by planning algorithms.

An example illustration of an indoor scene.

Local Replanning

The global planner incrementally builds a belief graph from onboard observations and LLM-based predictions, computes graph edit operations against the goal graph, and generates a candidate action sequence via topological sorting. The robot executes this sequence while continuously updating the belief graph — exploration and manipulation happen in a single closed loop.

When execution deviates from the nominal plan, the local planner invokes an LLM for situated replanning. The exceptions we consider in this work:

Examples of exceptions in motion planning. (a) Blocking: The robot must avoid collisions with other objects while placing or picking up objects. (b) Inaccessibility: Successful object retrieval or placement within a container requires the container to be opened first. (c) Collision: The robot must ensure that placed objects do not collide with the environment. (d) Instability: The robot must maintain the stability of stacked objects during manipulation; for example, retrieving a book beneath a cup may cause instability.

Simulation & Real-World Experiments

We evaluate EPoG on 5 long-horizon object-transport tasks across 46 household scenes from ProcThor-10k. Ablations show that the proposed formulation naturally couples exploration and manipulation, reducing total execution effort; EPoG outperforms purely LLM-based planners on long-horizon tasks; and LLM-guided heuristics plus the low-level planner reduce manual design effort while improving execution efficiency.

Illustrations of the robot's paths for two tasks, highlighting key locations and actions.

Real-world experiments. (a) The robot must place three objects (with one unknown object) on a coffee table. (b) The robot must retrieve a tissue box from a closet and place it on a coaster, first removing a cup and a tea can that obstruct task execution. Object states in the closet and on the coffee table are initially unknown.

EPoG: Integrated Exploration and Sequential Manipulation on Scene Graph with LLM-based Situated Replanning