Overview
KEA Planner is a hybrid planning stack built on top of ROS 2 and PlanSys2. It keeps the PlanSys2 model of domains, problems, actions, and execution, and adds LLM/VLM-based oversight to detect failures and propose recovery actions.
Any PlanSys2 example can generally run under KEA with minimal changes, as long as you provide a domain and problem file that PlanSys2 can parse.
What KEA adds on top of PlanSys2
- A plan oversight agent (LLM) that observes execution and emits structured recovery commands.
- An optional LLM-backed PlanSys2 plan solver plugin.
- A VLM-based anomaly detection agent for visual safety checks.
- Tooling for PDDL generation and large-scale testing.
Architecture at a glance
KEA extends PlanSys2 with a controller and agent node that sit in the execution loop. The controller mediates between PlanSys2 and the agent, collects execution status, and applies recovery commands from the LLM.
Package layout
This repository is a ROS 2 workspace containing multiple packages. The most important ones are:
| Package | Purpose |
|---|---|
kea_planner_core | Core KEA nodes, launch files, and parameter profiles (LLM/VLM). |
kea_plansys2_solvers | Custom PlanSys2 planner plugin that can call an LLM. |
kea_msgs | ROS 2 message and service definitions used by the controller and agents. |
kea_plan_gen | Utilities for generating PDDL prompts and prototype LLM training. |
kea_testing | Image generation and batch testing utilities for evaluation. |
kea_examples/kea_moleworks | Excavation-style planning example with simulated failure modes. |
kea_examples/kea_rock_breaking | Rock-breaking example using PlanSys2 BT actions. |
Top-level folders of note:
docs/: mdBook sources for this documentation.docker/: Dockerfile and docker-compose definitions for ROS + LLM/VLM services.data/andartifacts/: generated by dvc, datasets and output artifacts (not required for core use).
Current scope and limitations
- The plan-generation training pipeline is a prototype and ships with a debug reward function.