Skip to content

About

World In-Context Learning for Agentic Robot Manipulation

Topics

Resources

Contributing

Stars

301 stars

Watchers

32 watching

Forks

Repository files navigation

RoboScript

World In-Context Learning for Agentic Robot Manipulation

Heng Fang1 · Zehui Guo1 · Yuehan Zhang1 · Mingyang Du1,2 · Dingkang Liang1,2 · Xiang Bai1
1Huazhong University of Science and Technology    2ArcheOne AI

Paper on arXiv Project page License: Apache-2.0 Python 3.10+

Learn a manipulation task in context, not in weights:
one demonstration, rehearsal in a world, no gradient updates.

Learning with a world, the world as the robot's terminal, and success rates on RoboTwin 2.0

RoboScript is an agent framework for robot manipulation without a trained action policy. A frozen foundation model starts from one demonstration, rehearses the task in a world such as a simulator, and writes what it learns into a Task Memory in its context instead of its weights. We call this World In-Context Learning (WICL).

📣 News

  • [2026/11] We release the paper and the code.

🔍 Overview

The four stages of RoboScript: skills, task learning, deployment and reflection

Four stages: build reusable skills from demonstrations, learn a Task Memory from one demonstration, deploy the frozen Memory on new scenes with only what a real robot can sense, and improve by reflecting on past evaluations. Currently validated on dual-arm tabletop manipulation.

📈 Results

On RoboTwin 2.0 (50 bimanual tasks), RoboScript achieves 83.4% mean success with one demonstration per task and no gradient updates, compared to 61.7% for the strongest trained policy using 50 demonstrations per task. Rehearsal happens only on clean scenes; randomized scenes see almost no drop (83.6% → 83.2%). Results on additional benchmarks are in progress.

Full results table

Mean success rate over all 50 RoboTwin 2.0 tasks for trained policies, agents with trained policies and RoboScript

† Randomized setting only, five seeds per task; N is the unreported number of demonstrations used to fine-tune the frozen base.

Works on unseen scenes. Episodes from 20 tasks on clean and randomized scenes; randomized scenes are never rehearsed. Clips are sped up.

Twenty RoboTwin 2.0 tasks completed in simulation, on clean and randomized scenes

Runs on a real robot. Memories rehearsed on a simulated ALOHA run on a dual-arm PiPER through parameter remapping, without rehearsing again. Clips are sped up.

Twelve episodes of four tasks on a dual-arm PiPER robot: click bell, grab roller, move playing card away and stack bowls

🧱 Infrastructure

Infrastructure of RoboScript: foundation models behind a gateway, the agent plane, the world, the control plane, services, operations and plugins

Each component runs as its own service. Methods, benchmarks, embodiments and skillsets are plugins.

Layer What it does
Model gateway Routes LLM calls to any provider, with session pinning and failover
Agent plane Agent loop, tool Hub, skills — limited to what a real robot can sense
Control plane Runs the learning protocol, holds ground truth. Only the official check decides success
Services Motion planning (cuRobo), perception (GroundingDINO, SAM 3), blob storage

Add a benchmark, skillset, real robot or other extension without changing the core.

🛠️ Getting Started

Run one episode on the built-in Mock benchmark. It needs no simulator, GPU or model.

git clone https://github.com/ArcheOne-AI/RoboScript.git
cd RoboScript
uv sync --all-packages
cd examples/tutorials
uv run python -m quickstart_mock.run

Then continue with the tutorials, pick a benchmark from the table below, or read the documentation. The vocabulary defines terms such as skillset, brain and recipe. To change the code, see CONTRIBUTING.md.

🧩 What's Included

Framework core, three learning methods (wicl, rsi, skill_builder), the vision service, and a real-robot package. Pre-1.0; interfaces may change.

Benchmark Status
Mock Built in, no simulator needed. Used by the tutorials.
RoboTwin 2.0 Full pipeline with official evaluation.
BEHAVIOR Integration in development.

To add another benchmark, see add-a-sim-benchmark.

🗓️ TODO

  • Real-robot deployment code.

📝 Citation

@article{fang2026roboscript,
  title   = {World In-Context Learning for Agentic Robot Manipulation},
  author  = {Fang, Heng and Guo, Zehui and Zhang, Yuehan and Du, Mingyang and Liang, Dingkang and Bai, Xiang},
  journal = {arXiv preprint arXiv:2611.12885},
  year    = {2026}
}

📜 License

Apache License 2.0. See LICENSE and NOTICE, which lists adapted third-party code. Benchmark simulators and their assets are not part of this repository and come under their own licenses.

About

World In-Context Learning for Agentic Robot Manipulation

Topics

Resources

Contributing

Stars

301 stars

Watchers

32 watching

Forks

Contributors

Languages