The robot follows the actions; the objects do not
Given the current frame and a sequence of robot actions, a robotic world model predicts the future video. Rendering the robot into the input makes the arm follow the actions, but the objects in contact still go wrong: they collapse, or respond inconsistently with the action.
DROID test clips, Wan2.2. Videos play at half speed.
Separate dynamics from rendering
An action, numeric or rendered, specifies where the robot goes but not how the scene responds. A single video model must then infer the interaction dynamics while also rendering appearance.
Following code world models [1, 2, 3], we split the two jobs. A simulator evolves the world state and draws it into a coarse proxy video; the video model paints the final frames from it. For robot manipulation, the central design choice is the proxy.
What should the proxy be?
The proxy must represent any object, including deformable and articulated ones. It must be obtainable from a single real observation, and its dynamics must be predictable under robot actions.
Needs mass, friction, object meshes and a solver for each material, all of which are hard to obtain in a real scene.
A rule written as code can move the object held between the fingers, but cannot express pushing, dragging or deformation.
From one RGB-D frame, points represent any object, rigid or deformable. Their dynamics are learned from real robot data.
Simulate in 3D, render back to 2D
The simulator decides what happens; the video model decides how it looks. They meet in a pixel-aligned proxy.
Same first frame, same actions, different conditions
The left column shows the simulated point trajectories (purple to yellow over time). Without point dynamics, the arm moves correctly but the object does not; with the point dynamics proxy, the object responds as in the real video. Videos play at half speed.
DROID
Real robot
Two WidowX AI arms, one overhead RGB-D camera, 15 tasks; Wan2.2 and PointWorld fine-tuned on our recordings.
Editing the simulation steers the video
What the numbers say
- Data
- DROID: 326K training clips; 1,515 test clips from 900 held-out episodes in 13 labs. No wrist camera, no text.
- Renderers
- Wan2.2 (DiT) and Ctrl-World (SVD UNet), each trained with four conditions on the same data and budget.
- Metric
- Object motion error: object points are tracked in the generated and the real video; the pixel distance between the tracks. Lower is better.
Better point dynamics, more faithful world models
We decouple a robotic world model into a simulator that predicts how 3D scene points respond to the robot and a renderer that turns the prediction into video, connected by a pixel-aligned point dynamics proxy. Explicit point dynamics make object interaction more faithful, and the video follows the simulator. Progress in 3D point dynamics, learned or written as code, can thus carry over to robotic world models without changing how they render.
- Y. Chen, G. Lin, C. Zhang. Code World Model: Coding Agent as World Brain. arXiv:2608.25927, 2026.
- Z. Fang, Y. Shao, K. Zhu, et al. Code Plans, Diffusion Renders: Open-Ended Generative World Modeling. arXiv:2609.26458, 2026.
- Z.-H. Huang, G. Lin, J. Lin, et al. Programmable World Model. arXiv:2609.10540, 2026.
- W. Huang, Y.-W. Chao, A. Mousavian, et al. PointWorld: Scaling 3D World Models for In-The-Wild Robotic Manipulation. CVPR 2026.
BibTeX
@misc{an2026pointdynamicsproxyrobotic,
title = {Point Dynamics as Proxy for Robotic World Models},
author = {Dong An and Tianyu Liu and Haoran Ding and Wen Yang and Anqing Duan and
Zhengzhong Liu and Eric P. Xing and Guangyi Liu},
year = {2026},
eprint = {2611.25946},
archivePrefix = {arXiv},
primaryClass = {cs.CV},
url = {https://arxiv.org/abs/2611.25946}
}