Point Dynamics as Proxy for Robotic World Models

Robot world models follow the arm but not the objects it touches. We let a learned 3D point simulator predict how the scene responds, and let the video model render it.

Problem

The robot follows the actions; the objects do not

Given the current frame and a sequence of robot actions, a robotic world model predicts the future video. Rendering the robot into the input makes the arm follow the actions, but the objects in contact still go wrong: they collapse, or respond inconsistently with the action.

robot renderreal future
✗ Drag bread. The spatula moves away, but the bread stays on the plate.
robot renderreal future
✗ Release cup. The gripper opens, but the released cup rises with it.

DROID test clips, Wan2.2. Videos play at half speed.

Insight

Separate dynamics from rendering

Action-only conditioning: the video model must jointly infer interaction dynamics and render appearance. Ours: robot render plus point dynamics, so the video model mainly renders the simulated point dynamics.

An action, numeric or rendered, specifies where the robot goes but not how the scene responds. A single video model must then infer the interaction dynamics while also rendering appearance.

Following code world models [1, 2, 3], we split the two jobs. A simulator evolves the world state and draws it into a coarse proxy video; the video model paints the final frames from it. For robot manipulation, the central design choice is the proxy.

What should the proxy be?

The proxy must represent any object, including deformable and articulated ones. It must be obtainable from a single real observation, and its dynamics must be predictable under robot actions.

✗ Physics simulator

Needs mass, friction, object meshes and a solver for each material, all of which are hard to obtain in a real scene.

✗ Hand-written rules

A rule written as code can move the object held between the fingers, but cannot express pushing, dragging or deformation.

✓ 3D point dynamics

From one RGB-D frame, points represent any object, rigid or deformable. Their dynamics are learned from real robot data.

Method

Simulate in 3D, render back to 2D

The simulator decides what happens; the video model decides how it looks. They meet in a pixel-aligned proxy.

1 · Lift to 3DThe first RGB-D frame becomes scene points; forward kinematics turns the actions into robot points.
2 · SimulateA learned point dynamics model, PointWorld [4], predicts how every scene point responds to the robot.
3 · ProxyThe simulated points are drawn back into pixel-aligned videos: robot render, warp of the first frame, and point tracks.
4 · RenderA video model reads the proxy through its own VAE and paints the real arm, lighting and contact.
Plug-inOnly the video model's first layer is widened, with new weights starting at zero.
ModularAny simulator that predicts point displacements can be swapped in; we use two renderers of different architectures.
CheapSimulating and drawing the proxy takes 0.93 s per clip, next to 17.2 s for generating the video.
Results

Same first frame, same actions, different conditions

The left column shows the simulated point trajectories (purple to yellow over time). Without point dynamics, the arm moves correctly but the object does not; with the point dynamics proxy, the object responds as in the real video. Videos play at half speed.

DROID

simulated point dynamicsnumeric actionrobot renderrobot render + depthpoint dynamics proxy (ours)real future
Drag bread. With the numeric action or the robot render, the bread stays on the plate as the spatula moves away.
Release cup. Without point dynamics, the released cup rises with the open gripper.
Lift a towel. Without point dynamics, the towel bunches up.

Real robot

Two WidowX AI arms, one overhead RGB-D camera, 15 tasks; Wan2.2 and PointWorld fine-tuned on our recordings.

simulated point dynamicsnumeric actionrobot renderrobot render + depthpoint dynamics proxy (ours)real future
Unfold a bag. Without point dynamics, the bag is not unfolded.
Hand over a bag. Without point dynamics, the bag hangs from the wrong arm or appears twice.
Hand over a pillow. With the proxy, the pillow’s shape and position stay closest to the real handover.

Editing the simulation steers the video

uneditedcloth motion frozencloth motion doubled
Top: the simulated motion of the cloth, edited at test time. Bottom: the video generated from it, with the robot and the rest of the scene unchanged (a successful example; DROID, Wan2.2).
Experiments

What the numbers say

Data
DROID: 326K training clips; 1,515 test clips from 900 held-out episodes in 13 labs. No wrist camera, no text.
Renderers
Wan2.2 (DiT) and Ctrl-World (SVD UNet), each trained with four conditions on the same data and budget.
Metric
Object motion error: object points are tracked in the generated and the real video; the pixel distance between the tracks. Lower is better.
Object motion error on the DROID test set: point dynamics proxy 4.69 (Wan2.2) and 4.97 (Ctrl-World) pixels, against 5.13 and 5.30 for the robot render.
Point dynamics make interaction more faithful. Against the robot render, the proxy lowers object motion error by 8.6% on Wan2.2 and 6.2% on Ctrl-World, without degrading image quality.
Change of object motion error against the robot render: about -2% on clips without labeled motion, -12.3% (Wan2.2) and -8.2% (Ctrl-World) on clips with a non-rigid response.
The gain comes from non-rigid responses. Where objects are pushed, dragged or deformed, the error drops by 12.3% and 8.2%; clips without labeled motion change little.
Object motion error when the simulator is swapped at test time: static points and a hand-written rule are worse than the robot render alone; PointWorld-small, PointWorld-large and label point flow are better.
The video follows the simulator. Swapped at test time with the video model fixed, better point dynamics give a better video, from static points and a hand-written rule to learned simulators.
Conclusion

Better point dynamics, more faithful world models

We decouple a robotic world model into a simulator that predicts how 3D scene points respond to the robot and a renderer that turns the prediction into video, connected by a pixel-aligned point dynamics proxy. Explicit point dynamics make object interaction more faithful, and the video follows the simulator. Progress in 3D point dynamics, learned or written as code, can thus carry over to robotic world models without changing how they render.

  1. Y. Chen, G. Lin, C. Zhang. Code World Model: Coding Agent as World Brain. arXiv:2608.25927, 2026.
  2. Z. Fang, Y. Shao, K. Zhu, et al. Code Plans, Diffusion Renders: Open-Ended Generative World Modeling. arXiv:2609.26458, 2026.
  3. Z.-H. Huang, G. Lin, J. Lin, et al. Programmable World Model. arXiv:2609.10540, 2026.
  4. W. Huang, Y.-W. Chao, A. Mousavian, et al. PointWorld: Scaling 3D World Models for In-The-Wild Robotic Manipulation. CVPR 2026.
Citation

BibTeX

@misc{an2026pointdynamicsproxyrobotic,
  title         = {Point Dynamics as Proxy for Robotic World Models},
  author        = {Dong An and Tianyu Liu and Haoran Ding and Wen Yang and Anqing Duan and
                   Zhengzhong Liu and Eric P. Xing and Guangyi Liu},
  year          = {2026},
  eprint        = {2611.25946},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV},
  url           = {https://arxiv.org/abs/2611.25946}
}