Robotic Manipulation is Vision-to-Geometry Mapping ($f(v) \rightarrow G$): Vision-Geometry Backbones over Language and Video Models

Zijian Song1,   Qichang Li1,   Jiawei Zhou1,   Zhenlong Yuan4,   Tianshui Chen35,   Liang Lin123,   Guangrun Wang123*

1 Sun Yat-sen University, Guangzhou, China
2 Guangdong Key Laboratory of Big Data Analysis and Processing
3 X-Era AI Lab
4 AMAP, Alibaba
5 Guangdong University of Technology, Guangzhou, China

ArXiv GitHub HuggingFace

Abstract

At its core, robotic manipulation is a problem of vision-to-geometry mapping ($f(v) \rightarrow G$). Physical actions-such as reaching, grasping, and orienting-are fundamentally defined by geometric properties like 3D positions, rotations, and spatial relationships. Consequently, we argue that the foundation for generalizable robotic control should be a vision-geometry backbone, rather than the widely adopted vision-language or video models. Conventional Vision-Language-Action (VLA) and video-predictive models rely on backbones pretrained on large-scale 2D image-text or temporal pixel data. While effective, their representations are largely shaped by semantic concepts or 2D priors, which do not intrinsically align with the precise 3D geometric nature required for physical manipulation. Driven by this insight, we propose the Vision-Geometry-Action (VGA) model, which directly conditions action generation on pretrained native 3D representations. Specifically, VGA replaces conventional language or video backbones with a pretrained 3D world model, establishing a seamless vision-to-geometry mapping that translates visual inputs directly into physical actions. To further enhance geometric consistency, we introduce a Progressive Volumetric Modulation module and adopt a joint training strategy that simultaneously predicts actions and 3D properties, improving both representation fidelity and cross-modal interaction. Extensive experiments validate the effectiveness of our approach. In simulation benchmarks, VGA outperforms top-tier VLA baselines including $\pi_{0.5}$, SpatialVLA, and GeoVLA, demonstrating its superiority in precise spatial manipulation. More importantly, VGA exhibits remarkable zero-shot generalization to unseen viewpoints in real-world deployments, consistently outperforming $\pi_{0.5}$ in terms of success rate. These results highlight that operating on native 3D representations-rather than translating through language or 2D video priors-is a highly promising direction for achieving generalizable physical intelligence.

Overview

Fig.1 Robotic manipulation as vision-to-geometry mapping ($f(v) \rightarrow G$). Physical actions like reaching, grasping, and orienting are inherently driven by geometric properties, such as 3D position, rotation, and spatial relationships. Therefore, we argue that a vision-geometry backbone provides a stronger foundation for generalizable robotic control than prevalent vision-language-models or video prediction models.

Fig.2 Overview of our VGA model. (a) The left column compares our VGA framework with representative robot learning paradigms. VGA differs from them by leveraging a pretrained 3D world model as the backbone, providing native 3D representations aligned with physical actions. (b) The right column illustrates the workflow of the VGA model. Multimodal inputs are tokenized into a unified sequence and processed by a pretrained VGGT transformer with alternating attention. The resulting latent features are then mapped by task-specific heads to produce multimodal outputs, each with corresponding supervision.

Results

Simulation
Experiments

Real-World
Experiments

Ablation
Experiments

Simulation Experiments

Fig.3 The success rate comparison on the LIBERO benchmark. VGA consistently outperforms representative approaches across three major paradigms: standard VLAs, 3D-VLAs, and World Action Models (WAM). Compared to strong VLA baselines such as $\pi_{0.5}$ and OpenVLA-OFT, VGA achieves higher success rates, demonstrating the advantage of grounding action prediction in native 3D representations. Compared to 3D-VLA variants (e.g., SpatialVLA, GeoAware, and GeoVLA), VGA further shows that simply injecting geometric cues into existing pipelines is insufficient; instead, tightly coupling a 3D world model backbone with the action head enables more effective use of geometric priors. VGA also surpasses WAM approaches like UVA and Motus, indicating that pretrained 3D world models provide a strong and competitive alternative to video-based generative backbones. Overall, these results highlight VGA's ability to translate accurate 3D understanding into precise and robust physical manipulation.

LIBERO Visualization

(a)

(b)

(c)

(d)

Simulation rollouts and depth predictions on the LIBERO benchmark. The results show that VGA achieves precise physical manipulation with precise depth predictions.
(a) Put the yellow and white mug in the microwave and close it.
(b) Put the white mug on the plate and put the chocolate pudding to the right of the plate.
(c) Put the black bowl in the bottom drawer of the cabinet and close it.
(d) Turn on the stove and put the moka pot on it.

Real-World Experiments

Fig.4 The success rate comparison on the real-world experiments. We evaluate VGA under two real-world settings: (1) in-distribution, where the camera viewpoint during evaluation matches training, and (2) out-of-distribution, where the model is deployed under a novel, unseen viewpoint in a zero-shot manner. VGA achieves strong performance in the in-distribution setting, remaining competitive with the best baseline $\pi_{0.5}$ while significantly outperforming ACT and OpenVLA. More importantly, under out-of-distribution evaluation, VGA surpasses all baselines and achieves the highest average success rate, demonstrating clear advantages in generalizing to unseen viewpoints. These results suggest that VGA captures underlying 3D spatial structure rather than relying on viewpoint-specific visual cues, leading to more robust and reliable real-world manipulation.

Short-Horizon Tasks

(a)

(b)

(c)

(d)

(a) Real-World Pick Cube. (b) Real-World Press Button. (c) Real-World Stack Cube.

Long-Horizon Tasks

(a)

(b)

(c)

Language-conditioned grasping. This results presents the real-world grasping with three visually similar objects (namely eggplant (a), carrot (b), and cucumber (c)) arranged in different layouts. Each row corresponds to a different spatial configuration, and the robot is instructed to pick a target object. VGA consistently identifies and grasps the correct object regardless of its position, demonstrating robust language grounding and reliable real-world manipulation performance.

Citation

If you find our work useful, please consider citing:

@article{song2026robotic,
    title={Robotic manipulation is vision-to-geometry mapping (f (v)→ g): Vision-geometry backbones over language and video models},
    author={Song, Zijian and Li, Qichang and Zhou, Jiawei and Yuan, Zhenlong and Chen, Tianshui and Lin, Liang and Wang, Guangrun},
    journal={arXiv preprint arXiv:2604.12908},
    year={2026}
}

Let's Get In Touch!

If you have any questions, feel free to send us a message and we'll get back to you as soon as possible!
Email: songzj8@mail2.sysu.edu.cn