LexiconVLA Learning Reusable Atomic Action Codebooks
for Unseen Tasks

Zeming Wei1,* Jianheng Ye1,* Xinshuai Song1 Sirui Chen1 Yang Liu1,3,† Liang Lin1,2,3

1Sun Yat-sen University 2Pengcheng Laboratory 3X-Era AI Lab

* Equal contribution  ·  † Corresponding author

Paper arXiv Code Demos
“put the ball in the hoop”
  1. approach
  2. grasp
  3. transfer
  4. release
57,803atomic-action segments in AtomAction
69RLBench tasks · 18 action types
+17.50ppunseen-task success with BridgeVLA
71.08%overall success on 26 RLBench tasks
Figure 1. LexiconVLA and motivation. Left: LexiconVLA decomposes an instruction into atomic-action subtasks, executes each with a VLA policy conditioned on codes from a shared codebook, and advances, retries, or replans according to execution feedback; real-robot rollouts show step-wise execution and recovery from a wrong button press. Right: a diagnostic probe on seven tasks, where each atomic subtask is prompted alone from the demonstration state. For existing policies, high full-task success does not translate into reliable or task-consistent atomic execution.

01Abstract

What if a robot could reuse what it already knows?

Vision-language-action (VLA) models struggle to reuse recurring interactions in unseen tasks. Our diagnostic study reveals that reliable task completion does not imply consistent execution of constituent atomic actions across task contexts. We present LexiconVLA, a retrievable atomic-action lexicon for cross-task reuse.

Global and detail codebooks capture shared interaction structure and fine-grained execution variation, respectively, preserving both reusable patterns and execution details. Visual-Atomic Action Alignment couples trajectory reconstruction from visual state changes with visual outcome prediction from action codes, grounding the lexicon in motion and its effects. We learn these codebooks with trajectory reconstruction and visual alignment on our AtomAction Dataset of 57,803 segments from 69 tasks.

A planner and scene-aware adapter translate new goals into code-conditioned subtasks for a shared policy, without skill-specific experts or deployment-time parameter updates. Across five policy backbones on 26 RLBench tasks, LexiconVLA largely maintains performance on 18 seen tasks while improving success on 8 tasks held out from policy training. With BridgeVLA, unseen-task success rises from 16.67% to 34.17% (+17.50 percentage points), and overall success reaches 71.08%, the highest among methods with reported results. Real-robot experiments demonstrate stepwise execution and failure recovery.

02Motivation

Finishing a task ≠ mastering its parts.

We probe five pretrained manipulation policies of diverse architectures on seven tasks, prompting each atomic subtask alone from its demonstration start state. Task-level training alone does not yield atomic actions that can be reliably invoked on their own — a prerequisite for recombining them in unseen tasks.

Even the contact-free primitive approach varies widely across task contexts within a single policy.

03AtomAction Dataset

One segment, one mechanism.

Most VLA datasets pair a whole demonstration with one instruction, and coarse labels such as open lump together pulling, hinge rotation, and lid flipping. AtomAction instead cuts RLBench expert demonstrations into temporally localized segments that each realize exactly one mechanism-level atomic action.

Trajectory

Low-dimensional state of the phase: end-effector pose, joint positions, velocities and torques, gripper state and contact forces.

Observations

Five-view RGB, depth, and mask observations at the start and end boundaries of the segment.

Language

An atomic-action label plus eight phase-level descriptions written in different language styles.

Figure 2. The AtomAction Dataset. Left: atomic-action categories (inner ring) and contributing tasks (outer ring). Top right: segment counts per category; coloured block widths show individual task contributions. Bottom right: an example segment with start/end frames and language descriptions.

04Method

A shared vocabulary for robot action.

LexiconVLA conditions one shared manipulation policy on a frozen lexicon of global and detail codes. New goals become sequences of code-conditioned subtasks — no skill-specific experts, no parameter updates at deployment.

  1. 01

    Plan

    A VLM planner decomposes the task instruction into atomic subtask instructions.

  2. 02

    Retrieve

    A scene-aware adapter maps the active subtask and current images to global and detail codes from the lexicon.

  3. 03

    Execute

    A lightweight injection module conditions the shared VLA policy on the retrieved codes while it acts.

Figure 3. Overview of LexiconVLA. A VLM-based planner decomposes the task instruction into atomic subtasks. Given the active subtask instruction and current observations, a scene-aware adapter retrieves entries from the frozen atomic-action lexicon — global and detail codes — to condition a shared VLA policy. Execution feedback drives subtask progression and recovery through four skills.

Closed-loop scheduling

The planner assesses visual feedback after each step and picks one of four skills. Subtask transitions and replanning refresh the retrieved codes from updated observations.

NEXT

Advance once the active subtask is complete.

CONTINUE

Keep the active subtask when more execution is needed.

RETRY

Repeat a failed step, optionally returning to an earlier subtask.

REPLAN

Regenerate the plan when scene changes invalidate it.

Learning the lexicon

The Vector-Quantized Atomic Action Model (VQAA) grounds every code in both motion and its visual effect. Its decoder is used only during codebook training.

(a) Atomic Action-NSVQ

Global + detail codebooks

A Transformer encodes each 39-D segment; pooling gives one global code for reusable structure, learnable queries give detail codes for execution variation. Noise-substituted VQ and a conditional flow-matching decoder train them by trajectory reconstruction.

(b) Visual-Atomic Action Alignment

V3A: what the action changes

Actions are reconstructed from the visual change between initial and final frames, and final-frame features are predicted from the initial observation and the global code — tying each code to its observable outcome.

Figure 4. Vector-Quantized Atomic Action Model (VQAA). (a) Atomic Action-NSVQ encodes an atomic-action segment into discrete codes from two learnable codebooks using Noise Substitution in Vector Quantization; a conditional flow-matching decoder provides trajectory-reconstruction supervision. (b) V3A adds two auxiliary objectives: reconstructing actions from initial-to-final visual changes, and predicting final-frame features conditioned on the initial observation and the global action code.

05Results

Built to transfer beyond the seen.

RLBench with 18 seen tasks and 8 tasks held out from policy training. Full-task success over 25 episodes per task in each of three random seeds.

Unseen-task success · BridgeVLA backbone
16.67%34.17%

More than doubled, surpassing the strongest baseline (3D Diffuser Actor, 28.67%) by 5.50 points.

Overall success · 26 tasks
67.28%71.08%

Highest among all evaluated methods, while seen-task success is largely preserved.

Unseen-task success across five backbones

Success rate (%) on the 8 policy-unseen RLBench tasks, mean over three seeds.

Ablation · BridgeVLA backbone

Unseen-task success (%). Every component contributes to transfer.

Real robot · 5 tasks × 10 trials

Average success (%) under training conditions and under unseen perturbations.

Figure 5. Qualitative rollouts in simulation and the real world. Rows 1–2: policy-unseen RLBench tasks; rows 3–4: real-robot tasks. After a missed grasp (yellow box) or a human disturbance (red box), our planner triggers RETRY, re-executing the failed subtask with codes re-retrieved from the current observation, whereas the baselines fail to recover.

06Simulation demos · RLBench

Every task, step by step.

Pick a task to see LexiconVLA (BridgeVLA backbone) succeed outright, recover with RETRY or REPLAN, and the BridgeVLA baseline failing — often from the very same initial scene.

Speed

07Real-robot demos

Unseen conditions. Real actions.

A Franka arm compared with BridgeVLA and π0.5, all fine-tuned on the same 30 demonstrations per task. Each clip is one representative rollout; the success rates come from Table 3 of the paper (10 trials per condition).

Speed

08Citation

Cite LexiconVLA.

@misc{wei2026lexiconvlalearningreusableatomic,
  title={LexiconVLA: Learning Reusable Atomic Action Codebooks for Unseen Tasks},
  author={Zeming Wei and Jianheng Ye and Xinshuai Song and Sirui Chen and Yang Liu and Liang Lin},
  year={2026},
  eprint={2609.36774},
  archivePrefix={arXiv},
  primaryClass={cs.RO},
  url={https://arxiv.org/abs/2609.36774},
}