MemoryVLA: Perceptual-Cognitive Memory in Vision-Language-Action Models for Robotic Manipulation

ICLR 2026

CVPR Workshop Oral

Hao Shi1, Bin Xie2, Yingfei Liu2, Lin Sun4, Fengrong Liu5 Tiancai Wang2,
Erjin Zhou2, Haoqiang Fan2, Xiangyu Zhang3,6, Gao Huang1✉
1THU LeapLab, 2Dexmal, 3MEGVII, 4TJU, 5HIT, 6Stepfun
shihao1895@gmail.com
Paper arXiv Code Hugging Face Models Poster Slides Journal Version
中文解读 Talk@3D视觉工坊 Talk@具身智能之心

Introduction

Introduction. (a) In Push Buttons tasks, pre- and post-push states look nearly identical, calling for temporal modeling. (b) Humans handle manipulation tasks via a dual-memory system: working memory (neural activity) supports short-term control, while episodic memory (hippocampus) preserves long-term experience. (c) Inspired by this, MemoryVLA introduces a Perceptual-Cognitive Memory Bank that consolidates low-level perceptual details and high-level cognitive semantics for temporally aware decision making. (d) MemoryVLA outperforms state-of-the-art baselines.

1 Abstract

Temporal context is essential for robotic manipulation because such tasks are inherently non-Markovian, yet mainstream VLA models typically overlook it and struggle with long-horizon, temporally dependent tasks. Cognitive science suggests that humans rely on working memory to buffer short-lived representations for immediate control, while the hippocampal system preserves verbatim episodic details and semantic gist of past experience for long-term memory.

Inspired by these mechanisms, we propose MemoryVLA, a Cognition-Memory-Action framework for long-horizon robotic manipulation. A pretrained VLM encodes the observation into perceptual and cognitive tokens that form working memory, while a Perceptual-Cognitive Memory Bank stores low-level details and high-level semantics consolidated from it. Working memory retrieves decision-relevant entries from the bank, adaptively fuses them with current tokens, and updates the bank by merging redundancies. Using these tokens, a memory-conditioned diffusion action expert yields temporally aware action sequences.

We evaluate MemoryVLA on 150+ simulation and real-world tasks across three robots. On SimplerEnv-Bridge, Fractal, and Libero-5 suites, it achieves 71.9%, 72.7%, and 96.5% success rates, respectively, all outperforming state-of-the-art baselines CogACT and PI-0, with a notable +14.6 gain on Bridge. On 12 real-world tasks spanning general skills and long-horizon temporal dependencies, MemoryVLA achieves 84.0% success rate, with long-horizon tasks showing a +26 improvement over state-of-the-art baseline. Moreover, MemoryVLA exhibits strong robustness and generalization under various out-of-distribution conditions.

2 Method

2.1 Framework

Overall architecture of MemoryVLA. RGB observation and language instruction are encoded by a 7B VLM into perceptual and cognitive tokens, forming short-term working memory. The working memory queries a perceptual-cognitive memory bank (PCMB) to retrieve relevant historical context, including high-level semantics and low-level visual details, adaptively fuses it with current tokens, and consolidates the PCMB by merging the most similar neighbors. The memory-augmented tokens then condition a diffusion transformer to predict a sequence of future actions.

2.2 Module Details

Module details (a) Retrieval: current perceptual and cognitive tokens query the PCMB via cross-attention with timestep positional encoding to fetch relevant historical features. (b) Gate fusion: current and retrieved tokens are adaptively fused via a gate mechanism. (c) Consolidation: the fused tokens are updated into PCMB. When PCMB reaches its capacity, we compute similarities between adjacent entries and merge the most similar pair to maintain compactness.

3 Experiment

3.1 Experimental Overview

Experimental overview. Top: three simulation benchmarks, SIMPLER-Bridge with WidowX, SIMPLER-Fractal with Google Robot, and Libero with Franka. Bottom: real-world evaluation on two suites, General and Long-horizon Temporal. In total, we evaluate three robots across 10 suites, spanning over 150 tasks and 500 variations.

3.2 Real Robot Setup

3.3 Real Robot Evaluation

Sequential Push Buttons

Change Food

Guess Where

Clean Table & Count

Pick Place Order

Clean Restaurant Table

Insert Circle

Put Egg in Pan

Put Egg in Oven

Stack Cups

Stack Blocks

Pick Diverse Fruits (apple)

Pick Diverse Fruits (banana)

Pick Diverse Fruits (carrot)

Pick Diverse Fruits (chili)

Pick Diverse Fruits (grape)

Pick Place Order

Base

Unseen Background

Unseen Distractors

Unseen Lighting

Unseen Object

Unseen Container

Unseen Occlusion

Clean Restaurant Table

Base

Unseen Background

Unseen Distractors

Unseen Lighting

Unseen Object

Unseen Container

Unseen Occlusion

Real-world OOD Tests

OOD Real Results

3.4 Simulation Evaluation

SimplerEnv-Bridge Tasks

Put Spoon on Towel

Put Carrot on Plate

Stack Cube

Put Eggplant in Basket

SimplerEnv-Fractal Tasks

Pick Coke Can

Move Near

Open Drawer

Close Drawer

Put in Drawer

Libero-Spatial Tasks

Bowl: Cabinet → Plate

Bowl: Cookie Box → Plate

Bowl: Drawer → Plate

Bowl: Stove → Plate

Libero-Object Tasks

Alphabet Soup in Basket

Chocolate in Basket

Orange Juice in Basket

Tomato Sauce in Basket

Libero-Goal Tasks

Bowl in Top Drawer

Open Middle Drawer

Turn On Stove

Wine Bottle on Rack

Libero-Long Tasks

Alphabet Soup & Tomato Sauce in Basket

Bowl in Bottom Drawer & Close

Turn On Stove & Put Moka Pot

Yellow/White Mug in Microwave & Close

Intercept Medium

Remember Color 3

Remember Color 5

Remember Color 9

Shell Game Touch

Put in Drawer

Base

Unseen Background (bedroom)

Unseen Background (office)

Unseen Lighting (brighter)

Unseen Lighting (darker)

Unseen Texture

Pick and Move OOD Tests in Simulation

OOD Pick and Move Results

Hinge Object Manipulation OOD Tests in Simulation

OOD Drawer Results

4 Insights

Insight 1
Insight 2
Insight 3 Insight 4
Insight 5

5 MemoryVLA++: Full Temporal Aware

MemoryVLA++ Full Temporal Aware

MemoryVLA++ is the extended journal version of MemoryVLA. We compare three paradigms: (1) typical VLAs are reactive and rely only on the present observation, (2) MemoryVLA introduces a working memory-episodic memory mechanism to capture past temporal dependencies, and (3) MemoryVLA++ further extends this by incorporating future imagination via a world model for full temporal modeling.

MemoryVLA++

BibTeX

@article{shi2025memoryvla,
  title={MemoryVLA: Perceptual-Cognitive Memory in Vision-Language-Action Models for Robotic Manipulation},
  author={Shi, Hao and Xie, Bin and Liu, Yingfei and Sun, Lin and Liu, Fengrong and Wang, Tiancai and Zhou, Erjin and Fan, Haoqiang and Zhang, Xiangyu and Huang, Gao},
  journal={arXiv preprint arXiv:2508.19236},
  year={2025}
}

@article{shi2026memoryvla++,
  title={MemoryVLA++: Temporal Modeling via Memory and Imagination in Vision-Language-Action Models},
  author={Shi, Hao and Li, Weiye and Xie, Bin and Wang, Yulin and Zhou, Renping and Wang, Tiancai and Zhang, Xiangyu and Luo, Ping and Huang, Gao},
  journal={arXiv preprint arXiv:2606.09827},
  year={2026}
}