A Hierarchical Spatiotemporal Action Tokenizer for In-Context Imitation Learning in Robotics

arXiv cs.RO / 4/17/2026

📰 NewsIdeas & Deep AnalysisModels & Research

共有:

Key Points

The paper proposes a hierarchical spatiotemporal action tokenizer designed for in-context imitation learning in robotics, aimed at improving action representation from demonstrations.
It uses two successive vector-quantization levels, where a lower level assigns actions to fine-grained subclusters and a higher level groups those into broader clusters.
The hierarchical design outperforms a non-hierarchical tokenizer, primarily by leveraging spatial information through action reconstruction.
The extended method, HiST-AT, incorporates both spatial and temporal cues via multi-level clustering while also reconstructing actions together with their associated timestamps.
Experiments across multiple simulation and real robotic manipulation benchmarks report new state-of-the-art performance for in-context imitation learning.

Abstract

We present a novel hierarchical spatiotemporal action tokenizer for in-context imitation learning. We first propose a hierarchical approach, which consists of two successive levels of vector quantization. In particular, the lower level assigns input actions to fine-grained subclusters, while the higher level further maps fine-grained subclusters to clusters. Our hierarchical approach outperforms the non-hierarchical counterpart, while mainly exploiting spatial information by reconstructing input actions. Furthermore, we extend our approach by utilizing both spatial and temporal cues, forming a hierarchical spatiotemporal action tokenizer, namely HiST-AT. Specifically, our hierarchical spatiotemporal approach conducts multi-level clustering, while simultaneously recovering input actions and their associated timestamps. Finally, extensive evaluations on multiple simulation and real robotic manipulation benchmarks show that our approach establishes a new state-of-the-art performance in in-context imitation learning.