CLaD: Planning with Grounded Foresight via Cross-Modal Latent Dynamics

arXiv cs.RO / 4/1/2026

💬 OpinionIdeas & Deep AnalysisModels & Research

共有:

Key Points

The paper introduces CLaD, a robotics planning framework that explicitly aligns kinematic (proprioceptive) and semantic state transitions rather than planning in only one space.
CLaD uses asymmetric cross-attention where kinematic transitions query semantic ones, enabling “grounded latent foresight” predictions conditioned on both modalities.
It trains with self-supervised objectives, EMA target encoders, and auxiliary reconstruction losses to reduce representation collapse while keeping predictions anchored to observable states.
The predicted foresights are combined with current observations to condition a diffusion policy that generates actions.
On the LIBERO-LONG benchmark, CLaD reports a 94.7% success rate while remaining competitive with large vision-language-action models using fewer parameters.

Abstract

Robotic manipulation involves kinematic and semantic transitions that are inherently coupled via underlying actions. However, existing approaches plan within either semantic or latent space without explicitly aligning these cross-modal transitions. To address this, we propose CLaD, a framework that models how proprioceptive and semantic states jointly evolve under actions through asymmetric cross-attention that allows kinematic transitions to query semantic ones. CLaD predicts grounded latent foresights via self-supervised objectives with EMA target encoders and auxiliary reconstruction losses, preventing representation collapse while anchoring predictions to observable states. Predicted foresights are modulated with observations to condition a diffusion policy for action generation. On LIBERO-LONG benchmark, CLaD achieves 94.7\% success rate, competitive with large VLAs with significantly fewer parameters.