THOM: Generating Physically Plausible Hand-Object Meshes From Text

arXiv cs.CV / 4/6/2026

📰 NewsIdeas & Deep AnalysisModels & Research

共有:

Key Points

THOM is a training-free framework for generating physically plausible 3D hand-object interaction (HOI) meshes directly from text, targeting dexterous robotic grasping and VR/AR content creation needs.
It uses a two-stage pipeline that first generates hand and object Gaussians from the text, then performs physics-based HOI optimization after extracting meshes from those Gaussians.
The approach introduces a new mesh extraction method plus a vertex-to-Gaussian mapping that assigns Gaussian elements to mesh vertices, enabling topology-aware regularization.
To improve interaction plausibility, THOM adds VLM-guided translation refinement and contact-aware optimization during physics optimization.
Experiments reported in the paper indicate THOM outperforms existing methods on text alignment, visual realism, and interaction plausibility.

Abstract

The generation of 3D hand-object interactions (HOIs) from text is crucial for dexterous robotic grasping and VR/AR content generation, requiring both high visual fidelity and physical plausibility. Nevertheless, the ill-posed problem of mesh extraction from text-generated Gaussians, and physics-based optimization on the erroneous meshes pose challenges. To address these issues, we introduce THOM, a training-free framework that generates photorealistic, physically plausible 3D HOI meshes without the need for a template object mesh. THOM employs a two-stage pipeline, initially generating the hand and object Gaussians, followed by physics-based HOI optimization. Our new mesh extraction method and vertex-to-Gaussian mapping explicitly assign Gaussian elements to mesh vertices, allowing topology-aware regularization. Furthermore, we improve the physical plausibility of interactions by VLM-guided translation refinement and contact-aware optimization. Comprehensive experiments demonstrate that THOM consistently surpasses state-of-the-art methods in terms of text alignment, visual realism, and interaction plausibility.