PhyCo: Learning Controllable Physical Priors for Generative Motion

arXiv cs.CV / 5/1/2026

📰 NewsIdeas & Deep AnalysisModels & Research

共有:

Key Points

PhyCo is a new framework aimed at improving physical consistency in video diffusion models, addressing issues like object drift, unrealistic collisions, and mismatched material responses.
It combines a large photorealistic simulation dataset (100K+ videos with systematically varied friction, restitution, deformation, and forces), physics-supervised fine-tuning using a ControlNet conditioned on pixel-aligned physical property maps, and VLM-guided reward optimization via physics-targeted queries.
The method enables controllable, physically plausible motion generation without requiring a simulator or geometry reconstruction at inference time.
Experiments on the Physics-IQ benchmark show substantial gains in physical realism over strong baselines, and human evaluations indicate more faithful control over physical attributes.
Overall, the work proposes a scalable route to physically consistent and controllable generative video models that can generalize beyond the synthetic environments used for training.

Abstract

Modern video diffusion models excel at appearance synthesis but still struggle with physical consistency: objects drift, collisions lack realistic rebound, and material responses seldom match their underlying properties. We present PhyCo, a framework that introduces continuous, interpretable, and physically grounded control into video generation. Our approach integrates three key components: (i) a large-scale dataset of over 100K photorealistic simulation videos where friction, restitution, deformation, and force are systematically varied across diverse scenarios; (ii) physics-supervised fine-tuning of a pretrained diffusion model using a ControlNet conditioned on pixel-aligned physical property maps; and (iii) VLM-guided reward optimization, where a fine-tuned vision-language model evaluates generated videos with targeted physics queries and provides differentiable feedback. This combination enables a generative model to produce physically consistent and controllable outputs through variations in physical attributes-without any simulator or geometry reconstruction at inference. On the Physics-IQ benchmark, PhyCo significantly improves physical realism over strong baselines, and human studies confirm clearer and more faithful control over physical attributes. Our results demonstrate a scalable path toward physically consistent, controllable generative video models that generalize beyond synthetic training environments.