OneHOI: Unifying Human-Object Interaction Generation and Editing

arXiv cs.CV / 4/16/2026

📰 NewsSignals & Early TrendsModels & Research

共有:

Key Points

OneHOI introduces a unified diffusion transformer framework that combines human-object interaction (HOI) generation and text-based HOI editing into a single conditional denoising process.
The approach centers on a Relational Diffusion Transformer (R-DiT) that uses role-/instance-aware HOI tokens, layout-based action grounding, structured HOI attention for interaction topology, and HOI RoPE to disentangle multi-HOI scenes.
By training jointly with modality dropout on the HOI-Edit-44K dataset plus additional HOI and object-centric data, OneHOI can handle layout-guided, layout-free, arbitrary-mask, and mixed-condition (HOI + object-only) controls.
The paper reports state-of-the-art performance across both HOI generation and editing tasks, and provides code release via the project website.

Abstract

Human-Object Interaction (HOI) modelling captures how humans act upon and relate to objects, typically expressed as triplets. Existing approaches split into two disjoint families: HOI generation synthesises scenes from structured triplets and layout, but fails to integrate mixed conditions like HOI and object-only entities; and HOI editing modifies interactions via text, yet struggles to decouple pose from physical contact and scale to multiple interactions. We introduce OneHOI, a unified diffusion transformer framework that consolidates HOI generation and editing into a single conditional denoising process driven by shared structured interaction representations. At its core, the Relational Diffusion Transformer (R-DiT) models verb-mediated relations through role- and instance-aware HOI tokens, layout-based spatial Action Grounding, a Structured HOI Attention to enforce interaction topology, and HOI RoPE to disentangle multi-HOI scenes. Trained jointly with modality dropout on our HOI-Edit-44K, along with HOI and object-centric datasets, OneHOI supports layout-guided, layout-free, arbitrary-mask, and mixed-condition control, achieving state-of-the-art results across both HOI generation and editing. Code is available at https://jiuntian.github.io/OneHOI/.