LiTo: Surface Light Field Tokenization

arXiv cs.CV / 3/12/2026

📰 NewsIdeas & Deep AnalysisModels & Research

共有:

Key Points

The LiTo method introduces a 3D latent representation that jointly models object geometry and view-dependent appearance by encoding samples of a surface light field into a compact latent space.
It leverages RGB-depth data as samples of the surface light field to capture view-dependent effects such as specular highlights and Fresnel reflections under complex lighting.
A latent flow matching model is trained to predict the latent distribution conditioned on a single input image, enabling generation of 3D objects whose appearances are consistent with the input's lighting and materials.
Experiments show higher visual quality and input fidelity than existing methods.

Abstract

We propose a 3D latent representation that jointly models object geometry and view-dependent appearance. Most prior works focus on either reconstructing 3D geometry or predicting view-independent diffuse appearance, and thus struggle to capture realistic view-dependent effects. Our approach leverages that RGB-depth images provide samples of a surface light field. By encoding random subsamples of this surface light field into a compact set of latent vectors, our model learns to represent both geometry and appearance within a unified 3D latent space. This representation reproduces view-dependent effects such as specular highlights and Fresnel reflections under complex lighting. We further train a latent flow matching model on this representation to learn its distribution conditioned on a single input image, enabling the generation of 3D objects with appearances consistent with the lighting and materials in the input. Experiments show that our approach achieves higher visual quality and better input fidelity than existing methods.