LiTo: Surface Light Field Tokenization

LiTo: Surface Light Field Tokenization introduces a 3D latent representation that jointly models object geometry and view-dependent appearance by encoding random subsamples of surface light fields derived from RGB-depth images into a compact set of latent vectors . This approach enables the reproduction of view-dependent effects such as specular highlights and Fresnel reflections under complex lighting conditions . The model is further trained using a latent flow matching framework to learn the distribution of these representations conditioned on a single input image, facilitating the generation of 3D objects with appearance consistent with the input's lighting and materials . Experiments demonstrate that this method achieves higher visual quality and better input fidelity compared to existing approaches

LiTo introduces a novel 3D latent representation framework designed to address the dual challenge of modeling object geometry and view-dependent appearance simultaneously. Departing from traditional approaches that treat shape and radiance as separate entities, LiTo leverages the concept of Surface Light Fields (SLF) to encode the rich visual information contained within RGB-depth images. The core methodology involves sampling random ray directions from surface points—effectively capturing local light field patches—and compressing these subsamples into compact latent vectors via a dedicated tokenization process. This allows the model to represent complex scenes as a set of discrete tokens that encapsulate both the local surface normals and the photometric properties of the materials.

The significance of LiTo lies in its ability to provide a highly efficient and expressive representation that preserves high-frequency details and specular effects often lost in standard 3D reconstructions. By unifying geometry and appearance into a shared latent space, the approach facilitates downstream tasks such as high-fidelity reconstruction, novel view synthesis, and generative modeling without the prohibitive storage costs associated with raw light fields or the computational overhead of volumetric neural rendering. This work advances the state of 3D computer vision by offering a scalable primitive that bridges the gap between explicit geometric structures and implicit neural representations, making it particularly valuable for applications requiring compact yet photorealistic 3D asset management.

# LiTo: Surface Light Field Tokenization

This paper introduces LiTo (Light Field Tokenization), a novel 3D latent representation that jointly models both object geometry and view-dependent appearance by encoding surface light fields derived from RGB-D data. The approach leverages random subsampling of surface light fields—representations of how light interacts with an object’s surface across different viewpoints—and compresses them into compact latent vectors. By doing so, LiTo enables efficient and high-fidelity reconstruction of 3D objects while preserving fine-grained appearance details that are critical for photorealistic rendering. The method is trained in a self-supervised manner, learning to reconstruct input views from latent codes without explicit supervision, making it scalable and adaptable to diverse datasets.

The key contributions of LiTo include: 1. Joint Geometry-Appearance Representation: Unlike traditional methods that separate geometry (e.g., via NeRF or mesh-based approaches) and texture, LiTo encodes both aspects in a unified latent space, improving coherence in reconstruction. 2. Efficient Light Field Tokenization: By subsampling surface light fields and using a transformer-based architecture, LiTo balances computational efficiency with reconstruction fidelity, making it suitable for real-time or high-resolution applications. 3. Scalability and Generalization: The method demonstrates strong performance across different object categories and viewpoints, suggesting robustness in real-world scenarios where appearance changes with perspective.

This work is significant because it addresses a long-standing challenge in 3D computer vision and graphics: the efficient and accurate representation of view-dependent effects (e.g., specular highlights, shadows, or subsurface scattering) without relying on dense, computationally expensive light field captures. LiTo’s tokenizer-based approach could enable more efficient 3D asset compression, neural rendering, and AR/VR applications, where both geometry and appearance must be faithfully preserved. By bridging the gap between 3D reconstruction and photorealistic rendering, this method opens new avenues for generative 3D modeling and autonomous scene understanding.