GET3D

Description

GET3D is a novel generative model developed by Toronto AI Lab and NVIDIA, designed to produce high-quality 3D textured shapes directly from 2D image data. Addressing the growing demand for scalable 3D content creation tools in industries like gaming, virtual reality, and digital design, GET3D synthesizes textured meshes that are immediately compatible with standard 3D rendering engines. This bypasses the complexities of neural renderers often found in prior 3D generative models, making the output directly usable in common 3D software.

The model operates by generating a 3D Signed Distance Function (SDF) and a texture field, both derived from latent codes. A differentiable renderer, specifically DMTet, is employed to extract a 3D surface mesh from the SDF. The texture field is then queried at surface points to assign colors, resulting in high-fidelity textures. Training is achieved through adversarial losses defined on 2D images, utilizing two discriminators that classify real versus fake RGB images and silhouettes. This end-to-end trainable approach allows GET3D to learn from vast 2D image datasets.

GET3D demonstrates remarkable capabilities in generating diverse 3D assets, including cars, chairs, animals, motorbikes, human characters, and buildings. It excels in producing shapes with arbitrary topology, detailed geometry, and high-fidelity textures, significantly outperforming previous methods. A key feature is the disentanglement between geometry and texture, allowing for independent manipulation of shape and appearance. The model also supports meaningful interpolation in its latent space, enabling smooth transitions between different shapes and textures, and can generate novel shapes with slight local variations.

Furthermore, GET3D can be combined with other techniques for unsupervised material generation and can be adapted for text-guided shape generation. By fine-tuning the 3D generator using CLIP loss on rendered 2D images and user-provided text prompts, GET3D can produce meaningful shapes based on textual descriptions. This versatility makes it a powerful tool for content creators seeking to generate a wide range of 3D assets efficiently and with high visual quality.

GET3D Highlights

• Generates high-quality 3D textured meshes

• Directly usable in 3D rendering engines

• Synthesizes shapes with arbitrary topology

• Produces rich geometric details and high-fidelity textures

• Learns from 2D image collections

• Achieves disentanglement between geometry and texture

• Supports meaningful latent space interpolation

• Enables generation of novel shapes with variations

• Can be adapted for text-guided shape generation

• Trained using adversarial losses on 2D images

• Utilizes a differentiable renderer (DMTet)

• Generates 3D SDF and texture fields

Getting Started with GET3D

1. 1

   Access model: Obtain access to the GET3D model weights and code.

2. 2

   Set up environment: Configure the necessary software and hardware dependencies.

3. 3

   Integrate via API: Utilize provided libraries or APIs to interact with the model.

4. 4

   Generate 3D assets: Input latent codes or text prompts to synthesize 3D textured meshes.

5. 5

   Extract meshes: Use DMTet to extract surface meshes from generated SDFs.

6. 6

   Apply textures: Query the texture field for surface point colors.

7. 7

   Refine and export: Further process or export generated assets for downstream applications.

GET3D's Use Cases

• 3D Asset Generation
• Virtual World Content
• Game Development
• AR/VR Experiences
• Digital Design
• Text-to-3D Synthesis
• Topology Variation

FAQ from GET3D

• What is GET3D?

  GET3D is a generative AI model that synthesizes high-quality 3D textured meshes from 2D image collections, capable of producing diverse shapes with complex topology and rich details.

• How does GET3D generate 3D shapes?

  GET3D generates a 3D SDF and a texture field via latent codes. It uses DMTet to extract a 3D surface mesh from the SDF and queries the texture field for colors.

• What kind of 3D assets can GET3D generate?

  GET3D can generate a wide range of 3D assets, including cars, chairs, animals, motorbikes, human characters, and buildings, with high geometric detail and fidelity.

• Are the generated 3D meshes directly usable?

  Yes, GET3D synthesizes textured meshes that can be directly consumed by 3D rendering engines and are usable in common 3D software.

• Does GET3D support text-guided generation?

  Yes, GET3D can be fine-tuned using CLIP loss to generate shapes based on user-provided text prompts.

• What is the advantage of GET3D over previous 3D generative models?

  GET3D generates explicit textured 3D meshes with complex topology and rich geometric details, and its output is directly usable in standard 3D software, unlike models relying on neural renderers.

• Can geometry and texture be controlled independently in GET3D?

  Yes, GET3D achieves a good disentanglement between geometry and texture, allowing for independent manipulation of shape and appearance.

• How is GET3D trained?

  GET3D is trained with adversarial losses defined on 2D images, using two discriminators for RGB images and silhouettes, and is end-to-end trainable.

• What are the key components of the GET3D architecture?

  GET3D utilizes latent codes for SDF and texture fields, DMTet for mesh extraction, and a differentiable renderer with 2D discriminators for training.

• Where can I find the code for GET3D?

  The code for GET3D is available on GitHub via the nv-tlabs repository.