DreamFusion

DreamFusion represents a significant leap forward in the field of 3D model generation, leveraging the power of AI to convert textual descriptions into detailed 3D objects. This innovative approach utilizes a pretrained 2D text-to-image diffusion model, circumventing the need for large-scale datasets of labeled 3D assets or specialized architectures for 3D data denoising. By introducing a novel loss based on probability density distillation, DreamFusion enables the optimization of a parametric image generator using a 2D diffusion model as a prior. This process involves optimizing a randomly-initialized 3D model, specifically a Neural Radiance Field (NeRF), through gradient descent. The optimization ensures that 2D renderings of the 3D model from various angles achieve a low loss, resulting in a versatile 3D model that can be viewed from any angle, relit under arbitrary illumination, or integrated into any 3D environment. DreamFusion's methodology requires no 3D training data and no modifications to the image diffusion model, showcasing the efficacy of pretrained image diffusion models as priors for 3D synthesis. The tool generates relightable 3D objects with high-fidelity appearance, depth, and normals, represented as Neural Radiance Fields and leveraging a pretrained text-to-image diffusion prior such as Imagen. DreamFusion's capabilities extend to generating objects and scenes from diverse captions, with the ability to search through hundreds of generated assets in its gallery. Additionally, the generated NeRF models can be exported to meshes using the marching cubes algorithm, facilitating easy integration into 3D renderers or modeling software. DreamFusion's innovative Score Distillation Sampling (SDS) technique allows for the generation of samples from a diffusion model by optimizing a loss function, enabling optimization in an arbitrary parameter space, such as 3D space, provided a differentiable mapping back to images exists. The tool employs a 3D scene parameterization similar to Neural Radiance Fields to define this differentiable mapping, with additional regularizers and optimization strategies enhancing geometry. The resulting trained NeRFs are coherent, featuring high-quality normals, surface geometry, and depth, and are relightable with a Lambertian shading model. DreamFusion's approach to text-to-3D synthesis opens new avenues for creative and practical applications, from digital art and design to virtual reality and beyond.