DiLightNet
DiLightNet is a novel method presented at SIGGRAPH 2024 for achieving fine-grained lighting control in text-driven diffusion-based image generation. The system utilizes a three-stage workflow comprising provisional image generation, foreground synthesis, and background inpainting. This repository provides the source code and pre-trained ControlNet model, known as NeuralTextureControlNet, specifically for the foreground synthesis stage. This neural network accepts a provisional image, a mask, and radiance hints as inputs to generate a foreground image that conforms to a specified lighting environment. For the initial and final stages of the pipeline, users can integrate existing off-the-shelf models such as Stable Diffusion or commercial services like DALL-E 3 and MidJourney. The repository includes instructions for environment setup using Python 3.10 and CUDA, model loading, inference pipelines, and CLI tools for video generation with continuous lighting changes. It also features a Gradio demo for single-imag