About NeuralLift-360

NeuralLift-360 is an advanced computer vision tool designed to transform a single 2D photograph taken in the wild into a high-quality 3D object with full 360-degree views. Recognized as a highlight at CVPR 2023, this software leverages deep learning techniques to infer unseen geometry and textures, effectively lifting flat images into immersive volumetric representations. The pipeline integrates monocular depth estimation using models like LeRes and Boost Your Own Depth, alongside foreground segmentation to isolate subjects. Users can interact with the tool via a simplified Gradio interface for quick generation or utilize a detailed command-line workflow for training custom models. The system supports optional text inversion modules to refine text-based embeddings for specific objects and allows for fine-tuning to optimize 3D consistency. It is built upon the Stable DreamFusion architecture and outputs animated videos showcasing the generated 3D model from all angles. Ideal for developers, researchers, and cr

v

Published by

vita-group

Visit View Profile

README.md

View on GitHub

NeuralLift-360: Lifting An In-the-wild 2D Photo to A 3D Object with 360° Views

[Paper] [Website]

News

23.03.12 Basic workflow is released! Gradio App released!

We're working on rolling out new exciting features soon and will prepare tutorials and better interface using gradio. Stay tuned!

Pipeline

Environment

pip install -r requirements.txt will do the job.

pip install gradio if you want to run the Gradio App.

Data Preparation

In our experiments, we use the depth from Boost Your Own depth together with LeRes.

The colab notebook to export depth in numpy can be found here.

The foreground mask can be obtained using this repo: https://github.com/Ir1d/image-background-remove-tool

Gradio

python gradio_app.py
# add `--share` if you want an external public link

We prepare a Gradio App as well!

Note: Currently, we load config from a pre-defined yaml file and we'll update this later. Also, it's a bit slower compared to directly running the training script because we have to render during training.

Training

We prepare the config files in the yaml file in configs directory.

python main.py --config configs/cabin.yaml

Text inversion [Optional]

The following script runs text inversion to obtain a better text embedding.

export MODEL_NAME="runwayml/stable-diffusion-v1-5"
accelerate launch text_inversion.py \
  --pretrained_model_name_or_path=$MODEL_NAME \
  --learnable_property="object" \
  --placeholder_token="<cabin>" --initializer_token="cabin" \
  --resolution=512 \
  --train_batch_size=1 \
  --gradient_accumulation_steps=4 \
  --max_train_steps=1000 \
  --learning_rate=5.0e-04 --scale_lr \
  --lr_scheduler="constant" \
  --lr_warmup_steps=0 \
  --output_dir="cabin_ti" \
  --im_path='data/cabin4_centered.png' \
  --mask_path='data/cabin4_centered_mask.png'

test_dm.py can be used to validate the effectiveness of textual inversion.

An example config is provided in configs/cabin_ti.yaml

The corresponding training script is:

python main.py --config configs/cabin_ti.yaml

Imagic Finetuning [Optional]

Coming soon.

Testing

After training process finishes, the code will automatically generates a video named lift_ep0100_rgb.mp4

You can filter out all these videos in the log folder and sort them by time using the following code.

find ./ -name lift_ep0100_rgb.mp4 -printf "%T@ %Tc %p\n" | sort -n

If you want to run test only, go to the yaml config file, change test: False to test: True and ckpt: to the path of the trained checkpoint.

Acknowledgement

Codebase based on https://github.com/ashawkey/stable-dreamfusion . Thanks Jiaxiang Tang for sharing and the insightful discussions!

Citation

If you find this repo is helpful, please cite:


@InProceedings{Xu_2022_neuralLift,
author = {Xu, Dejia and Jiang, Yifan and Wang, Peihao and Fan, Zhiwen and Wang, Yi and Wang, Zhangyang},
title = {NeuralLift-360: Lifting An In-the-wild 2D Photo to A 3D Object with 360° Views},
journal={arXiv preprint arXiv:2211.16431},
year={2022}
}

NeuralLift-360