About VoxFormer

VoxFormer is an official PyTorch implementation of a state-of-the-art framework for camera-based 3D semantic scene completion, recognized as a CVPR 2023 Highlight paper. It transforms 2D images into complete 3D volumetric semantics by predicting the geometry and category of both visible and occluded scene structures. The framework utilizes a novel two-stage design: first, it generates a sparse set of voxel queries from depth estimates to focus on visible occupied regions, then densifies these queries into a full 3D grid using a masked autoencoder approach with deformable self-attention. This strategy efficiently propagates visual features from 2D images to all 3D voxels. Significant performance gains include a 20% improvement in geometry and 18% in semantics on the SemanticKITTI benchmark compared to previous methods, while reducing GPU memory requirements during training by approximately 45%. Key features include support for single or multi-view image inputs, integration with ResNet50 for feature extraction,

n

Published by

nvlabs

Visit View Profile

README.md

View on GitHub

VoxFormer: a Cutting-edge Baseline for 3D Semantic Occupancy Prediction

VoxFormer: Sparse Voxel Transformer for Camera-based 3D Semantic Scene Completion, CVPR 2023.

Yiming Li, Zhiding Yu, Chris Choy, Chaowei Xiao, Jose M. Alvarez, Sanja Fidler, Chen Feng, Anima Anandkumar

[PDF] [Project] [Intro Video]

News

[2023/07]: We release the code of voxformer with 3D deformable attention module, achieving slightly better performance.
[2023/06]: 🔥 We release SSCBench, a large-scale semantic scene completion benchmark derived from KITTI-360, nuScenes, and Waymo.
[2023/06]: Welcome to our CVPR poster session on 21 June (WED-AM-082), and check our online video.
[2023/03]: 🔥 VoxFormer is accepted by CVPR 2023 as a highlight paper (235/9155, 2.5% acceptance rate).
[2023/02]: Our paper is on arxiv.
[2022/11]: VoxFormer achieve the SOTA on SemanticKITTI 3D SSC (Semantic Scene Completion) Task with 13.35% mIoU and 44.15% IoU (camera-only)!

Abstract

Humans can easily imagine the complete 3D geometry of occluded objects and scenes. This appealing ability is vital for recognition and understanding. To enable such capability in AI systems, we propose VoxFormer, a Transformer-based semantic scene completion framework that can output complete 3D volumetric semantics from only 2D images. Our framework adopts a two-stage design where we start from a sparse set of visible and occupied voxel queries from depth estimation, followed by a densification stage that generates dense 3D voxels from the sparse ones. A key idea of this design is that the visual features on 2D images correspond only to the visible scene structures rather than the occluded or empty spaces. Therefore, starting with the featurization and prediction of the visible structures is more reliable. Once we obtain the set of sparse queries, we apply a masked autoencoder design to propagate the information to all the voxels by self-attention. Experiments on SemanticKITTI show that VoxFormer outperforms the state of the art with a relative improvement of 20.0% in geometry and 18.1% in semantics and reduces GPU memory during training by ~45% to less than 16GB.

Method


Figure 1. Overall framework of VoxFormer. Given RGB images, 2D features are extracted by ResNet50 and the depth is estimated by an off-the-shelf depth predictor. The estimated depth after correction enables the class-agnostic query proposal stage: the query located at an occupied position will be selected to carry out deformable cross-attention with image features. Afterwards, mask tokens will be added for completing voxel features by deformable self-attention. The refined voxel features will be upsampled and projected to the output space for per-voxel semantic segmentation. Note that our framework supports the input of single or multiple images.

Figure 1. Overall framework of VoxFormer. Given RGB images, 2D features are extracted by ResNet50 and the depth is estimated by an off-the-shelf depth predictor. The estimated depth after correction enables the class-agnostic query proposal stage: the query located at an occupied position will be selected to carry out deformable cross-attention with image features. Afterwards, mask tokens will be added for completing voxel features by deformable self-attention. The refined voxel features will be upsampled and projected to the output space for per-voxel semantic segmentation. Note that our framework supports the input of single or multiple images.

Getting Started

Model Zoo

The query proposal network (QPN) for stage-1 is available here. For stage-2, please download the trained models based on the following table.

Backbone	Method	Lr Schd	IoU	mIoU	Config	Download
R50	VoxFormer-T	20ep	44.15	13.35	config	model
R50	VoxFormer-S	20ep	44.02	12.35	config	model
R50	VoxFormer-T-3D	20ep	44.35	13.69	config	model
R50	VoxFormer-S-3D	20ep	44.42	12.86	config	model

Dataset

[x] SemanticKITTI
[ ] KITTI-360
[ ] nuScenes

Bibtex

If this work is helpful for your research, please cite the following BibTeX entry.

@InProceedings{li2023voxformer,
      title={VoxFormer: Sparse Voxel Transformer for Camera-based 3D Semantic Scene Completion}, 
      author={Li, Yiming and Yu, Zhiding and Choy, Christopher and Xiao, Chaowei and Alvarez, Jose M and Fidler, Sanja and Feng, Chen and Anandkumar, Anima},
      booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
      year={2023}
}

License

This work is made available under the Nvidia Source Code License-NC. Click here to view a copy of this license.

The pre-trained models are shared under CC-BY-NC-SA-4.0. If you remix, transform, or build upon the material, you must distribute your contributions under the same license as the original.

For business inquiries, please visit our website and submit the form: NVIDIA Research Licensing.

Star History

Acknowledgement

Many thanks to these excellent open source projects:

VoxFormer