Ha-NeRF😆: Hallucinated Neural Radiance Fields in the Wild

CVPR 2022


Xingyu Chen1, Qi Zhang2, Xiaoyu Li2, Yue Chen1, Ying Feng2, Xuan Wang2, Jue Wang2

1Xi'an Jiaotong University    2Tencent AI Lab

Abstract


Neural Radiance Fields (NeRF) has recently gained popularity for its impressive novel view synthesis ability. This paper studies the problem of hallucinated NeRF: i.e., recovering a realistic NeRF at a different time of day from a group of tourism images. Existing solutions adopt NeRF with a controllable appearance embedding to render novel views under various conditions, but they cannot render view-consistent images with an unseen appearance. To solve this problem, we present an end-to-end framework for constructing a hallucinated NeRF, dubbed as Ha-NeRF. Specifically, we propose an appearance hallucination module to handle time-varying appearances and transfer them to novel views. Considering the complex occlusions of tourism images, we introduce an anti-occlusion module to decompose the static subjects for visibility accurately. Experimental results on synthetic data and real tourism photo collections demonstrate that our method can hallucinate the desired appearances and render occlusion-free images from different views.


Appearance Hallucination


Brandenburg Gate

Trevi Fountain


Cross-Appearance Hallucination


From Trevi Fountain to Brandenburg Gate

From Brandenburg Gate to Trevi Fountain


Appearance Hallucination From Artworks



Citation


@inproceedings{chen2022hallucinated,
  title={Hallucinated neural radiance fields in the wild},
  author={Chen, Xingyu and Zhang, Qi and Li, Xiaoyu and Chen, Yue and Feng, Ying and Wang, Xuan and Wang, Jue},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={12943--12952},
  year={2022}
}