Abstract

We consider the challenging problem of predicting intrinsic object properties from a single image by exploiting differentiable renderers. Many previous learning-based approaches for inverse graphics adopt rasterization-based renderers and assume naive lighting and material models, which often fail to account for non-Lambertian, specular reflections commonly observed in the wild. In this work, we propose DIB-R++, a hybrid differentiable renderer which supports these photorealistic effects by combining rasterization and ray-tracing, taking the advantage of their respective strengths—speed and realism. Our renderer incorporates environmental lighting and spatially-varying material models to efficiently approximate light transport, either through direct estimation or via spherical basis functions. Compared to more advanced physics-based differentiable renderers leveraging path tracing, DIB-R++ is highly performant due to its compact and expressive shading model, which enables easy integration with learning frameworks for geometry, reflectance and lighting prediction from a single image without requiring any ground-truth. We experimentally demonstrate that our approach achieves superior material and lighting disentanglement on synthetic and real data compared to existing rasterization-based approaches and showcase several artistic applications including material editing and relighting.

Downloads

Publication

Code

  • Coming soon

BibTeX

Cite

Wenzheng Chen, Joey Litalien, Jun Gao, Zian Wang, Clément Fuji Tsang, Sameh Khamis, Or Litany, and Sanja Fidler. DIB-R++: Learning to Predict Lighting and Material with a Hybrid Differentiable Renderer. Neural Information Processing Systems, X(X), Article XX, December 2021. 
@inproceedings{chen2021dibrpp,
    title = {{DIB-R++}: Learning to Predict Lighting and Material with a Hybrid Differentiable Renderer,
    author = {Wenzheng Chen and
              Joey Litalien and
              Jun Gao and
              Zian Wang and
              Clement Fuji Tsang and
              Sameh Khalis and
              Or Litany and
              Sanja Fidler},
    year = {2021},
    journal = {Conference on Neural Information Processing Systems (NeurIPS)}
}

 Copy to clipboard