CVPR 2021 Tutorial on
Physics-Based Differentiable Rendering
June 20, 2021

Organizers

Shuang Zhao
University of California, Irvine
Ioannis Gkioulekas
Carnegie Mellon University

Invited Speaker

Sai Bangaru
Massachusetts Institute of Technology

Location

CVPR Zoom (Need to be registered to join.)

Time

June 20, 2021
11 am--2 pm PT, 2--5 pm ET

Agenda

  • Introduction (60 minutes)
    • What is differentiable rendering (DR)
    • Applications of DR
    • Why is physics-based DR difficult
    • Discussions & Common misconceptions
  • Differentiable rendering theory and algorithms (60 minutes)
    • Direct illumination, differentiating integrals with respect to different types of parameters, handling discontinuities
    • Algorithms for handling discontinuities, edge sampling, PSDR (material-form), global illumination
    • Reparameterization, warped area sampling
    • Systematic differentiation of discontinuities
  • Differentiable rendering systems and applications (30 minutes)
  • Q&A session (30 minutes)

Tutorial Recording

Take-home messages

  • Great progress has been made in physics-based differentiable rendering
    • Now capable of handling global illumination, arbitrary camera types (e.g., transient), and global scene parameters (e.g., object geometry) with decent efficiency
    • Can be applied to solve many general inverse problems
  • Ray tracing is no longer slow
    • Many efficient systems are being actively developed (e.g., Redner, PSDR-CUDA, Mitsuba 2, Teg)
    • Differentiable rendering is usually not the performance bottleneck
  • Gradient accuracy matters!
    • Approximated gradients can yield reduced result quality
  • Discontinuities always exist (due to visibility) and need to be properly handled
    • Auto-diffing a path tracer may not always work

Materials

  • Tutorial slides: PDF (90 MB)

References

  • Background on physics-based forward rendering
    • Direct illumination: Veach and Guibas, “Optimally combining sampling techniques for Monte Carlo rendering”, SIGGRAPH 1995
    • Path integral for global illumination: Veach, “Robust Monte Carlo methods for light transport simulation,” PhD Thesis 1998
  • Discontinuities in direct illumination
    • Explicit surfaces: Ramamoorthi et al., “A first-order analysis of lighting, shading, and shadows,” TOG 2007
    • Implicit surfaces: Gargallo et al., “Minimizing the reprojection error in surface reconstruction from images,” ICCV 2007
  • Discontinuities in global illumination and edge sampling
    • Surface light transport: Li et al., “Differentiable Monte Carlo ray tracing through edge sampling,” SIGGRAPH Asia 2018
    • Volumetric light transport: Zhang et al., “A differential theory of radiative transfer,” SIGGRAPH Asia 2019
  • Path integral for differentiable rendering
    • Surface light transport: Zhang et al., “Path-space differentiable rendering,” SIGGRAPH 2020
    • Volumetric light transport: Zhang et al., “Path-Space differentiable rendering of participating media,” SIGGRAPH 2021
  • Reparameterization techniques for differentiable rendering
    • Smooth visibility: Loubet et al., “Reparameterizing discontinuous integrands for differentiable rendering,” SIGGRAPH Asia 2019
    • Warped area sampling: Bangaru et al., “Unbiased warped-area sampling for differentiable rendering,” SIGGRAPH Asia 2020
  • Differentiable rendering for local parameters
    • Score estimator (original): Khungurn et al., “Matching real fabrics with micro-appearance models,” TOG 2015
    • Score estimator (more general discussion): Gkioulekas et al., “An evaluation of computational imaging techniques for heterogeneous inverse scattering,” ECCV 2016
    • Radiative backpropagation: Nimier-David et al., “Radiative backpropagation: an adjoint method for lightning-fast differentiable rendering,” SIGGRAPH 2020
    • Primary-sample-space estimator: Zeltner et al., “Monte Carlo Estimators for Differential Light Transport,” SIGGRAPH 2021
  • Differentiable rendering and computation systems
    • Redner: Li et al., “Differentiable Monte Carlo ray tracing through edge sampling,” SIGGRAPH Asia 2018
    • Mitsuba 2: Nimier-David et al., “Mitsuba 2: A retargetable forward and inverse renderer,” SIGGRAPH Asia 2019
    • PSDR-CUDA: Luan et al., “Unified shape and SVBRDF recovery using differentiable Monte Carlo rendering,” EGSR 2021
    • Teg: Bangaru et al., “Systematically differentiating parametric discontinuities,” SIGGRAPH 2021
  • Shape and reflectance
    • Diffuse shape from interreflections: Nayar et al., “Shape from interreflections,” IJCV 1991
    • Multi-view shape and SVBRDF: Luan et al., “Unified shape and SVBRDF recovery using differentiable Monte Carlo rendering,” EGSR 2021
    • BRDF from interreflections: Shem-Tov et al., “Towards reflectometry from interreflections,” ICCP 2019
  • Inverse scattering
    • Homogeneous inverse scattering: Gkioulekas et al., “Inverse volume rendering with material dictionaries,” SIGGRAPH Asia 2013
    • Learning-based inverse scattering: Che et al., “Towards learning-based inverse subsurface scattering,” ICCP 2019
    • Heterogeneous inverse scattering: Gkioulekas et al., “An evaluation of computational imaging techniques for heterogeneous inverse scattering,” ECCV 2016
    • Fabrics: Khungurn et al., “Matching real fabrics with micro-appearance models,” TOG 2015
    • Cloud tomography: Levis et al., “Airborne three-dimensional cloud tomography,” ICCV 2015
    • Material fabrication: Nindel et al., “A gradient-based framework for 3D print appearance optimization,” SIGGRAPH 2021
  • Non-line-of-sight imaging
    • Shape and BRDF: Tsai et al., “Beyond volumetric albedo---A surface optimization framework for non-line-of-sight imaging,” CVPR 2019
  • Physics-based learning
    • Combine encoders and differentiable rendering: Che et al., “Towards learning-based inverse subsurface scattering,” ICCP 2019
  • Others
    • Perceptual losses: Johnson et al., “Perceptual losses for real-time style transfer and super-resolution,” ECCV 2016
    • Optical gradient descent: Chen et al., “Auto-tuning structured light by optical stochastic gradient descent”, CVPR 2020
    • Ambiguities between reflectance and illumination: Romeiro and Zickler, “Blind reflectometry,” ECCV 2010
    • Ambiguities between shape and illumination: Xiong et al., “From shading to local shape,” PAMI 2014
    • Ambiguities between scattering parameters: Zhao et al., “High-order similarity relations in radiative transfer,” SIGGRAPH 2014
    • Interreflections and generalized bas-relief ambiguity: Chandraker et al., “Reflections on the generalized bas-relief ambiguity,” CVPR 2005