CVPR 2021 Tutorial on
PhysicsBased Differentiable Rendering
June 20, 2021
Location
CVPR Zoom (Need to be registered to join.)Time
June 20, 202111 am2 pm PT, 25 pm ET
Agenda

Introduction
 What is differentiable rendering (DR)
 Applications of DR
 Why is physicsbased DR difficult
 Discussions & Common misconceptions

Differentiable rendering theory and algorithms
 Direct illumination, differentiating integrals with respect to different types of parameters, handling discontinuities
 Algorithms for handling global illumination, edge sampling, pathspace methods
 Reparameterization, warpedarea sampling
 Systematic differentiation of discontinuities
 Differentiable rendering systems and applications
 Q&A
Tutorial Recording
Takehome messages

Great progress has been made in physicsbased differentiable rendering
 Now capable of handling global illumination, arbitrary camera types (e.g., transient), and global scene parameters (e.g., object geometry) with decent efficiency
 Can be applied to solve many general inverse problems

Ray tracing is no longer slow
 Many efficient systems are being actively developed (e.g., Redner, PSDRCUDA, Mitsuba 2, Teg)
 Differentiable rendering is usually not the performance bottleneck

Gradient accuracy matters!
 Approximated gradients can yield reduced result quality

Discontinuities always exist (due to visibility) and need to be properly handled
 Autodiffing a path tracer may not always work
Materials
 Tutorial slides: PDF (90 MB)
References

Background on physicsbased forward rendering
 Direct illumination: Veach and Guibas, “Optimally combining sampling techniques for Monte Carlo rendering”, SIGGRAPH 1995
 Path integral for global illumination: Veach, “Robust Monte Carlo methods for light transport simulation,” PhD Thesis 1998

Discontinuities in direct illumination
 Explicit surfaces: Ramamoorthi et al., “A firstorder analysis of lighting, shading, and shadows,” TOG 2007
 Implicit surfaces: Gargallo et al., “Minimizing the reprojection error in surface reconstruction from images,” ICCV 2007

Discontinuities in global illumination and edge sampling
 Surface light transport: Li et al., “Differentiable Monte Carlo ray tracing through edge sampling,” SIGGRAPH Asia 2018
 Volumetric light transport: Zhang et al., “A differential theory of radiative transfer,” SIGGRAPH Asia 2019

Path integral for differentiable rendering
 Surface light transport: Zhang et al., “Pathspace differentiable rendering,” SIGGRAPH 2020
 Volumetric light transport: Zhang et al., “PathSpace differentiable rendering of participating media,” SIGGRAPH 2021

Reparameterization techniques for differentiable rendering
 Smooth visibility: Loubet et al., “Reparameterizing discontinuous integrands for differentiable rendering,” SIGGRAPH Asia 2019
 Warped area sampling: Bangaru et al., “Unbiased warpedarea sampling for differentiable rendering,” SIGGRAPH Asia 2020

Differentiable rendering for local parameters
 Score estimator (original): Khungurn et al., “Matching real fabrics with microappearance models,” TOG 2015
 Score estimator (more general discussion): Gkioulekas et al., “An evaluation of computational imaging techniques for heterogeneous inverse scattering,” ECCV 2016
 Radiative backpropagation: NimierDavid et al., “Radiative backpropagation: an adjoint method for lightningfast differentiable rendering,” SIGGRAPH 2020
 Primarysamplespace estimator: Zeltner et al., “Monte Carlo Estimators for Differential Light Transport,” SIGGRAPH 2021

Differentiable rendering and computation systems
 Redner: Li et al., “Differentiable Monte Carlo ray tracing through edge sampling,” SIGGRAPH Asia 2018
 Mitsuba 2: NimierDavid et al., “Mitsuba 2: A retargetable forward and inverse renderer,” SIGGRAPH Asia 2019
 PSDRCUDA: Luan et al., “Unified shape and SVBRDF recovery using differentiable Monte Carlo rendering,” EGSR 2021
 Teg: Bangaru et al., “Systematically differentiating parametric discontinuities,” SIGGRAPH 2021

Shape and reflectance
 Diffuse shape from interreflections: Nayar et al., “Shape from interreflections,” IJCV 1991
 Multiview shape and SVBRDF: Luan et al., “Unified shape and SVBRDF recovery using differentiable Monte Carlo rendering,” EGSR 2021
 BRDF from interreflections: ShemTov et al., “Towards reflectometry from interreflections,” ICCP 2019

Inverse scattering
 Homogeneous inverse scattering: Gkioulekas et al., “Inverse volume rendering with material dictionaries,” SIGGRAPH Asia 2013
 Learningbased inverse scattering: Che et al., “Towards learningbased inverse subsurface scattering,” ICCP 2019
 Heterogeneous inverse scattering: Gkioulekas et al., “An evaluation of computational imaging techniques for heterogeneous inverse scattering,” ECCV 2016
 Fabrics: Khungurn et al., “Matching real fabrics with microappearance models,” TOG 2015
 Cloud tomography: Levis et al., “Airborne threedimensional cloud tomography,” ICCV 2015
 Material fabrication: Nindel et al., “A gradientbased framework for 3D print appearance optimization,” SIGGRAPH 2021

Nonlineofsight imaging
 Shape and BRDF: Tsai et al., “Beyond volumetric albedoA surface optimization framework for nonlineofsight imaging,” CVPR 2019

Physicsbased learning
 Combine encoders and differentiable rendering: Che et al., “Towards learningbased inverse subsurface scattering,” ICCP 2019

Others
 Perceptual losses: Johnson et al., “Perceptual losses for realtime style transfer and superresolution,” ECCV 2016
 Optical gradient descent: Chen et al., “Autotuning structured light by optical stochastic gradient descent”, CVPR 2020
 Ambiguities between reflectance and illumination: Romeiro and Zickler, “Blind reflectometry,” ECCV 2010
 Ambiguities between shape and illumination: Xiong et al., “From shading to local shape,” PAMI 2014
 Ambiguities between scattering parameters: Zhao et al., “Highorder similarity relations in radiative transfer,” SIGGRAPH 2014
 Interreflections and generalized basrelief ambiguity: Chandraker et al., “Reflections on the generalized basrelief ambiguity,” CVPR 2005