CVPR 2021 Tutorial on
Physics-Based Differentiable Rendering
June 20, 2021
Location
CVPR Zoom (Need to be registered to join.)Time
June 20, 202111 am--2 pm PT, 2--5 pm ET
Agenda
-
Introduction
- What is differentiable rendering (DR)
- Applications of DR
- Why is physics-based DR difficult
- Discussions & Common misconceptions
-
Differentiable rendering theory and algorithms
- Direct illumination, differentiating integrals with respect to different types of parameters, handling discontinuities
- Algorithms for handling global illumination, edge sampling, path-space methods
- Reparameterization, warped-area sampling
- Systematic differentiation of discontinuities
- Differentiable rendering systems and applications
- Q&A
Tutorial Recording
Take-home messages
-
Great progress has been made in physics-based differentiable rendering
- Now capable of handling global illumination, arbitrary camera types (e.g., transient), and global scene parameters (e.g., object geometry) with decent efficiency
- Can be applied to solve many general inverse problems
-
Ray tracing is no longer slow
- Many efficient systems are being actively developed (e.g., Redner, PSDR-CUDA, Mitsuba 2, Teg)
- Differentiable rendering is usually not the performance bottleneck
-
Gradient accuracy matters!
- Approximated gradients can yield reduced result quality
-
Discontinuities always exist (due to visibility) and need to be properly handled
- Auto-diffing a path tracer may not always work
Materials
- Tutorial slides: PDF (90 MB)
References
-
Background on physics-based forward rendering
- Direct illumination: Veach and Guibas, “Optimally combining sampling techniques for Monte Carlo rendering”, SIGGRAPH 1995
- Path integral for global illumination: Veach, “Robust Monte Carlo methods for light transport simulation,” PhD Thesis 1998
-
Discontinuities in direct illumination
- Explicit surfaces: Ramamoorthi et al., “A first-order analysis of lighting, shading, and shadows,” TOG 2007
- Implicit surfaces: Gargallo et al., “Minimizing the reprojection error in surface reconstruction from images,” ICCV 2007
-
Discontinuities in global illumination and edge sampling
- Surface light transport: Li et al., “Differentiable Monte Carlo ray tracing through edge sampling,” SIGGRAPH Asia 2018
- Volumetric light transport: Zhang et al., “A differential theory of radiative transfer,” SIGGRAPH Asia 2019
-
Path integral for differentiable rendering
- Surface light transport: Zhang et al., “Path-space differentiable rendering,” SIGGRAPH 2020
- Volumetric light transport: Zhang et al., “Path-Space differentiable rendering of participating media,” SIGGRAPH 2021
-
Reparameterization techniques for differentiable rendering
- Smooth visibility: Loubet et al., “Reparameterizing discontinuous integrands for differentiable rendering,” SIGGRAPH Asia 2019
- Warped area sampling: Bangaru et al., “Unbiased warped-area sampling for differentiable rendering,” SIGGRAPH Asia 2020
-
Differentiable rendering for local parameters
- Score estimator (original): Khungurn et al., “Matching real fabrics with micro-appearance models,” TOG 2015
- Score estimator (more general discussion): Gkioulekas et al., “An evaluation of computational imaging techniques for heterogeneous inverse scattering,” ECCV 2016
- Radiative backpropagation: Nimier-David et al., “Radiative backpropagation: an adjoint method for lightning-fast differentiable rendering,” SIGGRAPH 2020
- Primary-sample-space estimator: Zeltner et al., “Monte Carlo Estimators for Differential Light Transport,” SIGGRAPH 2021
-
Differentiable rendering and computation systems
- Redner: Li et al., “Differentiable Monte Carlo ray tracing through edge sampling,” SIGGRAPH Asia 2018
- Mitsuba 2: Nimier-David et al., “Mitsuba 2: A retargetable forward and inverse renderer,” SIGGRAPH Asia 2019
- PSDR-CUDA: Luan et al., “Unified shape and SVBRDF recovery using differentiable Monte Carlo rendering,” EGSR 2021
- Teg: Bangaru et al., “Systematically differentiating parametric discontinuities,” SIGGRAPH 2021
-
Shape and reflectance
- Diffuse shape from interreflections: Nayar et al., “Shape from interreflections,” IJCV 1991
- Multi-view shape and SVBRDF: Luan et al., “Unified shape and SVBRDF recovery using differentiable Monte Carlo rendering,” EGSR 2021
- BRDF from interreflections: Shem-Tov et al., “Towards reflectometry from interreflections,” ICCP 2019
-
Inverse scattering
- Homogeneous inverse scattering: Gkioulekas et al., “Inverse volume rendering with material dictionaries,” SIGGRAPH Asia 2013
- Learning-based inverse scattering: Che et al., “Towards learning-based inverse subsurface scattering,” ICCP 2019
- Heterogeneous inverse scattering: Gkioulekas et al., “An evaluation of computational imaging techniques for heterogeneous inverse scattering,” ECCV 2016
- Fabrics: Khungurn et al., “Matching real fabrics with micro-appearance models,” TOG 2015
- Cloud tomography: Levis et al., “Airborne three-dimensional cloud tomography,” ICCV 2015
- Material fabrication: Nindel et al., “A gradient-based framework for 3D print appearance optimization,” SIGGRAPH 2021
-
Non-line-of-sight imaging
- Shape and BRDF: Tsai et al., “Beyond volumetric albedo---A surface optimization framework for non-line-of-sight imaging,” CVPR 2019
-
Physics-based learning
- Combine encoders and differentiable rendering: Che et al., “Towards learning-based inverse subsurface scattering,” ICCP 2019
-
Others
- Perceptual losses: Johnson et al., “Perceptual losses for real-time style transfer and super-resolution,” ECCV 2016
- Optical gradient descent: Chen et al., “Auto-tuning structured light by optical stochastic gradient descent”, CVPR 2020
- Ambiguities between reflectance and illumination: Romeiro and Zickler, “Blind reflectometry,” ECCV 2010
- Ambiguities between shape and illumination: Xiong et al., “From shading to local shape,” PAMI 2014
- Ambiguities between scattering parameters: Zhao et al., “High-order similarity relations in radiative transfer,” SIGGRAPH 2014
- Interreflections and generalized bas-relief ambiguity: Chandraker et al., “Reflections on the generalized bas-relief ambiguity,” CVPR 2005