[PYTHON] Touch the latest physics-based renderer Mitsuba2 (3) Differentiable rendering

What is Mitsuba2

A free physics-based renderer for academics. There are new features such as differentiable rendering and polarized rendering. Other related articles I wrote are as follows.

-Touch the latest physics-based renderer Mitsuba2 (1) Introduction -Touch the latest physics-based renderer Mitsuba2 (2) Move from Python -Touch the latest physics-based renderer Mitsuba2 (4) Polarized rendering -Difference between BRDF and polarized BRDF (pBRDF)

It is assumed that the installation of Mitsuba2 has been completed.

What is Differentiable Rendering?

It is a rendering that can propagate differential information as it is. When the rendering algorithm is the function $ f (x) $, the input scene is $ x $, and the rendering result is $ y $, the following holds.

y = f(x)

In a differentiable renderer, this function $ f $ is differentiated to find $ \ frac {dy} {dx} $, so how to change $ x $ to get the desired rendering result $ y $ It will automatically ask you what to do. This means that you can solve the complicated inverse problem in CG, that is, inverse rendering, which was difficult in the past.

That said, I think it's difficult to immediately imagine what you can do with it, so I'll explain it using a concrete example.

Condensing design

Consider an example of projecting RGB light onto a wall through a glass panel. It is an image of sandwiching glass between the projector and the wall.

At this time, give the image you want to project as $ y $ in advance. Then, the fine shape (height profile) $ x $ of the glass panel that realizes refraction to obtain the desired image (target image) $ y $ is automatically optimized by the gradient method (SGD, Adam, etc.). I can. In the past, such problems had to be optimized heuristically by turning the loop of optical system design → evaluation → feedback, or by using dedicated optical software with various restrictions. You can easily and universally optimize with this framework.

Reflectance optimization

The reflectance of the object (BSDF parameter) can also be optimized to meet the desired image. The figure below shows how to optimize the reflectance of the left wall, which is a parameter of the scene, with the rightmost image as the target image $ y $.

The red color is the true value on the left wall, but you can see that even if the reflectance is initialized with an appropriate value, it finally converges to the true value (the image is rough because the number of samplings is small, If you sample enough each time, it will take a lot of calculation time for differentiable rendering, so you don't increase the number of samplings much during optimization). Such automatic optimization of reflectance and BSDF parameters has traditionally been a very difficult problem due to the effects of mutual reflection. So, in the existing research, I managed to solve it by adding a big constraint, but in differentiable rendering, it can be solved directly by the straightforward method.

Light source optimization

Object reflectance and BSDF parameters are uniform for each basic object, and the number of parameters is small, so it can be regarded as a simple problem. Mitsuba2's differentiable rendering can automatically optimize not only single scalar values such as reflectance, but also high-dimensional data such as ambient light maps. The figure below is an example of convergence to the true value even if the light source is properly initialized.

The top is the environment light source map $ x $, and the bottom is the rendering result. You can see that the light source map, which was initially uniform and white, changes, and the rendered result approaches the target image $ y $. Given only one image example, it means that the same appropriate light source can be set automatically without adjustment by the creator.

In principle, the volume and shape can be automatically optimized. However, the shape etc. have not been implemented yet. According to the author, it will be implemented soon (https://github.com/mitsuba-renderer/mitsuba2/issues/26). I think you should check GitHub frequently.

Move differentiable rendering

The formula has prepared the sample code for optimizing the reflectance and the light source mentioned in the above example, so let's move it.

As I wrote in Introduction, it is necessary to have a GPU environment in order to perform differentiable rendering. Also, the variant specified in mistuba.conf must contain at least one differentiable configuration. First of all, I think that gpu_autodiff_rgb is good.

The following describes the procedure for Windows. I think it's almost the same for Linux. GPU rendering doesn't work on MacOS. In the same way as Move from Python, first pass the PATH to the mitsuba module in the root directory, and then use the sample code of docs \ examples \ 10_inverse_rendering \ Move

mitsuba2> setpath.bat
mitsuba2> cd docs\examples\10_inverse_rendering\
mitsuba2\docs\examples\10_inverse_rendering>

Reflectance optimization

First, let's optimize the reflectance. Bring the data to work.

git clone https://github.com/mitsuba-renderer/mitsuba-data.git

As mentioned before, please put the cbox of the above data under 10_inverse_rendering for each directory. If you execute the following and images are generated one after another, it is successful.

python invert_cbox.py

I will explain the contents of the code. I added a comment in Japanese.

#Simple reverse rendering example: Render the target image of a Cornell box,
#Optimize one of the scene parameters (left wall reflectance) using differentiable rendering and gradient-based optimization

import enoki as ek
import mitsuba
mitsuba.set_variant('gpu_autodiff_rgb')

from mitsuba.core import Thread, Color3f
from mitsuba.core.xml import load_file
from mitsuba.python.util import traverse
from mitsuba.python.autodiff import render, write_bitmap, Adam
import time

#Load the cornell box
Thread.thread().file_resolver().append('cbox')
scene = load_file('cbox/cbox.xml')

#Find differentiable scene parameters
params = traverse(scene)

#Discard all parameters except the one you want to differentiate
params.keep(['red.reflectance.value'])

#Print the current value and generate a backup
param_ref = Color3f(params['red.reflectance.value'])
print(param_ref)

#Render the target image (not yet using derivatives)
image_ref = render(scene, spp=8)
crop_size = scene.sensors()[0].film().crop_size()
write_bitmap('out_ref.png', image_ref, crop_size)

#Change the left wall to white (default)
params['red.reflectance.value'] = [.9, .9, .9]
params.update()

#Select Adam as the parameter optimization method
opt = Adam(params, lr=.2)

time_a = time.time()

#Number of iterations
iterations = 100

for it in range(iterations):
    #Perform differentiable rendering of the scene
    image = render(scene, optimizer=opt, unbiased=True, spp=1)
    write_bitmap('out_%03i.png' % it, image, crop_size)

    #Take loss (MSE between rendered image and target image)
    ob_val = ek.hsum(ek.sqr(image - image_ref)) / len(image)

    #Backpropagate the error to the input parameters
    ek.backward(ob_val)

    #Perform gradient steps
    opt.step()

    #Outputs the result of comparing the reflectance with the value of Ground Truth
    err_ref = ek.hsum(ek.sqr(param_ref - params['red.reflectance.value']))
    print('Iteration %03i: error=%g' % (it, err_ref[0]), end='\r')

time_b = time.time()

#Output the average calculation time for each iteration
print()
print('%f ms per iteration' % (((time_b - time_a) * 1000) / iterations))

Light source optimization

Next, optimize the light source map. Download and unzip the sample data here. It's a metal Bunny surrounded by a museum-like environment, and I think it contains mesh files and scene description files.

Do the following to start optimizing the light source.

python invert_bunny.py

I will also explain the contents of the code with comments.

import enoki as ek
import mitsuba
mitsuba.set_variant('gpu_autodiff_rgb')

from mitsuba.core import Float, Thread
from mitsuba.core.xml import load_file
from mitsuba.python.util import traverse
from mitsuba.python.autodiff import render, write_bitmap, Adam
import time

#scene(bunny)Read
Thread.thread().file_resolver().append('bunny')
scene = load_file('bunny/bunny.xml')

#Find differentiable scene parameters
params = traverse(scene)

#Make a backup
param_res = params['my_envmap.resolution']
param_ref = Float(params['my_envmap.data'])

#Discard all parameters except the one you want to differentiate
params.keep(['my_envmap.data'])

#Render the target image (not yet using derivatives)
image_ref = render(scene, spp=16)
crop_size = scene.sensors()[0].film().crop_size()
write_bitmap('out_ref.png', image_ref, crop_size)

#Changed to a unified white environment lighting map
params['my_envmap.data'] = ek.full(Float, 1.0, len(param_ref))
params.update()

#Select Adam as the parameter optimization method
opt = Adam(params, lr=.02)

time_a = time.time()

#Number of iterations
iterations = 100

for it in range(iterations):
    #Perform differentiable rendering of the scene
    image = render(scene, optimizer=opt, unbiased=True, spp=1)
    write_bitmap('out_%03i.png' % it, image, crop_size)
    write_bitmap('envmap_%03i.png' % it, params['my_envmap.data'],
                 (param_res[1], param_res[0]))

    #Take loss (MSE between rendered image and target image)
    ob_val = ek.hsum(ek.sqr(image - image_ref)) / len(image)

    #Backpropagate the error to the input parameters
    ek.backward(ob_val)

    #Perform gradient steps
    opt.step()

    #Outputs the result of comparing the environment light source map with the value of Ground Truth.
    err_ref = ek.hsum(ek.sqr(param_ref - params['my_envmap.data']))
    print('Iteration %03i: error=%g' % (it, err_ref[0]), end='\r')

time_b = time.time()

#Output the average calculation time for each iteration
print()
print('%f ms per iteration' % (((time_b - time_a) * 1000) / iterations))

my_envmap.data is the parameter of the environment light source map, which is a 125K RGBA bitmap (one-dimensionalized). Its 125K parameters can be optimized at the same time.

As you can see by comparing the above two sample codes, by changing the parameters you want to derive, you can optimize with almost the same code, and you can see that it can be described very generally.

Summary

We have summarized the principles and applications of differentiable rendering made possible by Mitsuba2, and even the execution of sample code. Inverse rendering, which was difficult as an inverse problem in the past, can now be solved with a very concise description (in many cases), so I personally think that it is a revolutionary technology.

Next time, I will introduce the newly added function of polarized rendering. -Touch the latest physics-based renderer Mitsuba2 (4) Polarized rendering

that's all.