Hello, my name is Sayantan.
This is a work in collaboration with Carl Marshall, Derek Nowrouzezahrai,
Zhao Dong, and Zhengqin Li.
We present a novel graphics primitive that uses
a network of memory instead of connected layers of perceptrons.
One of the primary advantage of using our technique is that it
makes inferencing more efficient compared to traditional multi-layer perceptrons.
This paper shows a variety of application of our technique in the context of neural graphics
pipeline, such as Textures, Signed Distance Fields, Shading and Neural Radiance Fields.
We note that our technique is very general and may extend easily to
other applications in graphics and beyond.
At a high level, our technique blurs the line between learning
and memorization. We hope to spur more interesting discussions and ideas in
this context and in the context of neural networks in general.
Next, we introduce our technique differentiable indirection. It is
similar to pointer indirection, where we query a memory location
that points to another memory location containing the final output. However,
in our case, we learn the pointer values using gradient descent.
Both pointers and the outputs are stored on multi-dimensional
regular grids. Since the array cells are discrete,
we linearly interpolate the neighbouring cells to enable continuous output and backpropagation.
A key feature of such array is that they are fully differentiable. That is,
we compute the gradient not only w.r.t to the content of the table but also with respect to the
array coordinates. The latter enables learning of the pointers as stored in the first array.
We call the first array primary,
as shown in orange, and the second array as cascaded, highlighted in blue.
We use a combine primary and cascaded in a variety of configuration to enable various applications.
To better understand our technique, we highlight one simple application
of differentiable indirection in the context the context of PBR shading.
Specular BRDF lobes are often modelled using isotropic-GGX,
as described by the following equation. The equation has two
scalar inputs alpha and h; and outputs a single-channel, non-linear HDR value.
Here, we use differentiable indirection to approximate the function. The two inputs
to the function are used as the coordinates to sample the primary array; a 2D with 2-channels.
The primary array then returns the 2D coordinates to sample the cascaded array;
which contains the final output.
In this slide, we visualize the learned representation of
Isotropic-GGX as stored in the primary and cascaded arrays.
Here, we compare the analytic and approximate output on a
scene rendered with a variety of sharp and diffuse specular lobes.
The following plot shows the key takeaways of our approach.
We compare the resource utilization
for various alternative neural architectures for the same task.
Note that our technique requires fewer compute flops and memory
bandwidth compared to MLPs and lesser parameter space compared to single level lookup-table.
Next, we next show a variety of other applications of differentiable indirection.
We start with our SDF representation. Our technique uses a 3D primary
and a 3D cascaded array. We query the primary with a
spatial coordinate and the representation learns a mapping from spatial coordinates
to binary density, either a plus or a minus 1. Here, positive, and negative indicates exterior
and interior of a geometry respectively. The output of the representation is used
to construct a 3D mesh using marching cubes algorithm.
The video compares a reference Kd-tree oracle
on left, with our technique and multi-resolution hash-encoding.
Notice the piano keys and the space between the two plates of the piano cover.
The next application uses differentiable indirection for image compression.
Here we use uv-coordinates to query a 2D primary
with 4 channels while the cascaded is 4D array storing the output RGB values.
Our empirical observations show that increasing the dimensionality of the
cascaded array improves the quality of output at a given compression.
Here we show the output of our technique at 6x compression.
Next, we make a small modification to the compression-only network to also incorporate
texture sampling based on pixel footprint. The network additionally uses a
pixel-footprint estimate to output an alias-free, LOD appropriate RGB output.
We show its application in regular albedo textures and normal maps.
Notice how seamlessly we transition from coarse LOD to fine LOD using
our technique while compressing the representation by 6x at the same time.
We also perform a uv stress test comparing our technique with ASTC nearest neighbor sampling.
Our technique produces slightly blurrier (refer to uncompressed video link at the end) results at coarse LODs closer to the distant horizon.
ASTC produces much nosier results at comparable sample count.
Next application shows the approximation of Disney BRDF using our approach.
Similar to isotropic GGX, Disney BRDF models the appearance of a variety of complex materials.
In this application we split primary and cascaded into an offline-encoder
and a runtime-decoder to improve efficiency. The offline encoder encodes artist-control
parameters, which are often stored as regular textures into another set of encoded textures.
At runtime these encoded textures are used by the decoder.
Again, we compare our technique with the reference implementation.
Next, we show the an end-to-end combined texture
sampling and Disney shading for two different scenes.
We move to our final application in Radiance Field compression. Here,
we use two independent networks to compress the density and the radiance field. Both
networks are queried by a 3D spatial coordinate, One outputs a scalar density
and the other a 12 channel latent vector. The latent vector is then
combined with view-directions using positional encoding to output a view dependent RGB color.
The video shows the comparison between reference uncompressed grid on the top-left
and multi-resolution hash encoding compressed grid on the bottom-left.
Thus we conclude our video demonstration.
Thank you for watching.