Greetings everyone, my name is Sayantan. I am excited present our paper Efficient
Graphics Representation with Differentiable Indirection.
This is a work in collaboration with Carl Marshall, Derek Nowrouzezahrai, Zhao Dong
and Zhengqin Li. Let us start with
what differentiable indirection is. So differentiable indirection is very
much like pointer indirection, where we query a location using an index or an address
which stores the address to the next location which contains the final output.
However, in our case we learn these pointer values using gradient descent.
Next, we look at why one should be excited to use differentiable indirection.
The first reason is efficiency. Compared to Multi-layered perceptrons
our technique is order of magnitude more compute efficient,
and compared to feature grid representations, we are order of magnitude more space efficient.
Another reason to use to differentiable indirection is its flexibility. We may use the
technique in a variety applications such as Signed Distance Field representations,
texture compression, Texture filtering,
Complex BRDFs, and for compressing radiance fields.
Not only that, but our technique is also very generic and potentially extends to applications
beyond graphics. Finally,
our technique exhibits properties of adaptive spatial resolution using a single indirection.
In contrast, tree based representations such as an oct-tree or a kd-tree
require several levels of indirections to produce good quality results.
This translates to our representation being more memory coherent compared
to tree-based representations. So to summarize, our technique is adaptive,
efficient and flexible, with wide variety of applications in graphics and beyond.
Next, we look at the technical details of differentiable indirection.
At minimum, our technique requires two arrays. We call the first array storing the
pointers as primary, and the second array,
storing the output values, as cascaded. In the forward pass, we first query the primary
array with an input coordinate "x", and fetch the cells bounding the
query location. We pass the cell values
through a constraining non-linearity F. The purpose of the non-linearity is to bound
the output values between 0 to 1 such that it points to a valid memory location in the
cascaded array. At this juncture,
we note the constraining non-linearity can be baked directly into the array cells
during inference for additional efficiency. Since the output of the array-cells are bounded,
the baked-in values can be easily quantized for more efficiency.
Both these optimizations are extensively used in our applications.
And finally, we interpolate the adjacent cells to obtain,
the pointer value "y". Although we use linear interpolants, we note that any higher order
interpolant may be suitable in this context as long as the interpolant is differentiable.
Next, we use the pointer "y" to query the cascaded array.
We again fetch bounding cells and interpolate the values to obtain the final output.
For the backward pass, we compare the output with the target
and compute a loss which we then use for backpropagation through the arrays.
We compute the gradient of the loss w.r.t the output and
at the first node, we multiply the incoming gradient with
another gradient w.r.t. array cell content. This updates the cells in the cascaded array.
Note that, computing this gradient is not unique to our technique; in fact,
several prior work rely on this gradient to learn feature grid representations.
However, our technique uses a second set of gradient
w.r.t the input coordinate or the pointer value "y". This gradient is
used for backpropagation and for updating the primary array.
At the second node, following a similar recipe, we multiply the incoming gradient with another
gradient w.r.t the array cells. Finally, the other gradient w.r.t.
input coordinates can be used for multi-level indirection, if desired.
With this we conclude the technical summary of differentiable indirection.
Next, we look at an intuitive explanation of differentiable indirection.
We illustrate this in a 1D setting using a simple 1D array.
We begin by representing the array content as a histogram.
To obtain a continuous output, we interpolate the adjacent cells in the array as shown.
Next, we visualize the two gradients we compute at each node during backpropagation
The first gradient w.r.t to array cell contents,
and the second w.r.t. to array input coordinates.
The purpose of the first gradient is to update the array values while
the second gradient is used to learn pointer values in the primary array.
On our histogram, we represent the gradient w.r.t. to the cell values
using the vertical blue arrows as shown on left, while
the gradients w.r.t. to the input coordinate is shown in orange with horizontal arrows.
Let us take a moment to analyze the two gradients.
The gradient w.r.t to the cell content, indicated by blue arrows,
adjusts the cell heights in the histogram. But the gradient with respect to input coordinate
y is trying to adjust the cell widths. Let's look more closely at the second gradient.
On this slide, we highlight the intention to move the cell boundaries using orange arrows.
We also represent a hypothetical array in red with variable cell boundaries.
In such case, the gradients represented by the orange arrows should move the cell
boundaries on the red array as shown. In practice, the information regarding the
cell boundaries is stored as pointers in the primary array. The primary and the cascaded
array combined, represents the adaptive resolution as illustrated by the hypothetical red array.
With this we conclude our explanation regarding adaptive resolution in
differentiable indirection. Next, we look at a few exemplary
uses of differentiable indirection. Our technique is useful for compact SDF
representation where we use a simple network composed of a 3D primary and 3D cascaded array.
We can also use it for texture compression where the network is composed of a 2D
primary and 4D cascaded array. We noticed from our experiments
that a higher dimensional cascaded array yields better quality compression.
With a small modification to our compression only network, we can
jointly treat both compression and filtering. This technique is useful for real-time rendering
where we no longer need to have separate hardware for both compression and filtering.
In the context of shading, we can use our technique to represent the Isotropic GGX function,
used for modelling glossy surfaces. Similarly, we can also model a complex
BRDF such as Disney using our technique while being more efficient to evaluate
compared to the reference implementation. And finally, we apply our technique to radiance
field compression where we compress regular grids storing density and view dependent RGB
using two sets of 3D indirections. Next, we discuss empirical evidence of
efficiency, based on an example. We illustrate the efficiency of our
technique by approximating isotropic GGX, which is a 2D function with a highly
non-linear scalar output. Since it is a 2D function,
we visualize the function as monochromatic texture containing HDR values.
Such function is often used for shading glossy objects as shown on the right.
In this context, we compare and approximate Isotropic GGX using a
variety of neural techniques such as multi-layered perceptrons,
a single level interpolated lookup, A combination of single level lookup and MLP
and finally differentiable indirection. For this task differentiable indirection
requires 1 Kilobyte of parameter space, 36 FLOPs of compute and 16 bytes of memory transfer.
Compared to ours, an MLP only network requires 4 times more parameter space and 2 orders of
magnitude more compute and bytes transfer. Whereas a single level grid requires fewer
compute and byte transfer, the benefits are outweighed by the enormous parameter
space required by a single level lookup. Finally, a combination of MLP and lookup strikes
a better balance across the three criteria, but overall, our technique still performs better.
Next, we illustrate the efficiency of our technique using a few more examples.
For SDFs, we represent 12 GB worth of surface sample points in just 24 MB.
Then for texture compression, our scheme can compress several
times the raw size of an rgb texure. For real-time applications, we may also
use it for joint texture compression and filtering using a single network.
In the context of shading, we can efficiently approximate Disney BRDF by replacing 200+
FLOPs of compute with few lookups. Finally, differentiable indirection
can compress pre-trained radiance fields by orders of magnitude.
The video collage show differentiable indirection in action for a variety of applications.
Thus we end our presentation with the following conclusion:
that is, ifferentiable indirection
is an adaptive, highly efficient and flexible neural primitive with various applications
in the graphics and potentially beyond. Thank you for listening to our presentation
and please check our website for more details and videos.