Efficient Neural Graphics

Doctoral dissertation, Electrical and Computer Engineering, McGill University

Sayantan Datta

Abstract

Rapid increase in computational power have propelled both computer graphics and machine learning toward achieving increasingly complex objectives. The two fields have experienced a cross-pollination of ideas and technical expertise, advancing them collectively. Neural graphics explores the application of machine learning techniques in various aspects of computer graphics, including rendering, animation, view synthesis, and other visual tasks. It utilizes learned techniques to enhance traditional computer generated imagery (CGI) and visual content creation, while also enabling new pipelines for non-experts to create and consume immersive graphics experiences. The aim of neural graphics is to harness the power of data-driven models for enhancing bidirectional interactions between simulations and the real-world across various applications, from video games and virtual reality to computer aided design and digital art. These applications require efficient algorithms and paradigms to address their ever increasing computational demands.

 Efficiency is crucial for applications that must operate within strict energy budgets while maintaining desired performance and quality levels. Many real-time graphics applications demand precise refresh rates of 60 or 90 Hz, necessitating the development of specially crafted algorithms to meet the performance and efficiency targets. Efficient algorithms not only enhance existing applications but also enable new ones that were previously impractical due to energy or performance constraints. At its core, energy is essential for processing data through arithmetic manipulations or moving data between storage and processing elements. Often, the latter requires significantly more energy, and achieving maximum efficiency requires balancing these two aspects. In modern devices, memory operations are slower than arithmetic operations, and the scaling of memory subsystems lags behind arithmetic units with each new hardware generation. Consequently, more algorithms are becoming bottlenecked due to memory constraints rather than compute limitations. Hence, an efficient algorithm should not only require fewer computations but also minimize data movement, known as bandwidth. While neural networks have played a pivotal role in many graphics applications, their direct application in graphics may lead to inefficiencies due to their substantial bandwidth and compute requirements.

 This dissertation systematically addresses efficiency challenges by taking a bottomup approach at three levels: primitive, network, and application levels. Any efficiency improvement at a lower level permeates through the levels above. At the first level, we introduce a new primitive called differentiable indirection, which can be used to construct more complex networks or be combined with other neural primitives. Our novel primitive is more compact and intrinsically more compute and bandwidth efficient compared to other primitives, such as multilayer perceptrons or neural fields. It has been tested across various graphics tasks, including geometry representation, shading, texturing, radiance fields, and holds potential for applications beyond graphics. Moving to the network level, we demonstrate how to optimize a convolutional neural network to minimize bandwidth requirements, making it suitable for real-time graphics applications like shadow synthesis. Finally, at the application level, we optimize our pipeline to utilize smaller and more efficient networks in the context of shadow synthesis and soft-body animation. For shadow synthesis, we leverage domain knowledge to fine-tune input features, ensure more robust training for temporal stability, and perform network pruning based on the application environment. In the realm of soft-body simulation, we employ dimensionality reduction to develop a compact, reduced-space neural operator for the rapid synthesis of latent temporal trajectories.

[Thesis] [Bibtex]

Efficient Graphics Representation with Differentiable Indirection

ACM/SIGGRAPH Asia 2023 Conference Proceedings

Sayantan Datta, Carl Marshall, Derek Nowrouzezahrai, Zhao Dong, Zhengqin Li

Abstract

We introduce differentiable indirection – a novel learned primitive that employs differentiable multi-scale lookup tables as an effective substitute for traditional compute and data operations across the graphics pipeline. We demonstrate its flexibility on a number of graphics tasks, i.e., geometric and image representation, texture mapping, shading, and radiance field representation. In all cases, differentiable indirection seamlessly integrates into existing architectures, trains rapidly, and yields both versatile and efficient results.

[Paper] [Webpage] [Bibtex]

Adaptive Dynamic Global Illumination

High Performance Graphics 2022 Poster

Sayantan Datta, Negar Goli, Jerry Zhang

Abstract

We present an adaptive extension of probe based global illumination solution that enhances the response to dynamic changes in the scene while while also enabling an order of magnitude increase in probe count. Our adaptive sampling strategy carefully places samples in regions where we detect time varying changes in radiosity either due to a change in lighting, geometry or both. Even with large number of probes, our technique robustly updates the irradiance and visibility cache to reflect the most up to date changes without stalling the overall algorithm. Our bandwidth aware approach is largely an improvement over the original DDGI technique while also remaining orthogonal to the recent advancements in the technique.

[Paper] [Webpage] [Bibtex]

Neural Shadow Mapping

ACM/SIGGRAPH 2022 Conference Proceedings

Sayantan Datta, Derek Nowrouzezahrai, Christoph Schied, Zhao Dong

Abstract

We present a neural extension of basic shadow mapping for fast, high quality hard and soft shadows. We compare favorably to fast pre-filtering shadow mapping, all while producing visual results on par with ray traced hard and soft shadows. We show that combining memory bandwidth-aware architecture specialization and careful temporal-window training leads to a fast, compact and easy-to-train neural shadowing method. Our technique is memory bandwidth conscious, eliminates the need for post-process temporal anti-aliasing or denoising, and supports scenes with dynamic view, emitters and geometry while remaining robust to unseen objects.

[Paper] [Supplemental] [Webpage] [Bibtex]

Subspace Neural Physics: Fast Data-Driven Interactive Simulation

ACM/SIGGRAPH Symposium on Computer Animation (SCA)

Daniel Holden, Bang Chi Duong, Sayantan Datta, Derek Nowrouzezahrai

Abstract

Data-driven methods for physical simulation are an attractive option for interactive applications due to their ability to trade precomputation and memory footprint in exchange for improved runtime performance. Yet, existing data-driven methods fall short of the extreme memory and performance constraints imposed by modern interactive applications like AAA games and virtual reality. Here, performance budgets for physics simulation range from tens to hundreds of micro-seconds per frame, per object. We present a data-driven physical simulation method that meets these constraints. Our method combines subspace simulation techniques with machine learning which, when coupled, enables a very efficient subspace-only physics simulation that supports interactions with external objects – a longstanding challenge for existing sub-space techniques. We also present an interpretation of our method as a special case of subspace Verlet integration, where we apply machine learning to efficiently approximate the physical forces of the system directly in the subspace. We propose several practical solutions required to make effective use of such a model, including a novel training methodology required for prediction stability, and a GPU-friendly subspace decompression algorithm to accelerate rendering.

[Paper] [Bibtex]

A Numerical Study of Frictional Contact

Master's thesis, Computer Science, McGill University

Sayantan Datta

Abstract

Friction is a complex phenomenon resulting from elastic and plastic deformations coupled with molecular interaction along the contact boundary. When two surfaces touch, their roughness, and normal force determines the actual area under contact, governing the process of deformation and molecular interaction. A typical macroscopic interaction may involve millions of microscopic contacts and the aggregate of these forces give rise to the phenomenon of friction. In this thesis, our goal is to simulate the phenomenon of friction assuming unlubricated contact and elastic deformation at the contact asperities. We collect data by varying many parameters that affect friction between two surfaces and build a function approximator exploiting the correlation in data. Such an approximator is a computationally inexpensive, versatile and more accurate substitute for friction coefficient tables currently in use with various physically based simulators.

[Thesis] [Bibtex]

Reach me at