Deep learning holds incredible promise for the next chapter of super resolution. Hand engineered algorithms are fast and can do a fair job of approximating AI. But they’re brittle and prone to failure in corner cases.

Take self-driving cars, for instance. For years, engineers tried to hand code every potential scenario an automobile could find itself in. But this approach failed given the massive number of variables. Even locating cars reliably in a video is a tough task since they can look different based on distance, lighting, color, and shape. Self-driving cars needed AI to progress.

While self-driving cars operate in a complex physical world, games run in large virtual worlds. Accurately anticipating the color and motion of every pixel across many games is too difficult to code by hand. Rather than solve the problem via infinite conditional statements, deep learning learns the algorithm from the data.

Deep learning-based super resolution learns from tens of thousands of beautifully rendered sequences of images, rendered offline in a supercomputer at very low frame rates and 64 samples per pixel. Deep neural networks are then trained to recognize what beautiful images look like. Then these networks reconstruct them from lower-resolution, lower sample count images. The neural networks integrate incomplete information from lower resolution frames to create a smooth, sharp video, without ringing, or temporal artifacts like twinkling and ghosting.

There are many other examples of how deep learning is used to create super resolution images and video, create new frames of video, or transfer an artist’s style from one image to the next. Before Turing, none of this was possible in real-time. With Turing’s Tensor Cores, 110 teraflops of dedicated horsepower can be applied for real-time deep learning.

Let’s look at an example of our image processing algorithm vs. our AI research model. The video below shows a cropped Unreal Engine 4 scene of a forest fire with moving flames and embers. Notice how the image processing algorithm blurs the movement of flickering flames and discards most flying embers. In contrast, you’ll notice that our AI research model captures the fine details of these moving objects.