Researchers at NVIDIA have been working on an incredible deep learning method that can edit images and can reconstruct corrupted images. When a certain image has missing pixels or holes, the AI is able to fill up the holes and reconstruct a realistic image. In a certain way you can also use it to edit images by removing parts of content, the AI will fill up the deleted parts with a realistic alternative.

In their research paper called 'Image Inpainting for Irregular Holes Using Partial Convolutions' researchers explain:

“Our model can robustly handle holes of any shape, size location, or distance from the image borders. Previous deep learning approaches have focused on rectangular regions located around the center of the image, and often rely on expensive post-processing,” [..] “Further, our model gracefully handles holes of increasing size.”

The researchers first generated 55,116 masks of random streaks and holes of arbitrary shapes and sizes for training their neural network. Another 25,000 were generated for testing the AI. The masks are categorized into six categories based on sizes relative to the input image in order to improve the accuracy for reconstruction.

Examples of masks generated for training:

The Research group is led by Guilin Liu who has a lot of experience in deep learning, geometry&graphics, computer vision and motion planning of robotics.

They also released a demonstration which is pretty fun to watch: