Lighting design in Virtual Reality : getting it to look like the real stuff

Side by side comparison of Unreal Engine lighting and Nvidia’s VXGI real-time global illumination

Ever since OneBonsai has started developing Virtual Reality on Unreal Engine, we have always struggled with the lighting of our scenes. Looking for solutions, we tried the Nvidia’s VXGI global illumination engine out.

Unreal’s default lighting engine (left) compared to Nvidia’s Global Illumination Engine (right)

Unreal’s lighting engine

Unreal uses a very powerful lighting engine that is geared towards real-time performance. To get the best frame rates possible, Unreal separated static lighting from dynamic lighting. All static lights will be used to precompute the illumination of your scene. This precomputed illumination is baked into the textures of the scene. In other words, static lighting is computed only once and baked into the experience.

Then, Unreal will calculate dynamic light sources in real-time that will shine light on top of the static illumination.

This combination of static and dynamic lights works great. For example, let’s imagine a scene with a table supporting a vase. The lamp will not move, so its shadow on the table won’t change either. For desktop games or experiences, where users only move around the virtual space and click on interactive objects, this is good. The lamp cannot be picked up by the user, therefore is not interactive. Users don’t even expect the lamp to be interactive.

In flat screen experiences, there is a clear division between what is static and what is dynamic. Experiences can be built around that.

In Virtual Reality, everything should be interactive

In Virtual Reality, it’s more complicated. When immersed, users explore and interact with the virtual space with their own hands. “Oh my… There is a pretty statue on that table! I’m going to try and grab it.” is the usual thinking pattern. We find ourselves in a situation where users want to seize, move and play with everything in the virtual world. Hence, any object in the scene could be moved by a user. The problem is that all these objects still have their lighting baked into their textures ! Thus, if a user moves the lamp, he/she will see a burn mark (see below) where the lamp used to be. In a nutshell, in VR, objects’ static lighting information (its shadows) are stationnary even if they themselves are mobile.

A solution would be to lock all the objects in place. Users in Virtual Reality can try and pick up the lamp, but the lamp won’t budge. That would be a terrible shame, since it would fracture the immersion. In the real world, if you pick a lamp up, you pick the lamp up. Typically, lamps are not fixed to tables with superglue. We want our users to feel ‘at home’ in our VR experiences, so we have to let them play around freely with the objects in the virtual space.

In virtual reality, there is no clear line between what should be static and what should be dynamic. Everything should be interactive.

Unreal’s dynamic lights don’t feel right

Now, let’s switch everything to dynamic lighting in Unreal engine. It just doesn’t feel natural. The illumination coming from dynamic lighting doesn’t diffuse light. As a result, anything that is not directly illuminated will be pitch black.

In real life, light propagates from a source to an object. Then, that object scatters this light back into the world. Then, this scattered light reaches another object, and is scattered again, etc etc. (We will refer to this phenomenon as “secondary reflection” in the following.) If we want our Virtual Reality experiences to feel natural, we have to reproduce that light behaviour. Unfortunately, Unreal doesn’t allow that.

Real-time global illumination

So, we had to look into a different lighting engine. Nvidia VXGI is a ‘voxel-cone-tracing’-based implementation of a global illumination algorithm. It recomputes the entire illumination of a scene at every frame, including the secondary light reflections on the surfaces of your virtual world.

Side by side comparison : Classic unreal lighting and nvidia VXGI

In the above animation, a nightfall scene is diplayed. On the left, rendered in real-time VR is the classic Unreal Engine lighting ; on the right, the Nvidia VXGI global illumination.

If you look closely, you will notice that the lighting of the landscape and the houses on the left image never change. Night falls, yet the houses stay lit as during the day. This is because this scene’s lighting is not recalculated during the experience. It is pre-computed.

On the right image, it is the Nvidia’s VXGI at play. Contrary to the Unreal engine lighting, here, it is completely recalculated at each frame. As a result, day time shifts swiftly into night time. The secondary light reflections appear as well, when an near by object lights up.

Which one feels best to you ? We find the realism of the VXGI real-time global illumination stunning.

Side by side VXGI without secondary reflections (no diffuse) and with secondary reflections (diffuse 2 bounces)

Secondary reflections have a huge impact on realism. In the above animation, you’ll notice that without some diffusion of light, any object that is not directly illuminated by the light source (the sun) will be pitch black. The house is the most obvious example and it is not right at all. With secondary reflections (right), you’ll see the house clearly because some sunlight bounced from the ground back to the house. At night time, you’ll see some light coming from the window bouncing back on the outside walls of the medieval cabin.

This is phenomenal for us. Thanks to VXGI, we can use real-time global illumination in our VR experiences. This means that our users can interact with any object in the scene!

Tech demo of Nvidia’s VXGI global illumination

In the above animation, you can see the impact of secondary reflections with a moving light source. When the ball shines blue light on the right-side walls, the left-side walls are indirectly illuminated.

If real-time global illumination yields such spectacular results, we should use it in all our projects, right ? Well, not really.

Is real-time global illumination fast enough for Virtual Reality ?

Real-time global illumination is extremely taxing on the GPU. All the demos shown here run in Virtual Reality on an HTC Vive at 90 fps on a 1080 Ti GPU. So it runs nicely but only on high performance hardware. You also have to be very careful about optimisation. Too many light sources will break performance. We found that 4 light sources or less runs fine.

References

If you want to have a closer look at our side by side comparisons, see them on our short Youtube video:

Side by side comparison of Unreal’s lighting engine and Nvidia’s real-time global illumination

If you want to dig deeper on how the lighting algorithm works, you can find the complete explanation by the people who created it here.

Where can you find us ?

Our website : onebonsai.com

Twitter : https://twitter.com/One_Bonsai

Facebook : https://www.facebook.com/onebonzai/

Linkedin : https://www.linkedin.com/company/17992275/

Blog : https://lab.onebonsai.com/