Update as of December, 2019: Mask-based foveated rendering for UE4 has been deprecated as of UE4 4.24. Please refer to a previous version of UE4 on the Oculus GitHub for a reference to the implementation.

Mask-based foveated rendering (MBFR) is a rendering technique that decreases the shading rate of the peripheral region of the eye buffers by dropping some of the pixels, based on a checkboard pattern. It allows you to effectively reduce pixel shading cost from world rendering, provides good visual quality when dropping 50% of the pixels, and is compatible with all D3D11 level hardware - independent to IHV extensions.

In this post, we will walk through a masked scene that demonstrates MBFR and discuss when MBFR is most useful.

A Masked Scene

Here is a masked scene which demonstrates the pixel dropping. There are four rings in the masked image. The center circle preserves all the pixels. The other three rings drop 25%, 50%, and 75% respectively. The pixels are dropped in 2x2 quads.

Here is the reconstructed image:

Here is the original full resolution scene we use as the ground truth:

Here is a side-by-side comparison (magnified). The top is the masked image, the middle is the reconstructed image, and the bottom is the ground truth image.

Here is another comparison:

Integrating MBFR

MBFR is currently included on Oculus GitHub UE4 Repository (GitHub login required). It works with Unreal Engine 4.19+, while you should review our Unreal Get Started Page for Rift if you haven’t already.

To activate MBFR in your project, follow these steps:

Check “Forward Shading” in the UE4 Editor Project Settings, Rendering section

Without restarting the UE4 Editor, check “[VR] Enable Mask-based Foveated Rendering” in Project Settings, Rendering/Experimental section

Restart the Editor as prompted. The Editor will take a while to rebuild the shaders

MBFR will be activated in your project when previewing or running in VR mode

Console Commands

Here are some console commands that can be helpful when testing MBFR and tweaking the quality levels:

vr.Foveated.Mask.Enable 1 – activate MBFR

vr.Foveated.Mask.Enable 0 – deactivate MBFR

vr.Foveated.Mask.VisualizeMode 1 – disable the pixel reconstruction to reveal the masks

vr.Foveated.Mask.VisualizeMode 0 – enable the pixel reconstruction

vr.Foveated.Mask.HighResFov – The FOV (in degrees) of the full resolution region. Default is 46

vr.Foveated.Mask.MediumResFov – The FOV (in degrees) of the region where 25% of pixels will be dropped. Default is 60

vr.Foveated.Mask.LowResFov – The FOV (in degrees) of the region where 50% of pixels will be dropped. Default is 78

The region outside the LowResFov will have 75% of the pixels dropped

Implementation

Pixel Dropping

The pixels to drop are culled through a mask bit in the stencil buffer. Here are the visibility patterns in each density region:

Reconstruction

The dropped pixels are reconstructed in the PostProcessing stage, when the scene textures are fetched.

Medium and High-Density Regions

We use the 4 neighboring pixels to reconstruct each dropped pixel in the medium- and high-density regions, where the pixel dropping rate is less than or equal to 50%.

The naïve formula is to set the dropped pixel A to the linear interpolation of B, C, D and E.

A = (C + B) * 0.667 * 0.5 + (D + E) * 0.333 * 0.5

But this doesn’t work well and causes a lot of aliasing in the high-contrast / high-frequency areas. The artifact can be easily observed in the screenshots below and caused heavy flickering in VR.

The visual quality can be greatly improved by applying the directional weights computed from the neighborhood luminance. It enhances the edges and reduces the aliasing.

Finally, we achieve a good overall reconstruction quality on the high/medium density regions. Here is a side-by-side comparison on the high-density region (25% of the pixels are dropped), where the center image shows the reconstructed result and the image on the right is the ground truth:

And here are two comparisons on the medium-density region (50% of the pixels are dropped) where the center image is the reconstructed result and the image on the right is the ground truth:

Low Density Region (75% pixel drop rate)

We have fewer known pixels in the low-density region, at the most peripheral area. We reconstruct each dropped pixel with linear interpolation of the two neighbor pixels.

A = P * 0.667 + Q * 0.333

B = P * 0.667 + T * 0.333

C = P * 0.667 + S * 0.333

Since we only have ¼ of the effective pixels, the level of fidelity of the reconstruction is limited, with significant quality loss and distortion.

Here is a side-by-side comparison with the pre-distortion images (left: masked, center: reconstructed, right: original):

Because the low-density region is at the peripheral area, where the effective pixel density is low. It reduces the quality loss, but does not eliminate all the distortion artifacts. Here is approximately the same area on the post-distortion images, which has become much smaller after distortion:

Coarse Quality Reconstruction

The high-fidelity mask reconstruction is GPU-expensive and not every post-processing stage requires this level of precision. So, we also provide a coarse-quality mask reconstruction routine which reduces the reconstruction cost to a single texture fetch, from 2 or 4 texture fetches in the regular reconstruction. The coarse quality reconstruction is currently used in Bloom, Depth of Field, Down-sampling, Velocity flattening, and Screen-space reflection.

Performance

MBFR reduces GPU pixel shading cost by dropping a subset of the pixels in the world rendering passes. But it also introduces extra cost in the post-processing passes for reconstructing the dropped pixels.

By testing on a 4.65ms scene of Robo Recall (on GTX1080), it reduced the overall GPU cost to 4.25ms (-9%). Specifically, it reduced the world rendering (BasePass) cost from 2.51ms to 2.19ms (-13%); and the PostProcessing cost was increased from 0.47ms to 0.65ms (+0.18ms).

Since the cost of PostProcessing is relatively fixed in a game, and the world rendering is dynamic, we can expect more than 10% GPU performance savings from a scene with heavy world objects rendering.

Of course, the overall GPU performance is also heavily dependent on the FOVs of the density rings. An effective way to further optimize the GPU cost from MBFR is to configure them according to the content being rendered on each map. We can apply larger density rings in maps which contain high-frequency objects/textures to preserve the visual quality (like the cityscape maps in Robo Recall we tested), and reduce the size of density rings in other maps to receive the greatest performance benefit.

Performance Tips

Adding a separate reconstruction pass is usually much more expensive than combining the per-pixel reconstruction to the post-processing.

Although the mask can be generated in either depth or stencil buffer, using stencil is more optimal since it won’t break the Hi-Depth. Additionally, we can perform the full depth pass and avoid reconstructing the depth buffer too.

The major bottleneck in the pixel reconstruction is the texture fetching. Fetching more neighbor pixels (or the historical textures) in the reconstruction would give us better visual quality with the cost of GPU performance.

Conclusion

MBFR is not a free lunch. It reduces the pixel shading cost from world rendering but adds extra post-processing cost from the pixel reconstruction. Most VR projects are not using heavy post-processing. But to projects which use heavy post processing but little world rendering, MBFR may not bring any performance benefit.

The perceptual visual quality of MBFR can vary significantly based on the style of the content. The perceptual quality can be quite good when rendering low-contrast, low-frequency contents, but would be less optimal when there are a lot of high-frequency details in the content. Tweaking the radius of the foveation density rings would be necessary to balance the performance and perceptual visual quality according to the content being rendered.

Expected outcomes from using MBFR

Effectively reduces the pixel shading cost from the world rendering

Compatible with all D3D11 level hardware. Independent to IHV private extensions

Relatively temporally / spatially stable

Good visual quality when dropping 50% of the pixels or less

Able to combine with existing AA techniques, including MSAA and TAA

Considerations to keep in mind

Could result in extra performance cost in reconstructing the dropped pixels

Culling some of the pixels on every wavefront is not necessarily optimal with current GPU architectures. So dropping 50% of pixels may not provide 2x performance.

Potential quality loss when dropping 75% of the pixels

May not work effectively on Mobile GPUs with tiled on-chip memory

Additional Resources