This is the sixth part of a tutorial series about creating a custom scriptable render pipeline. It uses shadow masks to bake shadows while still calculating realtime lighting.

This tutorial is made with Unity 2019.2.21f1.

Baking Shadows

The advantage of using a light map is that we're not limited to a max shadow distance. Baked shadows don't get culled, but they also cannot change. Ideally, we could use realtime shadows up to the max shadow distance and baked shadows beyond that. Unity's shadow mask mixed lighting mode makes this possible.

Distance Shadow Mask Let's consider the same scene from the previous tutorial, but with the max shadow distance reduced such that part of the structure's interior doesn't get shadowed. This makes it very clear where realtime shadows end. We start with only a single light source. Baked indirect mixed lighting, max distance 11. Switch the mixed lighting mode to Shadowmask. This will invalidate the lighting data so it'll have to get baked again. Shadowmask mixed lighting mode. There are two ways to use shadow mask mixed lighting, which can be configured via the Quality project settings. We'll use the Distance Shadowmask mode. The other mode is known as just Shadowmask, which we'll cover later. Shadow mask mode set to distance. The two flavors of shadow mask mode use the same baked lighting data. In both cases the light map ends up containing the indirect lighting, exactly the same as the Baked Indirect mixed lighting mode. What's different is that there's now also a baked shadow mask map, which you can inspect via the baked light map preview window.

Baked indirect light and shadow mask. The shadow mask map contains the shadow attenuation of our single mixed directional light, representing the shadows cast by all static objects that contribute to global illumination. The data is stored in the red channel so the map is black and red. Just like the baked indirect lighting the baked shadows cannot change at runtime. However, the shadows will remain valid no matter the intensity or color of the light. But the light should not be rotated otherwise its shadows won't make sense. Also, you shoudn't vary the light too much if its indirect lighting is baked. For example, it would be obviously wrong if indirect lighting remained after a light is turned off. If a light changes a lot then you could set its Indirect Multiplier to zero so no indirect light gets baked for it.

Detecting a Shadow Mask To use the shadow mask our pipeline must first know that it exists. As it's all about shadows that's the job of our Shadows class. We'll use shader keywords to control whether shadow masks are used. As there are two modes we'll introduce another static keyword array, even though it contains just one keyword for now: _SHADOW_MASK_DISTANCE. static string[] shadowMaskKeywords = { "_SHADOW_MASK_DISTANCE" }; Add a boolean field to track whether we're using a shadow mask. We re-evaluate this each frame so initialize it to false in Setup . bool useShadowMask; public void Setup (…) { … useShadowMask = false; } Enable or distable the keyword at the end of Render . We have to do this even if we end up not rendering any realtime shadows, because the shadow mask isn't realtime. public void Render () { if (shadowedDirLightCount > 0) { RenderDirectionalShadows(); } buffer.BeginSample(bufferName); SetKeywords(shadowMaskKeywords, useShadowMask ? 0 : -1); buffer.EndSample(bufferName); ExecuteBuffer(); } To know whether a shadow mask is needed we have to check if there is a light that uses it. We'll do this in ReserveDirectionalShadows , when we end up with a valid shadow-casting light. Each light contains information about its baked data. It's stored in a LightBakingOutput struct that can be retrieved via the Light.bakingOutput property. If we encounter a light with its light map bake type set to mixed and its mixed lighting mode set to shadow mask then we're using the shadow mask. public Vector3 ReserveDirectionalShadows ( Light light, int visibleLightIndex ) { if (…) { LightBakingOutput lightBaking = light.bakingOutput; if ( lightBaking.lightmapBakeType == LightmapBakeType.Mixed && lightBaking.mixedLightingMode == MixedLightingMode.Shadowmask ) { useShadowMask = true; } … } return Vector3.zero; } That enables the shader keyword when needed. Add a corresponding multi-compile directive for it to the CustomLit pass of the Lit shader. #pragma multi_compile _ _CASCADE_BLEND_SOFT _CASCADE_BLEND_DITHER #pragma multi_compile _ _SHADOW_MASK_DISTANCE #pragma multi_compile _ LIGHTMAP_ON

Shadow Mask Data On the shader side we have to know whether a shadow mask is in use and if so what the baked shadows are. Let's add a ShadowMask struct to Shadows to keep track of both, with a boolean and a float vector field. Name the boolean distance€ to indicate whether distance shadow mask mode is enabled. Then add this struct to the global ShadowData struct as a field. struct ShadowMask { bool distance€; float4 shadows; }; struct ShadowData { int cascadeIndex; float cascadeBlend; float strength; ShadowMask shadowMask; }; Initialize the shadow mask to not-in-use by default in GetShadowData . ShadowData GetShadowData (Surface surfaceWS) { ShadowData data; data.shadowMask.distance€ = false; data.shadowMask.shadows = 1.0; … } Although the shadow mask is used for shadowing it is part of the baked lighting data of the scene. As such retrieving it is the responsibility of GI. So add a shadow mask field to the GI struct as well and also initialize it to not-in-use in GetGI . struct GI { float3 diffuse; ShadowMask shadowMask; }; … GI GetGI (float2 lightMapUV, Surface surfaceWS) { GI gi; gi.diffuse = SampleLightMap(lightMapUV) + SampleLightProbe(surfaceWS); gi.shadowMask.distance€ = false; gi.shadowMask.shadows = 1.0; return gi; } Unity makes the shadow mask map available to the shader via a unity_ShadowMask texture and accompanying sampler state. Define those in GI along with the other light map texture and sampler state. TEXTURE2D(unity_Lightmap); SAMPLER(samplerunity_Lightmap); TEXTURE2D(unity_ShadowMask); SAMPLER(samplerunity_ShadowMask); Then add a SampleBakedShadows function that samples the map, using the light map UV coordinates. Just like for the regular light map this only makes sense for lightmapped geometry, so when LIGHTMAP_ON is defined. Otherwise there are no baked shadows and the attenuation is always 1. float4 SampleBakedShadows (float2 lightMapUV) { #if defined(LIGHTMAP_ON) return SAMPLE_TEXTURE2D( unity_ShadowMask, samplerunity_ShadowMask, lightMapUV ); #else return 1.0; #endif } Now we can adjust GetGI so it enables the distance shadow mask mode and samples the baked shadows if _SHADOW_MASK_DISTANCE is defined. Note that this makes the distance€ boolean a compile-time constant so its usage won't result in dynamic branching. GI GetGI (float2 lightMapUV, Surface surfaceWS) { GI gi; gi.diffuse = SampleLightMap(lightMapUV) + SampleLightProbe(surfaceWS); gi.shadowMask.distance€ = false; gi.shadowMask.shadows = 1.0; #if defined(_SHADOW_MASK_DISTANCE) gi.shadowMask.distance€ = true; gi.shadowMask.shadows = SampleBakedShadows(lightMapUV); #endif return gi; } It's up to Lighting to copy the shadow mask data from GI to ShadowData , in GetLighting before looping through the lights. At this point we can also debug the shadow mask data by directly returning it as the final lighting color. float3 GetLighting (Surface surfaceWS, BRDF brdf, GI gi) { ShadowData shadowData = GetShadowData(surfaceWS); shadowData.shadowMask = gi.shadowMask; return gi.shadowMask.shadows.rgb; … } Initially it doesn't appear to work, as everything ends up white. We have to instruct Unity to send the relevant data to the GPU, just like we did in the previous tutorial for the light map and probes in CameraRenderer.DrawVisibleGeometry . In this case we have to add PerObjectData.ShadowMask€ to the per-object data. perObjectData = PerObjectData.Lightmaps | PerObjectData.ShadowMask€ | PerObjectData.LightProbe | PerObjectData.LightProbeProxyVolume Sampling shadow mask. Why does Unity bake lighting each time we change shader code? That happens when we change HLSL files that are included by a meta pass. You can prevent needless baking by temporarily disabling Auto Generate.

Occlusion Probes We can see that the shadow mask gets applied to lightmapped objects correctly. We also see that dynamic objects have no shadow mask data, as expected. They use light probes instead of light maps. However, Unity also bakes shadow mask data into light probes, referring to it as occlusion probes. We can access this data by adding a unity_ProbesOcclusion vector to the UnityPerDraw buffer in UnityInput. Place it in between the world transform parameters and light map UV transformation vector. real4 unity_WorldTransformParams; float4 unity_ProbesOcclusion; float4 unity_LightmapST; Now we can simply return that vector in SampleBakedShadows for dynamic objects. float4 SampleBakedShadows (float2 lightMapUV) { #if defined(LIGHTMAP_ON) … #else return unity_ProbesOcclusion; #endif } Again, we have to instruct Unity to send this data to the GPU, this time by enabling the PerObjectData.OcclusionProbe flag. perObjectData = PerObjectData.Lightmaps | PerObjectData.ShadowMask€ | PerObjectData.LightProbe | PerObjectData.OcclusionProbe | PerObjectData.LightProbeProxyVolume€ Sampling occlusion probes. Unused channels of the shadow mask are set to white for probes, so dynamic objects end up white when fully lit and cyan when fully shadowed, instead of red and black. Although this is enough to get shadow masks working via probes, it breaks GPU instancing. The occlusion data can get instanced automatically, but UnityInstancing only does this when SHADOWS_SHADOWMASK is defined. So define it when needed in Common before including UnityInstancing. This is the only other place where we have to explicitly check whether _SHADOW_MASK_DISTANCE is defined. #if defined(_SHADOW_MASK_DISTANCE) #define SHADOWS_SHADOWMASK #endif #include "Packages/com.unity.render-pipelines.core/ShaderLibrary/UnityInstancing.hlsl"

LPPVs Light probe proxy volumes can also work with shadow masks. Again we have to enable this by setting a flag, this time PerObjectData.OcclusionProbeProxyVolume . perObjectData = PerObjectData.Lightmaps | PerObjectData.ShadowMask€ | PerObjectData.LightProbe | PerObjectData.OcclusionProbe | PerObjectData.LightProbeProxyVolume€ | PerObjectData.OcclusionProbeProxyVolume Retrieving the LPPV occlusion data works the same as retrieving its light data, except that we have to invoke SampleProbeOcclusion instead of SampleProbeVolumeSH4 . It's stored in the same texture and requires the same arguments, with the sole exception that a normal vector isn't needed. Add a branch for this to SampleBakedShadows , along with a surface parameter for the now required world position. float4 SampleBakedShadows (float2 lightMapUV , Surface surfaceWS ) { #if defined(LIGHTMAP_ON) … #else if (unity_ProbeVolumeParams.x) { return SampleProbeOcclusion( TEXTURE3D_ARGS(unity_ProbeVolumeSH, samplerunity_ProbeVolumeSH), surfaceWS.position, unity_ProbeVolumeWorldToObject, unity_ProbeVolumeParams.y, unity_ProbeVolumeParams.z, unity_ProbeVolumeMin.xyz, unity_ProbeVolumeSizeInv.xyz ); } else { return unity_ProbesOcclusion; } #endif } Add the new surface argument when invoking the function in GetGI . gi.shadowMask.shadows = SampleBakedShadows(lightMapUV , surfaceWS ); Sampling LPPV occlusion.