Question unity_SH (Spherical Harmonics) values are different on the CPU vs GPU?

Hi, the thread title is funky but can't sum it up quite well in a short way there...

Also, skip to the bottom of the post for the TLDR if you are lazy...

Context: I'm working on a capsule shadows solution for AO/Shadowing for VR, and in order to get the correct direction in which light sources are coming from I to have a couple of options in place. You can sample light directionality from a global vector, or you can sample it through the directional light maps that are baked within the scene. Heres are the results of the working effect basically



There is one additional directional mode that I'm working on which is that you can also sample directionality from light probes from which dynamic objects typically have access. It's perfectly reasonable and doesn't require creating a specific buffer for it like I have to do with the lightmap directionality. However, I'm running into a strange issue with the output which is leading to incorrect directionality in regards to the spherical harmonics and therefore incorrect results from which shadows appear to be casting in my capsule shadows-like solution.

Before we get into it I'm going to elaborate first on how you would go about it on the GPU/Shader side first, before showing how it's done on the CPU side and the problems that I've run into...

GPU/SHADER
In a shader typically unity provides you with the following variables for spherical harmonics which are already assigned and have the necessary data in them which come from UnityShaderVariables.cginc...

Code (CSharp):

1. // SH lighting environment
2. half4 unity_SHAr;
3. half4 unity_SHAg;
4. half4 unity_SHAb;
5. half4 unity_SHBr;
6. half4 unity_SHBg;
7. half4 unity_SHBb;
8. half4 unity_SHC;

It's worth noting also that these are obviously used in SHEvalLinearL0L1 which is found in UnityCG.cginc to sample the L0L1 bands from the spherical harmonics, and essentially using them to get the diffuse probe color, and it works...

Code (CSharp):

1. // normal should be normalized, w=1.0
2. half3 SHEvalLinearL0L1 (half4 normal)
3. {
4.     half3 x;
5.  
6.     // Linear (L1) + constant (L0) polynomial terms
7.     x.r = dot(unity_SHAr,normal);
8.     x.g = dot(unity_SHAg,normal);
9.     x.b = dot(unity_SHAb,normal);
10.  
11.     return x;
12. }

Now a bit unrelated (but I think worth mentioning to show my thought process basically), In an uber shader I wrote I simply rearranged the function and was able to use it as basically the dominant light direction for the probe in order to retain/compute specularity on objects that are purely baked or probe-lit...

Code (CSharp):

1. float3 sphericalHarmonicsL1_direction = unity_SHAr.xyz + unity_SHAg.xyz + unity_SHAb.xyz;

Now that ends the GPU side basically, I can get the dominant direction from the probe with that following line and do what I need with it and the direction is pretty much correct. I know this approach works and achieves the intended result. Now onto the CPU side which is where the problems lie...

CPU SIDE
Unfortunately on the CPU side, we aren't as lucky to have a bunch of Vector4s with the values spoon-fed to us. So we gotta go and get them...

Now the first thing we need to do is obviously sample the probe, on the CPU side we have to use LightProbes.GetInterpolatedProbe which will spit out a SphericalHarmonicsL2 object. Now on the native API however there is no built-in API for getting those values as simply. (Yes there is a this[0, 0] however frankly I was a bit lazy and didn't understand how it really worked and what I needed to do to get what I needed from it) After some digging, I found that there are a couple of functions within Core RP Library where I can grab what I need more easily, specifically in SphericalHarmonicsL2Utils...

Code (CSharp):

1. public static void GetL1(SphericalHarmonicsL2 sh, out Vector3 L1_R, out Vector3 L1_G, out Vector3 L1_B)
2.  
3. public static void GetL2(SphericalHarmonicsL2 sh, out Vector3 L2_0, out Vector3 L2_1, out Vector3 L2_2, out Vector3 L2_3, out Vector3 L2_4)

For simplification, I'm just going to use GetL1 which spits out 3 Vector3s. I can only assume based on the GPU side that what I'm looking for can be found right here with those Vector3s.

Basically in my mind, I'm seeing that...
L1_R = unity_SHAr
L1_G = unity_SHAg
L1_B = unity_SHAb

So emulating what I did on the GPU side I wrote a function that basically replicates what I did but in C# CPU code.

Code (CSharp):

1. private static Vector3 GetSphericalHarmonicsDirection(SphericalHarmonicsL2 sphericalHarmonicsL2)
2.     {
3.         GetL1(sphericalHarmonicsL2, out Vector3 L1_R, out Vector3 L1_G, out Vector3 L1_B);
4.  
5.         Vector3 combined = Vector3.zero;
6.         combined += L1_R;
7.         combined += L1_G;
8.         combined += L1_B;
9.  
10.         return combined;
11.     }

This should work, I tested it and as it turns out my direction was very wrong and different compared to what I was getting with the shader code.

So I decided to check if they were even the same values, on the shader I simply output that sphericalHarmonicsL1_direction vector I had which gave me an orange color according to where the object was placed. Now on the CPU side I basically drew a solid cube, and essentially converted the Vector3 from the GetSphericalHarmonicsDirection function into a color so I could see it... and this is what I got....



Context: The sphere, in this case, is a material with a shader that samples the GPU code that I described. The solid cube intersecting the sphere here is the CPU side which is drawn via a gizmo as described.

In theory, the color of the cube should be the exact same as the sphere except it is not... which leads me to believe that either I'm doing something wrong, or there is some additional computation/steps done before you end up with what you actually get in the unity_SH vectors in the shader. So my question is, what is it? Is there anything that I'm misunderstanding? What is the proper way to get those values so that it matches what I'm getting in the shader?

I have tried reversing the values, changing the order, and even adding each band individually but inherently it's only revealed to me that there is some more complex math/computation being done since the values are still very different and I can't get them to match.

TL: DR - Why are the spherical harmonic bands wrong on the CPU side versus on the GPU side? There seem to be additional things done to the unity_SH variables on the GPU side versus on the CPU side which leads to incorrect results. What's the proper way to obtain them to get the intended result I need?

NOTE: Please do not suggest something that sidesteps the issue (Like suggesting buying an existing asset that does what I'm doing... because that is not going to happen), I know it's possible to get dominant directions from probes because I'm able to do it in a shader correctly. So how can I do it on the CPU side?

 

Solved it!

Writing this for future newcomers who are running into a similar issue. I asked for help on another place and I was pointed toward a script that was created by keijiro on github. Essentially his script fills in the blanks for how exactly those variables, specifically...

Code (CSharp):

1. // SH lighting environment
2. half4 unity_SHAr;
3. half4 unity_SHAg;
4. half4 unity_SHAb;
5. half4 unity_SHBr;
6. half4 unity_SHBg;
7. half4 unity_SHBb;
8. half4 unity_SHC;

Fills in the blanks for how those actually get assigned. Specifically, the key lines were the following in keijiro's script.

Code (CSharp):

1. // Constant + Linear
2.         for (var i = 0; i < 3; i++)
3.             properties.SetVector(_idSHA[i], new Vector4(
4.                 sh[i, 3], sh[i, 1], sh[i, 2], sh[i, 0] - sh[i, 6]
5.             ));

Seeing that I was able to ascertain what I needed to change in my GetSphericalHarmonicsDirection function. I basically ditched the Core RP Library functions I found as they didn't give me what I was looking for and led me astray. So now my function looks like the following...

Code (CSharp):

1. private static Vector3 GetSphericalHarmonicsDirection(SphericalHarmonicsL2 sphericalHarmonicsL2)
2.     {
3.         Vector4 unity_SHAr = new Vector4(sphericalHarmonicsL2[0, 3], sphericalHarmonicsL2[0, 1], sphericalHarmonicsL2[0, 2], sphericalHarmonicsL2[0, 0] - sphericalHarmonicsL2[0, 6]);
4.         Vector4 unity_SHAg = new Vector4(sphericalHarmonicsL2[1, 3], sphericalHarmonicsL2[1, 1], sphericalHarmonicsL2[1, 2], sphericalHarmonicsL2[1, 0] - sphericalHarmonicsL2[1, 6]);
5.         Vector4 unity_SHAb = new Vector4(sphericalHarmonicsL2[2, 3], sphericalHarmonicsL2[2, 1], sphericalHarmonicsL2[2, 2], sphericalHarmonicsL2[2, 0] - sphericalHarmonicsL2[2, 6]);
6.  
7.         Vector3 combined = Vector3.zero;
8.         combined += new Vector3(unity_SHAr.x, unity_SHAr.y, unity_SHAr.z);
9.         combined += new Vector3(unity_SHAg.x, unity_SHAg.y, unity_SHAg.z);
10.         combined += new Vector3(unity_SHAb.x, unity_SHAb.y, unity_SHAb.z);
11.  
12.         return combined;
13.     }

With the debugging I had, now the output of both is pretty much identical which means now that sampling the dominant direction from the light probe on the CPU is solved.



So with that now, I'll feed the final results. With my current capsule-shadows implementation (which I might open source soon) you can sample directionality from directional lightmaps. Which works fairly well in many circumstances however in areas with lots of lights, this can lead to funky results where shadows are not clearly defined but also can look distorted as shown here... (the shading on the character model also indicates where the light is supposed to be casting shadows from).



By instead refactoring the code a bit and making it so the shadow casters sample directionality from light probes, this leads to more consistent and better-defined shadows. However, there are still some drawbacks with this as depending on the environment lots of lights can cause the shadow to slide around as the directionality changes depending on the light probes in the area. But this still works well and can be chosen for certain scenarios.

 

This looks amazing. I found it while researching what can be done with lightprobes in VR/performance constrained environment. Did you end up turning this into a release/opensource?

 

Yes I have indeed! Although my current implementation is still a work in progress, as it's a struggle to get compute shader-based "post effects" to work in VR.

https://github.com/frostbone25/Unity-Capsule-Shadows