Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Question Baking Decals into Texture - incorrect normals

Discussion in 'Shaders' started by hippocoder, Jan 6, 2021.

  1. hippocoder

    hippocoder

    Digital Ape

    Joined:
    Apr 11, 2010
    Posts:
    29,723
    I am baking a decal from world space into a mesh's texture, and this works for colour, but the normal directions are all over the place.

    What works: The pixel is projected into the texture from world decal space to UV space correctly. It works fine for colour.

    What doesn't: But the normals are oriented incorrectly.

    The pixels are in the right place in the target UV map, just I'm not sure how to rotate the final normal direction by whatever tangent to account for it, it's not just a problem of reversing whatever rotation as the UV islands are rotated any angle (from the mesh's existing UVs).

    Any help would be super cool as this is my third whole day on this :/

    Code (CSharp):
    1. Shader "Simian/Internal/UnwrapNormal"
    2. {
    3.     Properties
    4.     {
    5.     }
    6.  
    7.     SubShader
    8.     {
    9.         Tags { "RenderType" = "Transparent" }
    10.         Blend SrcAlpha OneMinusSrcAlpha
    11.         Cull Off ZWrite Off ZTest Always
    12.         ColorMask RGB //don't affect alpha of final texture for dilate
    13.  
    14.         Pass
    15.         {
    16.             CGPROGRAM
    17.             #pragma vertex vert
    18.             #pragma fragment frag
    19.             #include "UnityCG.cginc"
    20.  
    21.             struct appdata
    22.             {
    23.                 float4 vertex : POSITION;
    24.                 float4 uv : TEXCOORD0;
    25.             };
    26.  
    27.             struct v2f
    28.             {
    29.                 float4 vertex : SV_POSITION;
    30.                 float3 worldPos : TEXCOORD0;
    31.             };
    32.  
    33.             sampler2D _MainTex;
    34.             float4 _MainTex_ST;
    35.             float4x4 worldToDecalMatrix;
    36.  
    37.             float2 Project(float3 worldPos)
    38.             {
    39.                 float4 decalPos = mul(worldToDecalMatrix, float4(worldPos.xyz, 1.0));
    40.                 float2 uv = saturate(float2(decalPos.xy + 0.5));
    41.                 return uv;
    42.             }
    43.  
    44.             v2f vert(appdata v)
    45.             {
    46.                 v2f o;
    47.  
    48.                 o.worldPos = mul(unity_ObjectToWorld, v.vertex);
    49.  
    50.                 //calculate unwrapped mesh verts
    51.                 o.vertex = mul(UNITY_MATRIX_P, float4(v.uv.xy, 0.0, 1.0));
    52.                 return o;
    53.             }
    54.      
    55.             float4 frag(v2f IN) : SV_Target
    56.             {
    57.                 float3 normal = UnpackNormal(tex2D(_MainTex, Project(IN.worldPos)));
    58.  
    59.                 //possibly reorient normals somehow here?
    60.  
    61.                 normal = normal * 0.5 + 0.5;
    62.        
    63.                 return float4(normal, 1);
    64.             }
    65.             ENDCG
    66.         }
    67.     }
    68. }
     
  2. bgolus

    bgolus

    Joined:
    Dec 7, 2012
    Posts:
    12,329
    The
    UnpackNormal
    function already did that for you.

    Your normal map is in "tangent space". In the case of a projected decal, that "tangent space" is based on the projection space. You might get something reasonably useable by applying the transpose or inverse of the projection matrix to the tangent normal.

    Code (csharp):
    1. // using mul with the matrix as the second argument applies the matrix transposed
    2. float3 worldNormal = normalize(mul(normal, worldToDecalMatrix));
    However if your projection isn't aligned to the surface normal it's projecting on, this probably won't look great. A reasonable expectation is for a projected normal decal to have "out" be the surface normal it's projecting onto, but just using the projection matrix won't get you that. The "out" will just be the projection's orientation in world space, as will both the up and right directions. So what you really need to do is get the world space up and right and project those onto the geometry's surface normal. To do that you do need to pass the inverse matrix to the shader, since it'd be expensive to calculate in the shader. You also need to pass the vertex normal from the vertex shader to the fragment shader.

    Code (csharp):
    1. // in the vertex shader
    2. o.normal = UnityObjectToWorldNormal(v.normal);
    3.  
    4. // function to do vector projection
    5. float3 projectOnPlane( float3 vec, float3 normal )
    6. {
    7.    return vec - normal * dot( vec, normal );
    8. }
    9.  
    10. // in the fragment shader
    11. float3 surfaceNormal = normalize(IN.normal);
    12.  
    13. float3 decalRight = mul(decalToWorldMatrix, float3(1,0,0));
    14. float3 decalUp = mul(decalToWorldMatrix, float3(0,0,1));
    15.  
    16. float3x3 decalTangentToWorld = float3x3(
    17.   normalize(projectOnPlane(decalRight, surfaceNormal)),
    18.   normalize(projectOnPlane(decalUp, surfaceNormal)),
    19.   surfaceNormal
    20. );
    21.  
    22. float3 decalNormal = UnpackNormal(tex2D(_MainTex, Project(IN.worldPos)));
    23. float3 worldNormal = normalize(mul(decalNormal, decalTangentToWorld));
    I think that's right, but I haven't tested it.

    Alternatively, you could use screen space derivatives to construct the tangent to world rotation matrix. I use this function:
    Code (csharp):
    1. // Unity version of http://www.thetenthplanet.de/archives/1180
    2. float3x3 cotangent_frame( float3 normal, float3 position, float2 uv )
    3. {
    4.   // get edge vectors of the pixel triangle
    5.   float3 dp1 = ddx( position );
    6.   float3 dp2 = ddy( position ) * _ProjectionParams.x;
    7.   float2 duv1 = ddx( uv );
    8.   float2 duv2 = ddy( uv ) * _ProjectionParams.x;
    9.   // solve the linear system
    10.   float3 dp2perp = cross( dp2, normal );
    11.   float3 dp1perp = cross( normal, dp1 );
    12.   float3 T = dp2perp * duv1.x + dp1perp * duv2.x;
    13.   float3 B = dp2perp * duv1.y + dp1perp * duv2.y;
    14.   // construct a scale-invariant frame
    15.   float invmax = rsqrt( max( dot(T,T), dot(B,B) ) );
    16.   // matrix is transposed, use mul(VECTOR, MATRIX) order
    17.   return float3x3( T * invmax, B * invmax, normal );
    18. }
    19.  
    20. // in fragment shader
    21. float3 surfaceNormal = normalize(IN.normal);
    22. float2 decalUV = Project(IN.worldPos);
    23. float3 decalNormal = UnpackNormal(tex2D(_MainTex, decalUV));
    24. float3x3 decalTangentToWorld = cotangent_frame(surfaceNormal, IN.worldPos, decalUV);
    25. float3 worldNormal = normalize(mul(decalNormal, decalTangentToWorld));
     
    hippocoder likes this.
  3. hippocoder

    hippocoder

    Digital Ape

    Joined:
    Apr 11, 2010
    Posts:
    29,723
    Thank you very much for responding. I tried to implement your suggestions and I believe I'm being a bit thick because I'm just not seeing the problem clear enough. To help diagnose, I've enclosed the full shaders used (yours and mine) with test images. The model is a difficult to UV map test and the normal borrowed from one of your resources.

    I think I've made a silly mistake in implementing your suggestions. Where am I going wrong?

    Decal Colour and Normal maps



    Untextured source model with UVs


    Render with Colour


    Colour generated texture

    Shader (for Normals, Colour is basically the same shader)
    Code (CSharp):
    1. Shader "Simian/Internal/UnwrapNormal"
    2. {
    3.     Properties
    4.     {
    5.     }
    6.  
    7.     SubShader
    8.     {
    9.         Tags { "RenderType" = "Transparent" }
    10.         Blend SrcAlpha OneMinusSrcAlpha
    11.         Cull Off ZWrite Off ZTest Always
    12.         ColorMask RGB //don't affect alpha of final texture for dilate
    13.  
    14.         Pass
    15.         {
    16.             CGPROGRAM
    17.             #pragma vertex vert
    18.             #pragma fragment frag
    19.             #include "UnityCG.cginc"
    20.  
    21.             struct appdata
    22.             {
    23.                 float4 vertex : POSITION;
    24.                 float4 uv : TEXCOORD0;
    25.             };
    26.  
    27.             struct v2f
    28.             {
    29.                 float4 vertex : SV_POSITION;
    30.                 float3 worldPos : TEXCOORD0;
    31.             };
    32.  
    33.             sampler2D _MainTex;
    34.             float4 _MainTex_ST;
    35.             float4x4 worldToDecalMatrix;
    36.  
    37.             float2 Project(float3 worldPos)
    38.             {
    39.                 float4 decalPos = mul(worldToDecalMatrix, float4(worldPos.xyz, 1.0));
    40.                 float2 uv = saturate(float2(decalPos.xy + 0.5));
    41.                 return uv;
    42.             }
    43.  
    44.             v2f vert(appdata v)
    45.             {
    46.                 v2f o;
    47.  
    48.                 o.worldPos = mul(unity_ObjectToWorld, v.vertex);
    49.  
    50.                 //calculate unwrapped mesh verts
    51.                 o.vertex = mul(UNITY_MATRIX_P, float4(v.uv.xy, 0.0, 1.0));
    52.                 return o;
    53.             }
    54.  
    55.             float4 frag(v2f IN) : SV_Target
    56.             {
    57.                 float3 normal = UnpackNormal(tex2D(_MainTex, Project(IN.worldPos)));
    58.  
    59.                 //possibly reorient normals somehow here?
    60.  
    61.                 normal = normal * 0.5 + 0.5;
    62.  
    63.                 return float4(normal, 1);
    64.             }
    65.             ENDCG
    66.         }
    67.     }
    68. }
    Render with Normal


    Render with Colour and Normal


    Render Texture from Unwrap Shader (I used * 0.5 + 0.5 for testing)

    Passing in this matrix and inverse:
    Code (csharp):
    1. Matrix4x4 worldToDecalMatrix = decal[i].transform.worldToLocalMatrix * decalObject.transform.localToWorldMatrix;
    2. Matrix4x4 decalToWorldMatrix = worldToDecalMatrix.inverse;
    Shader
    Code (CSharp):
    1. Shader "Simian/Internal/UnwrapNormal"
    2. {
    3.     Properties
    4.     {
    5.     }
    6.  
    7.     SubShader
    8.     {
    9.         Tags { "RenderType" = "Transparent" }
    10.         Blend SrcAlpha OneMinusSrcAlpha
    11.         Cull Off ZWrite Off ZTest Always
    12.         ColorMask RGB //don't affect alpha of final texture for dilate
    13.  
    14.         Pass
    15.         {
    16.             CGPROGRAM
    17.             #pragma vertex vert
    18.             #pragma fragment frag
    19.             #include "UnityCG.cginc"
    20.          
    21.             sampler2D _MainTex;
    22.             float4 _MainTex_ST;
    23.             float4x4 worldToDecalMatrix;
    24.             float4x4 decalToWorldMatrix; //inverse
    25.  
    26.             // function to do vector projection
    27.             float3 projectOnPlane(float3 vec, float3 normal)
    28.             {
    29.                return vec - normal * dot(vec, normal);
    30.             }
    31.  
    32.             float2 Project(float3 worldPos)
    33.             {
    34.                 float4 decalPos = mul(worldToDecalMatrix, float4(worldPos.xyz, 1.0));
    35.                 float2 uv = saturate(float2(decalPos.xy + 0.5));
    36.                 return uv;
    37.             }
    38.  
    39.             struct appdata
    40.             {
    41.                 float4 vertex : POSITION;
    42.                 float4 normal : NORMAL;
    43.                 float4 uv : TEXCOORD0;
    44.             };
    45.  
    46.             struct v2f
    47.             {
    48.                 float4 vertex : SV_POSITION;
    49.                 float3 worldPos : TEXCOORD0;
    50.                 float3 normal : TEXCOORD1;
    51.             };
    52.  
    53.             v2f vert(appdata v)
    54.             {
    55.                 v2f o;
    56.  
    57.                 o.worldPos = mul(unity_ObjectToWorld, v.vertex);
    58.  
    59.                 // in the vertex shader
    60.                 o.normal = UnityObjectToWorldNormal(v.normal);
    61.              
    62.                 //calculate unwrapped mesh verts
    63.                 o.vertex = mul(UNITY_MATRIX_P, float4(v.uv.xy, 0.0, 1.0));
    64.                 return o;
    65.             }
    66.  
    67.             float4 frag(v2f IN) : SV_Target
    68.             {
    69.                 // in the fragment shader
    70.                 float3 surfaceNormal = normalize(IN.normal);
    71.  
    72.                 float3 decalRight = mul(decalToWorldMatrix, float3(1,0,0));
    73.                 float3 decalUp = mul(decalToWorldMatrix, float3(0,0,1));
    74.  
    75.                 float3x3 decalTangentToWorld = float3x3(
    76.                   normalize(projectOnPlane(decalRight, surfaceNormal)),
    77.                   normalize(projectOnPlane(decalUp, surfaceNormal)),
    78.                   surfaceNormal
    79.                 );
    80.  
    81.                 float3 decalNormal = UnpackNormal(tex2D(_MainTex, Project(IN.worldPos)));
    82.                 float3 worldNormal = normalize(mul(decalNormal, decalTangentToWorld));
    83.  
    84.                 return float4(worldNormal, 1);
    85.             }
    86.             ENDCG
    87.         }
    88.     }
    89. }
    Render with Normal


    Render with Colour and Normal


    Render Texture from Unwrap Shader

    Passing in this matrix:
    Code (csharp):
    1. Matrix4x4 worldToDecalMatrix = decal[i].transform.worldToLocalMatrix * decalObject.transform.localToWorldMatrix;
    Shader
    Code (CSharp):
    1. Shader "Simian/Internal/UnwrapNormal"
    2. {
    3.     Properties
    4.     {
    5.     }
    6.  
    7.     SubShader
    8.     {
    9.         Tags { "RenderType" = "Transparent" }
    10.         Blend SrcAlpha OneMinusSrcAlpha
    11.         Cull Off ZWrite Off ZTest Always
    12.         ColorMask RGB //don't affect alpha of final texture for dilate
    13.  
    14.         Pass
    15.         {
    16.             CGPROGRAM
    17.             #pragma vertex vert
    18.             #pragma fragment frag
    19.             #include "UnityCG.cginc"
    20.  
    21.             sampler2D _MainTex;
    22.             float4 _MainTex_ST;
    23.             float4x4 worldToDecalMatrix;
    24.             float4x4 decalToWorldMatrix;
    25.  
    26.             // Unity version of http://www.thetenthplanet.de/archives/1180
    27.             float3x3 cotangent_frame(float3 normal, float3 position, float2 uv)
    28.             {
    29.                 // get edge vectors of the pixel triangle
    30.                 float3 dp1 = ddx(position);
    31.                 float3 dp2 = ddy(position) * _ProjectionParams.x;
    32.                 float2 duv1 = ddx(uv);
    33.                 float2 duv2 = ddy(uv) * _ProjectionParams.x;
    34.                 // solve the linear system
    35.                 float3 dp2perp = cross(dp2, normal);
    36.                 float3 dp1perp = cross(normal, dp1);
    37.                 float3 T = dp2perp * duv1.x + dp1perp * duv2.x;
    38.                 float3 B = dp2perp * duv1.y + dp1perp * duv2.y;
    39.                 // construct a scale-invariant frame
    40.                 float invmax = rsqrt(max(dot(T,T), dot(B,B)));
    41.                 // matrix is transposed, use mul(VECTOR, MATRIX) order
    42.                 return float3x3(T * invmax, B * invmax, normal);
    43.             }
    44.  
    45.             float2 Project(float3 worldPos)
    46.             {
    47.                 float4 decalPos = mul(worldToDecalMatrix, float4(worldPos.xyz, 1.0));
    48.                 float2 uv = saturate(float2(decalPos.xy + 0.5));
    49.                 return uv;
    50.             }
    51.  
    52.             struct appdata
    53.             {
    54.                 float4 vertex : POSITION;
    55.                 float4 normal : NORMAL;
    56.                 float4 uv : TEXCOORD0;
    57.             };
    58.  
    59.             struct v2f
    60.             {
    61.                 float4 vertex : SV_POSITION;
    62.                 float3 worldPos : TEXCOORD0;
    63.                 float3 normal : TEXCOORD1;
    64.             };
    65.  
    66.             v2f vert(appdata v)
    67.             {
    68.                 v2f o;
    69.  
    70.                 o.worldPos = mul(unity_ObjectToWorld, v.vertex);
    71.                 o.normal = v.normal;
    72.  
    73.                 // in the vertex shader
    74.                 o.normal = UnityObjectToWorldNormal(v.normal);
    75.  
    76.                 //calculate unwrapped mesh verts
    77.                 o.vertex = mul(UNITY_MATRIX_P, float4(v.uv.xy, 0.0, 1.0));
    78.                 return o;
    79.             }
    80.  
    81.             float4 frag(v2f IN) : SV_Target
    82.             {
    83.                 // in fragment shader
    84.                 float3 surfaceNormal = normalize(IN.normal);
    85.                 float2 decalUV = Project(IN.worldPos);
    86.                 float3 decalNormal = UnpackNormal(tex2D(_MainTex, decalUV));
    87.                 float3x3 decalTangentToWorld = cotangent_frame(surfaceNormal, IN.worldPos, decalUV);
    88.                 float3 worldNormal = normalize(mul(decalNormal, decalTangentToWorld));
    89.  
    90.                 return float4(worldNormal, 1);
    91.             }
    92.             ENDCG
    93.         }
    94.     }
    95. }
    Render with Normal


    Render with Colour and Normal


    Render Texture from Unwrap Shader

    Thank you for taking a look.
     
  4. bgolus

    bgolus

    Joined:
    Dec 7, 2012
    Posts:
    12,329
    The above functions get the transform for decal projection into world space. You still need to transform the resulting world space vector into the mesh's tangent space after that.
    https://forum.unity.com/threads/flat-lighting-without-separate-smoothing-groups.280183/#post-5057189

    Oh, and when outputting a normal map from a shader, you want to rescale it from the -1.0 to 1.0 range the normal vector is in to a 0.0 to 1.0 range that can actually be stored in a texture.
    return float4(normal * 0.5 + 0.5, 1.0);


    And you need to make sure you're rendering in the correct color space. If you're using a project set to use Linear Color Space, by a default render textures will be doing color space conversions, which you do not want.
     
    Last edited: Jan 7, 2021
    hippocoder likes this.
  5. hippocoder

    hippocoder

    Digital Ape

    Joined:
    Apr 11, 2010
    Posts:
    29,723
    I think we are getting somewhere, almost!

    Using this code:
    Code (CSharp):
    1. Shader "Simian/Internal/UnwrapNormal"
    2. {
    3.     Properties
    4.     {
    5.     }
    6.  
    7.     SubShader
    8.     {
    9.         Tags { "RenderType" = "Transparent" }
    10.         Blend SrcAlpha OneMinusSrcAlpha
    11.         Cull Off ZWrite Off ZTest Always
    12.         ColorMask RGB //don't affect alpha of final texture for dilate
    13.  
    14.         Pass
    15.         {
    16.             CGPROGRAM
    17.             #pragma vertex vert
    18.             #pragma fragment frag
    19.             #include "UnityCG.cginc"
    20.        
    21.             sampler2D _MainTex;
    22.             sampler2D _BumpMap;
    23.             float4 _MainTex_ST;
    24.             float4x4 worldToDecalMatrix;
    25.             float4x4 decalToWorldMatrix; //inverse
    26.  
    27.             // function to do vector projection
    28.             float3 projectOnPlane(float3 vec, float3 normal)
    29.             {
    30.                return vec - normal * dot(vec, normal);
    31.             }
    32.  
    33.             float2 Project(float3 worldPos)
    34.             {
    35.                 float4 decalPos = mul(worldToDecalMatrix, float4(worldPos.xyz, 1.0));
    36.                 //float4 decalPos = mul(decalToWorldMatrix, float4(worldPos.xyz, 1.0));
    37.                 float2 uv = saturate(float2(decalPos.xy + 0.5));
    38.                 return uv;
    39.             }
    40.  
    41.             float3 WorldNormalVector(float3 TtoW0, float3 TtoW1, float3 TtoW2, float3 normal)
    42.             {
    43.                 return float3(dot(TtoW0, normal), dot(TtoW1, normal), dot(TtoW2, normal));
    44.             }
    45.  
    46.             float3x3 WorldToTangentMatrix(float3 normal, float4 tangent)
    47.             {
    48.                 float3 binormal = cross(normal, tangent.xyz) * tangent.w;
    49.                 float3x3 rotation = float3x3(tangent.xyz, binormal, normal);
    50.  
    51.                 float3 TtoW0 = mul(rotation, ((float3x3)unity_ObjectToWorld)[0].xyz) * 1.0;
    52.                 float3 TtoW1 = mul(rotation, ((float3x3)unity_ObjectToWorld)[1].xyz) * 1.0;
    53.                 float3 TtoW2 = mul(rotation, ((float3x3)unity_ObjectToWorld)[2].xyz) * 1.0;
    54.  
    55.                 float3 worldT = WorldNormalVector(TtoW0, TtoW1, TtoW2, float3(1, 0, 0));
    56.                 float3 worldB = WorldNormalVector(TtoW0, TtoW1, TtoW2, float3(0, 1, 0));
    57.                 float3 worldN = WorldNormalVector(TtoW0, TtoW1, TtoW2, float3(0, 0, 1));
    58.  
    59.                 half3x3 w2tRotation;
    60.                 w2tRotation[0] = worldB.yzx * worldN.zxy - worldB.zxy * worldN.yzx;
    61.                 w2tRotation[1] = worldT.zxy * worldN.yzx - worldT.yzx * worldN.zxy;
    62.                 w2tRotation[2] = worldT.yzx * worldB.zxy - worldT.zxy * worldB.yzx;
    63.  
    64.                 half det = dot(worldT.xyz, w2tRotation[0]);
    65.  
    66.                 w2tRotation *= rcp(det);
    67.  
    68.                 return w2tRotation;
    69.             }
    70.  
    71.             struct appdata
    72.             {
    73.                 float4 vertex : POSITION;
    74.                 float3 normal : NORMAL;
    75.                 float4 tangent : TANGENT;
    76.                 float4 uv : TEXCOORD0;
    77.             };
    78.  
    79.             struct v2f
    80.             {
    81.                 float4 vertex : SV_POSITION;
    82.                 float3 worldPos : TEXCOORD0;
    83.                 float3 normal : TEXCOORD1;
    84.                 float4 tangent : TEXCOORD2;
    85.                 float2 uv : TEXCOORD3;
    86.             };
    87.  
    88.             v2f vert(appdata v)
    89.             {
    90.                 v2f o;
    91.  
    92.                 o.worldPos = mul(unity_ObjectToWorld, v.vertex);
    93.                 o.tangent = v.tangent;
    94.                 o.uv = v.uv.xy;
    95.  
    96.                 // in the vertex shader
    97.                 o.normal = UnityObjectToWorldNormal(v.normal);
    98.  
    99.                 //calculate unwrapped mesh verts
    100.                 o.vertex = mul(UNITY_MATRIX_P, float4(v.uv.xy, 0.0, 1.0));
    101.                 return o;
    102.             }
    103.  
    104.             float4 frag(v2f IN) : SV_Target
    105.             {
    106.                 // in the fragment shader
    107.                 float3 surfaceNormal = normalize(IN.normal);
    108.  
    109.                 float3 decalRight = mul(decalToWorldMatrix, float3(1,0,0));
    110.                 float3 decalUp = mul(decalToWorldMatrix, float3(0,0,1));
    111.  
    112.                 float3x3 decalTangentToWorld = float3x3(
    113.                   normalize(projectOnPlane(decalRight, surfaceNormal)),
    114.                   normalize(projectOnPlane(decalUp, surfaceNormal)),
    115.                   surfaceNormal
    116.                 );
    117.  
    118.  
    119.                 float3 decalNormal = UnpackNormal(tex2D(_MainTex, Project(IN.worldPos)));
    120.                 float3 worldNormal = normalize(mul(decalNormal, decalTangentToWorld));
    121.  
    122.                 float3x3 tangentMatrix = WorldToTangentMatrix(IN.normal, IN.tangent);
    123.                 float3 normal = normalize(mul(tangentMatrix, worldNormal));
    124.  
    125.                 return float4(normal * 0.5 + 0.5, 1);
    126.             }
    127.             ENDCG
    128.         }
    129.     }
    130. }
    Results in this output:



    The problems I have are:

    1. Am I following your advice correctly? I am insecure about this. I think I'm doing something wrong still.

    2. I want the highest quality as it's for baking. I picked the Inverse Rotation matrix code from your other thread as you said it was the correct/most accurate. I am not sure if I should pass in an inverted matrix from C# too, because my brain is now properly scrambled! I think this is inverting it so I don't need to?

    3. I am happy to go with the partial derivatives version or this - whichever is better looking. Which one should I pick?

    4. I have access to the high poly to low poly baked normal map. This is from Simplygon and very high quality, should I input these or the actual mesh normals for the best result? If so, how do I do that properly?

    5. There seems to be some pinching of the sphere normal, I think perhaps a linear issue? I made sure to create the render texture as linear, and the project is linear.

    6. It seems possibly the normal is also lit the wrong angle, uniformly, I suspect I have to probably counter the decal z or roll somehow. Is there a trick to this? Maybe it will go away once I sort the rest out. The decal projector's transform is the only rolling being done.

    I'm very thankful for your advice, I think things are untangling mentally but please bear with me on these final steps, very grateful as the amount of stress/pressure I have had with this has been tremendous :)

    Thanks for help and sharing knowledge!
     
  6. bgolus

    bgolus

    Joined:
    Dec 7, 2012
    Posts:
    12,329
    Something is definitely wrong with my "project on plane" based solution, the
    cotangent_frame
    function was getting much closer to correct results, though the results aren't going to be quite as smooth. I honestly don't know quite what's wrong.

    The matrix invert there is working on the world to tangent matrix, which is unique per pixel. There's no way to pre-calculate that one from c#, hence why it has to be done in the shader. You already appear to be passing in the inverse decal matrix, which is the only one that's constant for the entire mesh.

    Actual mesh normals, and mesh tangents. That's what the texture is going to be applied to, so that's what it should be baked onto.

    Part of this might be due to the decal to world function being wrong. But yes, in the above texture the "flat" areas are showing as RGB(188,188,255) which means there's most certainly an gamma problem. How are you creating he render texture you're rendering to? Is it set to
    RenderTextureReadWrite.Linear
    ?
     
  7. bgolus

    bgolus

    Joined:
    Dec 7, 2012
    Posts:
    12,329
    Oh, and while it doesn't really matter quite yet since there are so many issues still, for the built in rendering path you really do need to calculate the bitangent in the vertex shader and send that to the fragment shader instead of calculating the bitangent in the fragment for it to be 100% correct.
     
    hippocoder likes this.
  8. bgolus

    bgolus

    Joined:
    Dec 7, 2012
    Posts:
    12,329
    I figured out at least part of it.
    Code (csharp):
    1. float3 decalUp = mul(decalToWorldMatrix, float3(0,1,0));
    I had it as
    float3(0,0,1)
    instead of
    float3(0,1,0)
     
    hippocoder likes this.
  9. hippocoder

    hippocoder

    Digital Ape

    Joined:
    Apr 11, 2010
    Posts:
    29,723
    Thank you for your advice and tutorials here and online. By reading all of your resources I'd gained enough understanding to correct my mistakes. I'm using the cotangent_frame version, and will also try the other you've corrected out of interest, but I understand the cotangent_frame method is probably going to be the highest quality from what you've said.

    Thanks, much appreciated!
     
  10. bgolus

    bgolus

    Joined:
    Dec 7, 2012
    Posts:
    12,329
    The
    cotangent_frame
    function isn’t the best quality. It’s just one I know produces usable results. If you look at the normal maps it produces, there are odd hard edges that show up. The other method produces smoother normals.