Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

(Solved) Clip space to world space in a vertex shader

Discussion in 'Shaders' started by bitinn, May 15, 2018.

  1. bitinn

    bitinn

    Joined:
    Aug 20, 2016
    Posts:
    961
    Hi all,

    I am trying to find a good way to convert clip space position into world space position in vertex shader, so that I can pass on the interpolated world position to fragment shader.

    (just make sure we are on the same page for terminology)

    07.png

    So we need inverse of projection and view matrix. But due to confusing terminology in Unity shader variable naming, I don't actually know if it exists already:

    - official doc suggests there isn't one.
    - actual UnityShaderVariables.cginc suggests there should be one:

    Code (CSharp):
    1. #define UNITY_MATRIX_P glstate_matrix_projection
    2. #define UNITY_MATRIX_V unity_MatrixV
    3. #define UNITY_MATRIX_I_V unity_MatrixInvV
    4. #define UNITY_MATRIX_VP unity_MatrixVP
    5. #define UNITY_MATRIX_M unity_ObjectToWorld
    6.  
    7. //...
    8.  
    9. // Projection matrices of the camera. Note that this might be different from projection matrix
    10. // that is set right now, e.g. while rendering shadows the matrices below are still the projection
    11. // of original camera.
    12. float4x4 unity_CameraProjection;
    13. float4x4 unity_CameraInvProjection;
    14. float4x4 unity_WorldToCamera;
    15. float4x4 unity_CameraToWorld;
    ALL examples I find online try to pass cameraToWorldMatrix from script to shader.

    But I got questions:

    1. Do I have to do this through script? eg. is cameraToWorldMatrix the only reliable inverse VP source?

    2. Is the camera space in the doc the exact same thing as clip space in image above?

    3. Does Unity use this term consistently in their shader code? Say I am writing a post-processing image effect, is unity_CameraToWorld the same as cameraToWorldMatrix?

    People of unity forum, we have witnessed too many questions and arguments over the years, let's settle this one. (using Unity 2018 as baseline if you could).

    Thx!
     
    asdzxcv777 likes this.
  2. bgolus

    bgolus

    Joined:
    Dec 7, 2012
    Posts:
    12,329
    1. Yes and no. The cameraToWorldMatrix isn't the inverse VP matrix. It is only the inverse view matrix. But you are correct that the shader does not automatically have access to the inverse VP matrix and it is most easily accessed via script.

    2. No, as mentioned above it's the view matrix.

    3. Lets return to that later.


    Generally the UNITY_MATRIX_* and unity_Camera* matrices are different, though represent similar things.

    The reason is the UNITY_MATRIX_* matrices are used for actual rendering, with the UNITY_MATRIX_V view matrix being in OpenGL style -Z forward and the UNITY_MATRIX_P projection matrix being in whatever form necessary for the current platform, and may have the Y inverted or Z reversed. The view matrix and the camera.cameraToWorldMatrix should match, but the projection matrix and camera.projectionMatrix may not.

    The unity_CameraProjection matrix on the other hand matches the values you get from camera.projectionMatrix, which may not match the UNITY_MATRIX_V for the reason I mentioned above. This also means that though Unity provides the inverse view matrix and inverse camera projection matrix, they do not provide an inverse matrix for the projection matrix used to calculate the final clip space position. I don't know for sure if the unity_CameraToWorld and UNITY_MATRIX_I_V match, or if the former is the camera.transform.localToWorldMatrix.


    Here's the thing though. All of this is moot. If you want the world space position in the vertex shader you should use:

    float3 worldPos = mul(unity_ObjectToWorld, float4(v.vertex.xyz, 1.0)).xyz;


    Now for image effects, this is a totally different beast. In that case you are trying to extrapolate world positions from a depth buffer. You could pass in an inverse view projection matrix, but the depth is in window coordinate space. A common trick is to calculate the world space view direction to the vertices with a view depth of 1 unit and then multiply that interpolated vector by the linearized depth value. At one point Unity handled this by having the normals for the Blit() quad already be those calculated view dir vectors, but I believe they pass in the inverse matrices manually via script at this point. Part of the problem is by the time you're doing image effects the projection matrix values normally passed to a shader may not have valid values anymore during a Blit(). I believe using the inverse matrices is more accurate, but the interpolated vector is way faster.
     
  3. bitinn

    bitinn

    Joined:
    Aug 20, 2016
    Posts:
    961
    @bgolus to the rescue again!

    I think my personal problem with these built-in matrices are: I am never quite sure whether they would work cross-platform and remain unchanged during rendering lifecycle.

    Let us look some concrete examples specific on image effect + pixel world position calculation, here are 2 approaches, they do the same thing, but using different matrices:

    - https://github.com/keijiro/DepthInverseProjection/blob/master/Assets/InverseProjector.shader

    - https://github.com/chriscummings100...ldSpacePostEffect/WorldSpacePostEffect.shader

    keijiro does it in vertex shader, using basically:

    Code (CSharp):
    1. // this I think is the far clipping plane vertices in clip space
    2. float4 vpos = float4(i.vertex.xy, 1, 1);
    3.  
    4. // this I think returns it to eye space direction?
    5. float3 rayPers = mul(unity_CameraInvProjection, vpos) *_ProjectionParams.z;
    6.  
    7. // then later in fragment shader, use eye space direction with linear depth to get eye space position
    8. float3 viewPos = rayPers * Linear01Depth(depthTextureSample);
    9.  
    10. // use eye space position to get world space position
    11. float3 worldPos = mul(camera.cameraToWorldMatrix, float4(viewPos, 1)).xyz
    while chriscummings does it in fragment shader, using basically:

    Code (CSharp):
    1. // this basically restore object space vertex.xy from uv, as unity image effect fullscreen triangle vertices are known
    2. float2 uvClip = i.uv * 2.0 - 1.0;
    3.  
    4. // note that depth here is non-linear
    5. float4 clipPos = float4(uvClip, depthTextureSample, 1.0);
    6.  
    7. // this probably returns it to eye coordinate??
    8. float4 viewPos = mul(GL.GetGPUProjectionMatrix(camera.projectionMatrix, true).inverse, clipPos);
    9. viewPos /= viewPos.w;
    10.  
    11. // and this finally move it from eye to world coordinate??
    12. float3 worldPos = mul(camera.cameraToWorldMatrix, viewPos).xyz;
    They both did the same thing, I have tested both to work on my macOS (using Metal).

    But the second example is more convoluted, adding the confusing terminology, I have to check it line by line to understand what's happening. To be fair, chriscummings' solution isn't bad at all, I have seem much worse in some unity asset store implementations and on the Internet...

    So I am asking:

    - is keijiro's way, the overall better way to compute pixel world position in image effect?


    - if I replace cameraToWorldMatrix with unity_CameraToWorld, the shader still works, but some lines are colored differently, so how can I judge whether unity_CameraToWorld a good replacement here?

    cameraToWorldMatrix

    Screen Shot 2018-05-15 at 19.00.45.png

    unity_CameraToWorld

    Screen Shot 2018-05-15 at 19.00.56.png

    Thx in advance!

    (EDIT: to use the same terminology as OP's diagram..)
     
    Last edited: May 15, 2018
  4. bitinn

    bitinn

    Joined:
    Aug 20, 2016
    Posts:
    961
    I don't get why this is happening:

    I have implemented keijiro's solution as-is, but my world space result is not stable under camera z rotation.



    And I know my view space result is stable:



    And there is just one added calculation between them:

    Code (CSharp):
    1.     // create world space position
    2.     float3 CreateWorldSpacePosition (float3 viewPosition) {
    3.         return mul(_MainCameraToWorld, float4(viewPosition, 1.0)).xyz;
    4.     }
    while _MainCameraToWorld is simply camera.cameraToWorldMatrix.

    I don't know how I could have messed this up? The only significant differences between his code and mine are:

    1. he is using vertex ID to generate the fullscreen triangle, while I am using the fullscreen triangle from post-processing stack v2

    2. he is using default Graphics.Blit, while I am using the v2 stack renderer's context.command.BlitFullscreenTriangle

    3. he is writing it in CGINCLUDE block, while I am writing it in HLSLINCLUDE block (per v2 stack suggestion).

    I am dying to find a reason for this, but just couldn't spot it, if anyone cares, please help:

    Code (CSharp):
    1.  
    2. Shader "Hidden/BeginHQ/FogMachine" {
    3.     HLSLINCLUDE
    4.  
    5.     #include "PostProcessing/Shaders/StdLib.hlsl"
    6.  
    7.     /* effect parameters */
    8.  
    9.     TEXTURE2D_SAMPLER2D(_MainTex, sampler_MainTex);
    10.     TEXTURE2D_SAMPLER2D(_CameraDepthTexture, sampler_CameraDepthTexture);
    11.     half _Blend;
    12.     float4x4 _MainCameraToWorld;
    13.  
    14.     /* unity parameters */
    15.  
    16.     float4x4 unity_CameraToWorld;
    17.     float4x4 unity_CameraInvProjection;
    18.  
    19.     // vertext input (vertex in object space)
    20.     struct VertexInput {
    21.         float3 vertex : POSITION;
    22.     };
    23.  
    24.     // vertex to fragment input (vertex in clip space)
    25.     struct Vertex2Fragment {
    26.         float4 vertex : SV_POSITION;
    27.         float2 texcoord : TEXCOORD0;
    28.         float3 ray : TEXCOORD1;
    29.     };
    30.  
    31.     // create view space position
    32.     float3 CreateViewSpacePosition (float3 ray, float depth) {
    33.         // remap depth to linear [0..1] range, 1 being the far plane depth
    34.         return ray * Linear01Depth(depth);
    35.     }
    36.  
    37.     // create world space position
    38.     float3 CreateWorldSpacePosition (float3 viewPosition) {
    39.         return mul(_MainCameraToWorld, float4(viewPosition, 1.0)).xyz;
    40.     }
    41.  
    42.     // create color from input position
    43.     half4 CreateEffectColor (float3 position) {
    44.         return half4(pow(abs(cos(position * PI * 4)), 20), 1.0);
    45.     }
    46.  
    47.     // blend color
    48.     half4 BlendEffectColor (half4 sourceColor, half4 effectColor) {
    49.         return half4(lerp(sourceColor.rgb, effectColor.rgb, _Blend), sourceColor.a);
    50.     }
    51.  
    52.     // default vert
    53.     Vertex2Fragment FogMachineVert (VertexInput i) {
    54.         Vertex2Fragment o;
    55.  
    56.         // object space
    57.         float2 vertexXY = i.vertex.xy;
    58.  
    59.         // mesh input is actually a triangle that fills the screen
    60.         // object space to clip space, assume: far plane, orthographic view
    61.         // remember clip space xy is [-1..1]
    62.         o.vertex = float4(vertexXY, 1.0, 1.0);
    63.  
    64.         // vertex.xy are (-1,-1), (-1,3), (3,-1)
    65.         // this hack just put uv back to [0..1] range on screen
    66.         o.texcoord = (o.vertex.xy + 1.0) * 0.5;
    67.  
    68.         // directx and metal requires Y-axis flip and use (0,1) as origin
    69.         #if UNITY_UV_STARTS_AT_TOP
    70.         o.texcoord = o.texcoord * float2(1.0, -1.0) + float2(0.0, 1.0);
    71.         #endif
    72.  
    73.         // view space vertex direction
    74.         o.ray = mul(unity_CameraInvProjection, o.vertex).xyz * _ProjectionParams.z;
    75.  
    76.         return o;
    77.     }
    78.  
    79.     // view space pass
    80.     half4 FogMachineFragVS (Vertex2Fragment i) : SV_Target {
    81.         half4 outColor;
    82.  
    83.         // sample camera textures
    84.         half4 color = SAMPLE_TEXTURE2D(_MainTex, sampler_MainTex, i.texcoord);
    85.         float depth = SAMPLE_DEPTH_TEXTURE(_CameraDepthTexture, sampler_CameraDepthTexture, i.texcoord);
    86.  
    87.         // view position from ray and depth
    88.         float3 viewPosition = CreateViewSpacePosition(i.ray, depth);
    89.  
    90.         // visual color
    91.         half4 visualColor = CreateEffectColor(viewPosition);
    92.  
    93.         // blend visual with source color
    94.         outColor = BlendEffectColor(color, visualColor);
    95.  
    96.         return outColor;
    97.     }
    98.  
    99.     // world space pass
    100.     half4 FogMachineFragWS (Vertex2Fragment i) : SV_Target {
    101.         half4 outColor;
    102.  
    103.         // sample camera textures
    104.         half4 color = SAMPLE_TEXTURE2D(_MainTex, sampler_MainTex, i.texcoord);
    105.         float depth = SAMPLE_DEPTH_TEXTURE(_CameraDepthTexture, sampler_CameraDepthTexture, i.texcoord);
    106.  
    107.         // view position from ray and depth
    108.         float3 viewPosition = CreateViewSpacePosition(i.ray, depth);
    109.  
    110.         // world position from view position
    111.         float3 worldPosition = CreateWorldSpacePosition(viewPosition);
    112.  
    113.         // visual color
    114.         half4 visualColor = CreateEffectColor(worldPosition);
    115.  
    116.         // blend visual with source color
    117.         outColor = BlendEffectColor(color, visualColor);
    118.  
    119.         return outColor;
    120.     }
    121.  
    122.     ENDHLSL
    123.  
    124.     SubShader {
    125.         Cull Off
    126.         ZWrite Off
    127.         ZTest Always
    128.  
    129.         Pass {
    130.             HLSLPROGRAM
    131.  
    132.             #pragma target 3.5
    133.             #pragma vertex FogMachineVert
    134.             #pragma fragment FogMachineFragVS
    135.  
    136.             ENDHLSL
    137.         }
    138.  
    139.         Pass {
    140.             HLSLPROGRAM
    141.  
    142.             #pragma target 3.5
    143.             #pragma vertex FogMachineVert
    144.             #pragma fragment FogMachineFragWS
    145.  
    146.             ENDHLSL
    147.         }
    148.     }
    149. }
    150.  
     
  5. bgolus

    bgolus

    Joined:
    Dec 7, 2012
    Posts:
    12,329
    It would appear that Keijiro's solution is wrong in some way. Might be a simple issue of it being wrong on some platforms but not others, or he just didn't test it well enough. You mentioned you're using MacOS w/ Metal, I would wonder if it works only when using OpenGL instead, or Windows & DirectX. There are a lot of fun platform specific gotchas with UVs being flipped, or the projection's Y axis being inverted. If a shader doesn't account for those correctly it may work on one platform but not another (or sometimes just in a different use case on the same platform).

    I would wonder if using UNITY_MATRIX_I_V would work instead of passing the cameraToWorldMatrix in. My understanding is they should be the same, but I could be wrong.

    You are correct that your solution is working properly and that Keijiro's is not. If what you have works, I'd say go with that. However if my above assumption is correct it's possible your solution won't work on a different platform.

    The specifics of calculating a ray in the vertex shader and then multiplying the depth value is indeed cheaper on older or low powered devices, though modern desktop computers are stupid-fast at doing math so it may be about the same either way with Chris Cumming's method potentially being faster. There's a cost to transferring data from the vertex shader to the fragment shader that has not decreased as quickly as GPUs ability to do math has increased.
     
  6. bitinn

    bitinn

    Joined:
    Aug 20, 2016
    Posts:
    961
    Thx for looking into this, from your reply I am not sure if you have misread my replies or not, so just to clarify:

    (a) These 2 screenshots are actually demo scene from keijiro's repo, I was trying it in my local project and camera z rotation doesn't "zoom" the visual grid (it fits my expectation as we are dealing with world coordinate, which doesn't change when camera move or rotate.)

    (b) I also tried Chris Cumming's solution and it works the same way.

    (c) As I prefer keijiro's approach, I implemented his solution, and got those 2 short videos: view space is working as expected, but world space visual changes when I rotate camera. Which means I wasn't implementing it correctly. And I should add I am already passing the cameraToWorldMatrix, unless it's broken, I assume my implementation is at fault here.

    (d) Since I put these all in the same project: it means they all use the same Metal API, this confuses the hell out of me. What could have gone wrong here? so I ran through the main differences in my last reply.
     
  7. bitinn

    bitinn

    Joined:
    Aug 20, 2016
    Posts:
    961
    Another thing I noticed, with camera rotation set to (35.3, 45, 0) in both scene, aka the isometric perspective, my implementation's world space color line direction doesn't fit my expectation, while keijiro's works fine:

    Mine:

    Screen Shot 2018-05-16 at 12.26.44.png

    Keijiro:

    Screen Shot 2018-05-16 at 12.27.03.png
     
  8. bitinn

    bitinn

    Joined:
    Aug 20, 2016
    Posts:
    961
    Code (CSharp):
    1.     // directx and metal requires Y-axis flip and use (0,1) as origin
    2.     #if UNITY_UV_STARTS_AT_TOP
    3.     o.texcoord = o.texcoord * float2(1.0, -1.0) + float2(0.0, 1.0);
    4.     #endif
    I have basically isolated this problem down to this line. Somehow I need to flip this image correctly for Metal without affecting depth sampling. However, AFAIK, this is the correct way to create uv (this is also done in image effect stdlib default vertex shader)

    (without above line, rendering is flipped, but uv is stable under camera movement/rotation)

    Screen Shot 2018-05-16 at 14.05.04.png
     
  9. bgolus

    bgolus

    Joined:
    Dec 7, 2012
    Posts:
    12,329
    Me either! There was a lot to unpack there. I think I got confused with the talk about view space and world space between the two posts.

    Flip the o.vertex.y then instead.
     
    bitinn likes this.
  10. bitinn

    bitinn

    Joined:
    Aug 20, 2016
    Posts:
    961
    Thx, now I have figured it out: my texcoord is fine, I just need to flip o.vertex.y, and I need to flip it after the o.ray line, otherwise the ray direction is messed up, thus causing my swimming uv problem...

    ----

    A side note:

    My still don't fully understand why the default vertex shader in Post Process Stack V2 flip texcoord (defined in stdlib.hlsl), as it will likely cause issues like mine (I ported Keijiro's code directly to v2 stack, and its rendering got flipped as well)

    Code (CSharp):
    1. VaryingsDefault VertDefault(AttributesDefault v)
    2. {
    3.     VaryingsDefault o;
    4.     o.vertex = float4(v.vertex.xy, 0.0, 1.0);
    5.     o.texcoord = TransformTriangleVertexToUV(v.vertex.xy);
    6.  
    7. #if UNITY_UV_STARTS_AT_TOP
    8.     o.texcoord = o.texcoord * float2(1.0, -1.0) + float2(0.0, 1.0);
    9. #endif
    10.  
    11.     o.texcoordStereo = TransformStereoScreenSpaceTex(o.texcoord, 1.0);
    12.  
    13.     return o;
    14. }
     
  11. bgolus

    bgolus

    Joined:
    Dec 7, 2012
    Posts:
    12,329
    The texcoords and projection get flipped for all sorts of reasons.

    https://docs.unity3d.com/Manual/SL-PlatformDifferences.html

    My understanding of why isn't complete, but the main thing is trying to get everything to match OpenGL. For instance all textures in Unity are uploaded upside-down on non-OpenGL platforms to match the UV layout of 0,0 at the bottom left. This means fewer changes to meshes and shaders in general, but causes some weirdness with render textures as they can't be flipped afterwards. So instead they flip the projection matrix so it renders upside-down. But then there are still cases where things get flipped. Like if you're reading an upside-down texture, but have a flipped projection matrix, the two work against each other to double flip the image.
     
    bitinn likes this.
  12. bitinn

    bitinn

    Joined:
    Aug 20, 2016
    Posts:
    961
    Yeah I read that doc, and the number of "might" on that page is alarming; discussion on reddit suggest this is just one of those things we have to test and figure out.

    ----

    But please indulge me for a minute here, if we exclude VR for now, the main factors for flipped render are:

    - projection
    - uv
    - texture
    - image effect
    - blit

    All of them can be flipped by unity, my understanding is, on my macOS using Metal:

    - projection (flipped internally, as _ProjectionParams.x is -1)
    - uv (not flipped, though UNITY_UV_STARTS_AT_TOP is 1)
    - texture (flipped internally, but we are not using texture here)
    - image effect (source image is not flipped, per document)
    - blit (will flip image effect render texture for simple cases)

    So my reasoning on why o.vertex.y flipping works:

    - Image effect doesn't flip source texture, so we shouldn't flip uv.
    - Post processing stack v2 does blit internally, so we shouldn't flip our render texture.
    - However projection has been flipped already, hence we need o.vertex.y * _ProjectionParams.x to restore it.

    I think this is what happened, and let me know if you spot anything incorrect.
     
    MadeFromPolygons likes this.
  13. AndrewP72

    AndrewP72

    Joined:
    Nov 5, 2017
    Posts:
    5
    @bittin Did you ever find a solution to this? I am trying to figure out some light scattering, post process and need the world space vertex position. I have the solution working in a standard vertex / frag shader but am having issues converting over and I'm pretty sure its just the world space pos thats the problem. You mention "now I have figured it out: my texcoord is fine, I just need to flip o.vertex.y, and I need to flip it after the o.ray line, otherwise the ray direction is messed up, thus causing my swimming uv problem..." Do you have any code on how you do this? Alternatively if you found a nicer way it'd be nice to know :).
    Cheers
     
  14. bitinn

    bitinn

    Joined:
    Aug 20, 2016
    Posts:
    961
    Yeah I did, it's hard to explain though (see my previous reply, any of those factors could cause your render to flip), so just take a look at kejiro's repo, he has updated it to work with Post Processing Stack recently.

    https://github.com/keijiro/DepthInverseProjection
     
    orangetech and noio like this.
  15. AndrewP72

    AndrewP72

    Joined:
    Nov 5, 2017
    Posts:
    5
    Thanks so much, that part now works like a charm. Cheers