Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

XR Rendering Broken for Right Eye with Custom Shaders (Unity 2020.1.5, Windows XR Plugin 4.0.2)

Discussion in 'VR' started by ceitel, Sep 16, 2020.

  1. ceitel

    ceitel

    Joined:
    Jan 3, 2017
    Posts:
    35
    I have several custom shaders, some very simple variants of the built-in shaders, some more complicated volume rendering shaders. Just migrated from 2019.4 to 2020.1 and switched over from the built in XR system to Windows XR plugin 4.0.2 (note: I also tried Window XR plugin 3.3.1 without success)

    Each of the below was working correctly before migration.

    Simple variant of the built-in default UI "sprite" shader (see uploaded shader)
    Note: icons at the bottom of the screen are not present in the right eye rendering, also transparent clouds can be seen coming down below the mountains in the left eye, and not in the right.
    opaque_ui_shader.jpg

    The following is actually a second scene, but a static image from the first scene gets stuck on the right eye unless I disable my custom OnRenderImage script on my camera in the second scene.
    Here several custom shaders are used to do a volumetric rendering of a cube.
    Note: only the custom rendering is complete of the cube, OnRenderImage should be merging the volumetric rendering with the rest of the scene using Graphics.Blit(source, destination) and we should see a black skybox as well as controllers and UI which are not rendered.
    volumetric_shader.jpg

    I have tried switching between UWP and Windows Standalone platforms with no luck.
    This seems to be a "multipass" issue, but the settings no longer exist as they did in the built in XR system. I have output the Rendering mode during playback to see what is being used and it does claim "StereoRenderingMode: MultiPass"
     

    Attached Files:

  2. ThomasZeng

    ThomasZeng

    Unity Technologies

    Joined:
    Jun 24, 2019
    Posts:
    84
    Hello @ceitel,
    First of all, thanks for sharing your use case here. We appreciate the details you provided here :)
    I have take a look at the attached shaders and I think I understand what caused this discrepancy between built-in WMR and XRSDK WMR.
    From what I can tell, the shader is not Single Pass Instanced compatible. I believe when using built-in WMR, the project was configured to run multipass stereo mode, that's why it worked. This stereo mode is no longer supported in XRSDK WMR. That's what caused the breakage.

    I would suggest to upgrate the shader to be SPI compatible and also update it to be URP shaders if you were using URP.
    The SPI doc can be found here: https://docs.unity3d.com/Manual/SinglePassInstancing.html

    I have made some some quick edits to your shader to show what I mean by updating shader. Please note I didn't tested the shader and it might not compile :). After the proper shader upgrade, this custom shader effect should work with WMR XR. Let us know how it goes!
     

    Attached Files:

    JasonCostanza likes this.
  3. ceitel

    ceitel

    Joined:
    Jan 3, 2017
    Posts:
    35
    @ThomasZeng Thank you for the quick reply! I will try this out immediately and let you know how it goes. Out of curiosity, why does the system report "StereoRenderingMode: MultiPass" at runtime even though it is not supported, as you said?

    EDIT: works for the shader you edited, now I'm having a little trouble with my volumetric shader in that the image in the right eye is the same as that in the left (not stereo), I think that it is related to using _WorldSpaceCameraPos and it now returning the same position for both left and right eye in the shader frag, do you have any suggestions here? Is there an alternative to _WorldSpaceCameraPos for SPI or maybe I should be passing this info in the v2f into frag instead?
     
    Last edited: Sep 16, 2020
  4. ThomasZeng

    ThomasZeng

    Unity Technologies

    Joined:
    Jun 24, 2019
    Posts:
    84
    Hello @ceitel
    Glad to hear it solves the sprite effect. I would assume you were acceessing _WorldSpaceCameraPos inside the fragement shader stage? If that is the case, please try adding UNITY_SETUP_STEREO_EYE_INDEX_POST_VERTEX(i) to your fragment shader. You could find more detail under Post-Processing shaders section in https://docs.unity3d.com/Manual/SinglePassInstancing.html. The main idea behind this magic line is to set up eye index in fragment shader stage so that when you access _WorldSpaceCameraPos, it knows where to pull the data for that specific eye.

     
  5. ceitel

    ceitel

    Joined:
    Jan 3, 2017
    Posts:
    35
    @ThomasZeng I did try this, but am still getting the same image as though _WorldSpaceCameraPos is still returning the same value for both eyes. Any other ideas?


    float4 frag(v2f i) : COLOR
    {
    UNITY_SETUP_STEREO_EYE_INDEX_POST_VERTEX(i);
    float3 viewDirection = normalize(i.wPos - _WorldSpaceCameraPos);
    ...
    ...

    I have even tried using _WorldSpaceCameraPos from within the vert function and storing it as another TEXCOORD in v2f, but no luck.

    monoscopic.jpg
     
    Last edited: Sep 16, 2020
  6. ThomasZeng

    ThomasZeng

    Unity Technologies

    Joined:
    Jun 24, 2019
    Posts:
    84
    Sounds odd, if this is URP project would you mind submit a bug report with the repro case? We can take a deeper look.
    In the meantime, you could also try retrieve position from inv View Matrix(unity_MatrixInvV).
     
  7. ceitel

    ceitel

    Joined:
    Jan 3, 2017
    Posts:
    35
    @ThomasZeng actually it appears that the problem is NOT with _WorldSpaceCameraPos, but is happening earlier with the vertex positions... I'll investigate further, but I am not using URP at this time, just the built in default render pipeline.
     
  8. ceitel

    ceitel

    Joined:
    Jan 3, 2017
    Posts:
    35
    @ThomasZeng so it seems to be an issue with Graphics.Blit(source, destination, blendMat) and/or with my blendShader (attached), OR with rendering to a renderTexture from the MainCamera:

    private void OnRenderImage(RenderTexture source, RenderTexture destination)
    {
    ...
    MainCamera.targetTexture = volumeTex //a RenderTexture;
    MainCamera.Render();
    ...
    this.blendMat.SetTexture("_VolumeTex", volumeTex);
    Graphics.Blit(source, destination, this.blendMat);
    }


    I am wondering if there is a specific way I should create a temprorary renderTexture, ie:
    volumeTex = RenderTexture.GetTemporary(source.width, source.height, source.depth, source.format, RenderTextureReadWrite.Default, 1, RenderTextureMemoryless.None, VRTextureUsage.DeviceSpecific);


    Let me know if anything obvious stands out to you,
    Thanks for all your help thus far!


    EDIT: when I use
    volumeTex = RenderTexture.GetTemporary(source.descriptor);
    I get the following error at runtime:
    Error assigning 2DArray texture to 2D texture property '_VolumeTex': Dimensions must match
    If I change the shader property to a 2DArray, it works for the left eye only.
    I think the standard SetTexture("_TexName", RenderTexture) is mistakenly treating the vr renderTexture as an array, and therefore not allowing it to be assigned to the material
     

    Attached Files:

    Last edited: Sep 17, 2020
  9. ceitel

    ceitel

    Joined:
    Jan 3, 2017
    Posts:
    35
    To summarize a simple repro:

    0. using XRSDK WMR
    1. have a simple post processing Material using a shader that has 2 2D texture properties "_MainTex" and "_VolumeTex".
    2. use the following to make the shader SPI compatible:
    UNITY_DECLARE_SCREENSPACE_TEXTURE(_MainTex); //Insert
    UNITY_DECLARE_SCREENSPACE_TEXTURE(_VolumeTex); //Insert
    3. have an active script that uses OnRenderImage(RenderTexture source, RenderTexture destination).
    4. attempt to assign the source (XR) RenderTexture to the material _VolumeTex from (1).

    Fails with:
    Error assigning 2DArray texture to 2D texture property '_VolumeTex': Dimensions must match
     
  10. ceitel

    ceitel

    Joined:
    Jan 3, 2017
    Posts:
    35
    @ThomasZeng
    After even further investigation, I am able to get the _VolumeTex to work if I do not define it in properties, however, the renderTexture (volumeTex) is still only displaying the left eye, whether I pass it as the _VolumeTex or _MainTex, while the source renderTexture is behaving correctly whether I pass it as _VolumeTex or _MainTex. I think the issue is in using a copied camera to render to the temp render texture. I am creating a repro project now.
     
  11. ceitel

    ceitel

    Joined:
    Jan 3, 2017
    Posts:
    35
    @ThomasZeng

    Narrowed the issue down to rendering from a camera to a render texture does NOT render the right eye.
    Very simple repro:
    Add the attached script to the scene. Press play. Only left eye is rendered.

    Please let me know if there is a variant of Camera.Render() that is SPI compatible, or if this is a known issue.

    Thanks
     

    Attached Files:

  12. ThomasZeng

    ThomasZeng

    Unity Technologies

    Joined:
    Jun 24, 2019
    Posts:
    84
    @ceitel

    Thanks you for isolating the issue! Appreciate your effort here. Unfortunately built-in renderer is not my area. I will ask around internally and get back to you soon. Thank you for your patience!

    Regards,
    Thomas
     
  13. ThomasZeng

    ThomasZeng

    Unity Technologies

    Joined:
    Jun 24, 2019
    Posts:
    84
    Hello @ceitel,
    Would you be able to send us a repro project so that we can do further investigation? That would accelerate the process a lot! Thanks in advance!
     
  14. mikesf

    mikesf

    Unity Technologies

    Joined:
    Jul 14, 2017
    Posts:
    14
  15. ceitel

    ceitel

    Joined:
    Jan 3, 2017
    Posts:
    35
    Will do, I'll try to get this done Sunday!
     
  16. ceitel

    ceitel

    Joined:
    Jan 3, 2017
    Posts:
    35
    @ThomasZeng I have issued a bug report (Case 1279112)
     
  17. ThomasZeng

    ThomasZeng

    Unity Technologies

    Joined:
    Jun 24, 2019
    Posts:
    84
    Thank you @ceitel for sending us the repro project,

    I have looked into the repro project and investigated the issue.
    Unfortunately, this case is by design. Built-in renderer turns off stereo rendering path when camera renders to a targetTexture.

    You have mentioned this use case was working before the migration effort. So I guess you were using the same volume texture for both eyes? If so you could try to configure the targetTexture to be a 2D RenderTexture and use it for both eyes.

    Regards,
    Thomas
     
  18. ceitel

    ceitel

    Joined:
    Jan 3, 2017
    Posts:
    35
    @ThomasZeng
    Yes this is a migration effort from multi-pass to single-pass rendering: when using multi-pass, OnRenderImage occurs twice, once for each eye, where Source and Destination are the expected eyes defined by Camera.stereoActiveEye of the rendering camera, therefore I am able to correctly use Camera.Render() once for each stereoscopic eye. Now, with single-pass, there is only one call to OnRenderImage which passes a stereoscopic image as Source, and expects a stereoscopic image as Destination, however, Camera.stereoActiveEye is now always left, and there is no way to use Camera.Render() to render the right eye to a render texture to do post-processing for the right eye. Is there some method to achieve Camera.stereoActiveEye = Right during single-pass??
     
    Last edited: Sep 22, 2020
  19. ceitel

    ceitel

    Joined:
    Jan 3, 2017
    Posts:
    35
    @ThomasZeng
    It seems that what I'm looking for is a time during rendering that Camera.stereoActiveEye == Right. I am using shaders that need to use shaderReplacement (and other stereoscopic eye specific post-processing rendering):
    https://docs.unity3d.com/Manual/SL-ShaderReplacement.html
    I have checked the value Camera.stereoActiveEye at OnPreRender, OnPostRender, OnRenderObject, and OnRenderImage, all have the value set to Left. At Update and LateUpdate the values are set to mono.
     
  20. ceitel

    ceitel

    Joined:
    Jan 3, 2017
    Posts:
    35
    @ThomasZeng ,

    I am wondering if there has been any further development on this as this issue breaks post-processing with single-pass instanced VR rendering and our team is not able to migrate from Unity 2019 to 2020 until this issue is solved.

    Again the issue lies in Camera.Render() only rendering the left eye during single-pass instanced VR rendering. This was not an issue with multi-pass rendering as Camera.Render() would work correctly for each eye, but now that Unity has removed the option for multi-pass in 2020 we are forced to use single-pass instanced where Camera.Render() no longer works for the right eye.

    Thanks again,

    Chad
     
  21. badawim-idcc

    badawim-idcc

    Joined:
    Dec 10, 2015
    Posts:
    10
    Hmmm, this might also explain the issue I'm having. Will keep an eye on this thread just in case.
     
  22. ceitel

    ceitel

    Joined:
    Jan 3, 2017
    Posts:
    35
  23. jonathanglitchers

    jonathanglitchers

    Joined:
    Jan 5, 2023
    Posts:
    8
    So is this still broken, or just abandoned?
    I'm still only getting stereoActiveEye = Left for OnPreRender(), OnPostRender(), etc.
     
  24. mrphilipjoel

    mrphilipjoel

    Joined:
    Jul 6, 2019
    Posts:
    54
    I'm using shadergraph but having this issue. Everything looks fine when using multipass. But, if I switch to singlepass, it only renders in left eye. The shadergraph appears to have automatically added the key word "UNITY_SINGLE_PASS_STEREO" which from my understanding is what is needed to render to both eyes.