Search Unity

  1. The 2022.1 beta is now available for testing. To find out what's new, have a look at our 2022.1 beta blog post.
    Dismiss Notice

VR Single Pass Stereo with [ImageEffectOpaque] OnRenderImage

Discussion in '5.4 Beta' started by anollan, May 27, 2016.

  1. anollan

    anollan

    Joined:
    Jul 4, 2015
    Posts:
    4
    I have an effect that renders a skybox and fog at the end of the opaque queue, using OnRenderImage and [ImageEffectOpaque]. I'm trying to get this effect to work with Single-Pass Stereo, but it seems that it's not behaving the same way OnRenderImage works without [ImageEffectOpaque]. Instead of executing my shader once per eye, it's executed once for the whole buffer. Also, unity_StereoScaleOffset, unity_StereoEyeIndex, and _MainTex_ST don't contain any useful data.

    The biggest problem I'm having is going from screenspace back to worldSpace, so I can sample the skybox texture. Which matrices should I use for this?

    When doing a graphics capture, I see that there are borders around the left and right eye buffers. What's the purpose of these buffers? How can I programatically determine the size of these borders?

    Is there documentation available for Single-Pass integration?
     
  2. Tim-C

    Tim-C

    Unity Technologies

    Joined:
    Feb 6, 2010
    Posts:
    2,163
    Hi, I know in a beta very soon (maybe the next?) we will have many of the standard asset image effects upgraded for single pass stereo. I think it also needs some minor changes on the unity.exe side, so you might have to wait. That being said the standard asset stuff looks like this:

    old
    Code (csharp):
    1.  
    2. fixed4 original = tex2D(_MainTex, i.uv);
    3.  
    new
    Code (csharp):
    1.  
    2. half4 _MainTex_ST;
    3. fixed4 frag (v2f_img i) : SV_Target
    4. {
    5.   fixed4 original = tex2D(_MainTex, UnityStereoScreenSpaceUVAdjust(i.uv, _MainTex_ST));
    6.   ..
    7. }
    8.  
     
  3. anollan

    anollan

    Joined:
    Jul 4, 2015
    Posts:
    4
    Currently UnityStereoScreenSpaceUVAdjust doesn't work, because this is a "ImageEffectOpaque" post process effect, and it seems that Unity forgot about "ImageEffectOpaque" techniques, so they work even less in VR single pass mode. This is unique to "ImageEffectOpaque" effects. If I remove "ImageEffectOpaque", UnityStereoScreenSpaceUVAdjust then works as expected.
     
  4. robinb-u3d

    robinb-u3d

    Unity Technologies

    Joined:
    Jul 13, 2015
    Posts:
    18
    Hi
    The borders are probably areas that we cull out using a mesh written to the nearest value in the depth buffer just after the scene is cleared. This mesh reflects the area that will not end up being displayed in the HMD due to the lens distortion. It varies depending on the HMD in question (DK2 has a very small area, others have a much larger area).

    Can I ask how you are rendering your image effect? Is it using Graphics.Blit() or is it using some other method? Also where are you creating the destination texture that you are rendering to (or where are you getting the temp texture) if you are not rendering directly to the image effect destination?

    Cheers

    Robin
     
  5. anollan

    anollan

    Joined:
    Jul 4, 2015
    Posts:
    4
    Hi Robin,

    Thank you for the reply!

    The rendering of my image effect is fairly simple, I'm using Graphics.Blit(Texture, RenderTexture, Material, int pass), and am not using any intermediate render targets.

    As far as I can tell, unity is then submitting a full-size quad to the GPU, drawing the post-process effect over all pixels in the buffer, including left and right eye, and the bordered areas. The behavior changes if [ImageEffectOpaque] is not specified before OnRenderImage

    -Andrew
     
  6. robinb-u3d

    robinb-u3d

    Unity Technologies

    Joined:
    Jul 13, 2015
    Posts:
    18
    Hi

    I'll add this case to my list to check and fix. It should be using two quads covering each eye and in the case of Graphics.Blit() it should set up the _ST parameter for each texture if we think they are likely to be a VR packed texture to allow you to sample from the correct area related to the eye you are drawing to using the code Tim posted above (which then compiles out to nothing in non single pass vr).

    Cheers

    Robin
     
  7. DylanF

    DylanF

    Joined:
    Jun 25, 2013
    Posts:
    29
    anollan:
    Are you running 5.4.0b20?

    I'm getting "undeclared identifier UnityStereoScreenSpaceUVAdjust" with 5.4.0b20 (32-bit), so this doesn't seem to be available yet. Also, none of the standard asset image effect shaders that came with 5.4.0b20 use UnityStereoScreenSpaceUVAdjust.
     
  8. DylanF

    DylanF

    Joined:
    Jun 25, 2013
    Posts:
    29
    5.4.0b21 seems to have support for it. The fixed example shaders aren't available yet though (not in the standard assets bundle on the 5.4.0b21 beta download page anyway).
     
  9. Tinus

    Tinus

    Joined:
    Apr 6, 2009
    Posts:
    436
    On b21 using UnityStereoScreenSpaceUVAdjust fixed several of my image effects, which is great. On b22 though, they're completely broken again.

    If I modify an effect that worked on b21 to act as a simple pass-through effect (output == input) it still doesn't work at all.
     
  10. robinb-u3d

    robinb-u3d

    Unity Technologies

    Joined:
    Jul 13, 2015
    Posts:
    18
    Hi Tinus

    b22 had a load of fixes for single pass vr go in so in theory should be the better of the two. Can you file a bug with an example of your effect that doesn't work so I can have a look or post an example here?

    Cheers

    Robin
     
  11. Tinus

    Tinus

    Joined:
    Apr 6, 2009
    Posts:
    436
    @robinb-u3d Sure thing! Here's a project with pretty much just the failing effect in it: https://fogbugz.unity3d.com/default.asp?808089_nkn1ode1rb97eiaa
     
  12. Tinus

    Tinus

    Joined:
    Apr 6, 2009
    Posts:
    436
    Here's a screenshot of the intended result (rendered without VR) and the current result (with VR and Single-pass), as taken from the test case included with the report.



    As mentioned, the specifics of the shaders don't really matter. Something seems to be going wrong either before or after the fragment shader, not inside of it.
     
  13. robinb-u3d

    robinb-u3d

    Unity Technologies

    Joined:
    Jul 13, 2015
    Posts:
    18
    Hi Tinus

    So the bad news is that your image fx is currently being drawn using a GL.PushMatrix(); GL.LoadOrtho() sequence before rendering an quad. This is an approach I am currently fixing up for some of the existing standard asset image effects but one that is still broken in b22 for single pass vr as the changes to matrix never make it to the stereo matrices. The good news is that your effect does appear to work with the fixes I have already got (apart from then needed to adjust the UV's to avoid doubling up).

    To work around this issue for the time being if you wanted to you would either have to
    1) Convert the effect over to using Blit() rather than the CustomBlit() script call that draws the Quad manually but then move over to using SV_VertexID to identify vertex that is being used an use that instead of the vertex id being encoded into the depth value.
    2) Transform the verticies of the quad manually in the C# script into clip space and modify the shader to not transform them. You will probably need to also use SV_VertexID for your index here.

    Hope this helps.

    Robin
     
  14. Tinus

    Tinus

    Joined:
    Apr 6, 2009
    Posts:
    436
    @robinb-u3d Thanks, that clarifies a lot. To gauge whether to invest time in this, do you have a loose indication of when this round of fixes will make it into a release? :)
     
  15. robinb-u3d

    robinb-u3d

    Unity Technologies

    Joined:
    Jul 13, 2015
    Posts:
    18
    Hi Tinus

    This batch of fixes is going through review at the moment but I don't have an indication yet of when they might land.

    Cheers

    Robin
     
  16. bgolus

    bgolus

    Joined:
    Dec 7, 2012
    Posts:
    11,344
    When using UnityStereoScreenSpaceUVAdjust() with single pass the right eye blit is offset one pixel to the left when using the Rift (haven't tried on the Vive). This isn't super noticeable with most image effects but it results in the right eye being many pixels off when using bloom or similar image effects that use downscaled blur as 1 pixel of a 4x downscaled image is 4 pixels in the final image multiplied by each downscaling and blur pass. The this means the standard assets blur can be significantly offset by the time it renders to screen. Usually this isn't super obvious since multiple iterations usually results in a diffuse enough blur that the stereo disparity isn't jarring. If you use the bloom optimized script with the blur cranked down as low as possible and the intensity cranked up it's more obvious.

    This is using 5.4b24, didn't check b25 yet.

    This issue is not present on the PS4.
     
    Last edited: Jul 12, 2016
  17. bgolus

    bgolus

    Joined:
    Dec 7, 2012
    Posts:
    11,344
    A related note is I think almost all of the standard effects are using UnityStereoScreenSpaceUVAdjust() incorrectly, though I haven't entirely confirmed it.

    Example use case is often something like this:
    uv = UnityStereoScreenSpaceUVAdjust(v.texcoord.xy + _MainTex_TexelSize.xy * 0.5, _MainTex_ST);

    However I would expect the proper form to be like this:
    uv = UnityStereoScreenSpaceUVAdjust(v.texcoord.xy, _MainTex_ST) + _MainTex_TexelSize.xy * 0.5;

    Assuming it's supposed to work exactly like the TRANSFORM_TEX macro (which it does), and _MainTex_TexelSize is being set the same way it is in the usual non singlepass stereo case then applying the uv adjust to a line with a uv offset will cause the offset to be half as narrow as it should be. The result is down sampling passes will introduce vertical aliasing, and blur passes will blur more vertically than horizontally.


    Also, unrelated to these issues, the down sample pass in MobileBloom.shader is done asymmetrically causing it to be way more aliased than it needs to be. The first sample is done with an offset of 1 texel, and the other 3 are half texel offsets. With the default hidden "low" resolution preset they should all be 1 texel offsets, and the "high" should be 0.5. Pretty sure this bug has been there for a few years.
     
    Last edited: Jul 12, 2016
  18. robinb-u3d

    robinb-u3d

    Unity Technologies

    Joined:
    Jul 13, 2015
    Posts:
    18
    Hi

    The trouble with the Rift and single pass stereo is related to the default RT size which is different on the PS VR. We have a fix to help this issue coming in the build after B25. This will round up the size of the packed eye targets when using single pass vr to help avoid these issues.

    You are correct in your understanding on the UnityStereoScreenSpaceUVAdjust calls. I must have been a little to heavy handed in a few of the fixups as the texel offsets should be done after the adjustment as the size of the texel is correctly calculated. If you could open a bug about the effects that have UnityStereoScreenSpaceUVAdjust done as in your first case (or at least one sample effect) then that will help us track it at this end and fix those up.

    Cheers

    Robin
     
  19. bgolus

    bgolus

    Joined:
    Dec 7, 2012
    Posts:
    11,344
    The UnityStereoScreenSpaceUVAdjust issue has been posted as issue #813534
     
  20. dtavares

    dtavares

    Joined:
    Feb 24, 2016
    Posts:
    4
    Hey @robinb-u3d, have these fixes been released in one of the beta releases yet? I'm using 5.4.0f1 RC 1 and fixing the ColorfulFog shader, which we use in our game. Looking at the GlobalFog.shader I was able to fix the double rendering issue that was present. Now the problem is that the canvas and some other meshes with some image effects are rendering upside down. One of these meshes uses the Particles/Additive material, if I change it to the standard shader, it fixes the flipping but doesn't render the transparency correctly.

    Thanks!
     
  21. dtavares

    dtavares

    Joined:
    Feb 24, 2016
    Posts:
    4
    Turns out the GlobalFog effect that ships with the standard assets package also has this issue of rendering things upside down. I submitted a test scene with the bug on issue #50895.
     
unityunity