Search Unity

Bug GPU Crash when using lightloop scalarization

Discussion in 'High Definition Render Pipeline' started by Kolyasisan, May 24, 2023.

  1. Kolyasisan

    Kolyasisan

    Joined:
    Feb 2, 2015
    Posts:
    397
    The title should be self-explanatory.

    When using DXC and enabled SM6.0 wave intrinsics, the Lit Forward+ shaders will crash the GPU in the lightloop.

    However, I cannot determine the specific issue, as even though it is possible to embed debug symbols into the shaders with -Zi DXC flag or enabled_d3d11_debug_symbols in shaderlab, the debug tools like PIX For Windows or Nsight Aftermath + Nsight Graphics still cannot access this info and pinpoint the exact instruction or HLSL line that causes this issue. Nsight strictly requires a separate PDB file.

    I tried to fork over and compile DXC myself in order to enforce the export of PDB files. However, despite everything seeming correct with the CLI arguments, those still never get exported.

    What should I do? Some help is needed. Forward+ seems like a useful feature for our specific case.

    Unity version is 2022.2.18. The GPU is RTX 3060 Ti (happens on a separate machine with RTX 3070).
     
  2. Kolyasisan

    Kolyasisan

    Joined:
    Feb 2, 2015
    Posts:
    397
    Investigating this issue further I've determined that this issue happens due to the lightloop while cycle never breaking when evaluating punctual lights and/or reflection probes. I wrote a simple debugger to indicate which fragments behave this way and enter an infinite loop. However, none of the visible fragments report it being this way.

    This leads me to believe that this may be an issue with helper lanes that run for masked fragments on the edges of primitives to provide DDX/DDY functionality. On GameCore (judging by the code) there is actually a check if a current thread in a warp is a helper lane, which early-outs the entire loop. However, for some reason, DXC refuses to compile this intrinsic and gives me an error that it doesn't exist.

    Any ideas on this? Isn't WaveIsHelperLane() a first-party feature in SM6.0? I triple-checked to make sure that it only runs in a fragment shader, as it's the only domain where the intrinsic is supported.
     
    Last edited: May 25, 2023
  3. Kolyasisan

    Kolyasisan

    Joined:
    Feb 2, 2015
    Posts:
    397
  4. Kolyasisan

    Kolyasisan

    Joined:
    Feb 2, 2015
    Posts:
    397
  5. Qleenie

    Qleenie

    Joined:
    Jan 27, 2019
    Posts:
    867
    could it be that you posted to the wrong forum? Forward+ is URP specific afaik.
     
  6. Kolyasisan

    Kolyasisan

    Joined:
    Feb 2, 2015
    Posts:
    397
    HDRP has a Forward+ rendering path as well, in addition to tiled deferred.
     
  7. SebLagarde

    SebLagarde

    Unity Technologies

    Joined:
    Dec 30, 2015
    Posts:
    934
    Hey, we are investigating this issue, we will come back to you. And yes HDRP have both forward and deferred (at the same time - Lit Material are deferred or forward and every other Material type are forward (Fabric, stacklit,eye , transparent etc...)
     
    Kolyasisan likes this.
  8. mgeorgedeveloper

    mgeorgedeveloper

    Joined:
    Jul 10, 2012
    Posts:
    324
    I guess what was confusing what the "Forward+" terminology, which I've only ever seen in the context of URP.
     
  9. gregorymitrano

    gregorymitrano

    Unity Technologies

    Joined:
    Sep 2, 2021
    Posts:
    2
    Hey so I looked into this a bit and I wasn't able to reproduce the hang locally. (I'm using an AMD GPU though) My setup was just the HDRP sample scene with everything forced to the forward path. This particular setup must manage to avoid the hang condition somehow. :(

    Unfortunately, regardless of the hang, I didn't see much of a performance difference. I double checked the shader assembly in RenderDoc too to verify that we were actually running the scalarized path.

    As for the WaveIsHelperLane() intrinsic, I think there might be a documentation error? The helper lane functionality was actually added in shader model 6.6 according to this link. The MSDN docs claim it's 6.0 here though even though the intrinsic is missing from the main SM 6.0 page. This likely explains why the compiler can't find the function when you use it since it typically only compiles against SM 6.0 by default when compiling with DXC.

    How exactly were you triggering DXC compilation? It's possible that you can force the compiler into enabling SM6.6 by adding:

    `#pragma require Int64BufferAtomics`

    This is definitely a hack and has side-effects, but it might allow you to confirm if the helper lane logic fully fixes the problem or not.
     
    Kolyasisan likes this.
  10. Kolyasisan

    Kolyasisan

    Joined:
    Feb 2, 2015
    Posts:
    397
    Thank you very much for looking into it.

    I was triggering the DXC compilation simply by using #pragma require with a commented out pragma target. Additionally, I needed to forcefully define PLATFORM_SUPPORTS_WAVE_INTRINSICS to run with scalarization, as it was not getting defined when using DXC and requesting wave features. I was using the HDRP sample scene as well.

    I will get back later today after investigating further and trying to request an SM6.6 feature to manually enable support for WaveIsHelperLane(). I'll also try running it on an AMD GPU as well (strange that I never even thought of doing that in the first place). I always assumed that it was a 6.6 feature thanks to that page, thank you for clarifying.
     
  11. Kolyasisan

    Kolyasisan

    Joined:
    Feb 2, 2015
    Posts:
    397
    Requesting an SM6.6 feature simply fails to compile the shader. The FallbackError shader is used instead. There are no compilation errors logged in either the shader inspector or the console. Trying to inspect the shader through the "Compile and show code" button never compiles the shader and litters the memory until Unity crashes (I'm on a 32 gig system).

    And indeed, there appears to be no such issue on my AMD Vega 6 embedded GPU. Tested on one more Nvidia device and sure enough, the crash happens there as well.
     
    Last edited: Jun 8, 2023
  12. Kolyasisan

    Kolyasisan

    Joined:
    Feb 2, 2015
    Posts:
    397
    Right, and completely forgot to check Editor.log for additional information. Saw these lines when the shader is being compiled with the request of SM 6.6 features:


    WARNING: Shader Unsupported: 'HDRP/Lit' - All subshaders removed
    WARNING: Shader Did you use #pragma only_renderers and omit this platform?
    WARNING: Shader If subshaders removal was intentional, you may have forgotten turning Fallback off?


    Turning off the fallback completely and removing #pragma only_renderers produced this warning message in the shader inspector:


    HDRP/Lit shader is not supported on this GPU (none of subshaders/fallbacks are suitable)


    This sounds a little strange though, as the device this was compiled on most definitely supports SM 6.6 and on latest drivers to boot
     
    Last edited: Jun 8, 2023
  13. gregorymitrano

    gregorymitrano

    Unity Technologies

    Joined:
    Sep 2, 2021
    Posts:
    2
    Hmmmm odd, I did try this on an internal build and I was able to get a fragment shader (not HDRP/Lit, just HDRP/FinalPass) compiling at SM 6.6 with the extra requirement. (Confirmed both vertex and frag stages were compiled as 6.6 in the DXIL)

    I also figured out that the intrinsic is called "IsHelperLane()" not "WaveIsHelperLane()" like the MSDN documentation suggests... Correct naming is in this blog post: https://devblogs.microsoft.com/directx/hlsl-shader-model-6-6/. Once I used the correct name, it compiled ok and I can verify the intrinsic usage appears in the fragment shader DXIL.

    I'll try again on 22.2 next week and see if I can reproduce the compilation failure you're seeing.
     
    Kolyasisan likes this.
  14. Kolyasisan

    Kolyasisan

    Joined:
    Feb 2, 2015
    Posts:
    397
    Reporting back in.

    Using IsHelerpLane() instead completely fixes the issue on the NVidia hardware. It works even when the shader is compiled with SM 6.0 (as noted in the blogpost that the feature can be emulated on previous shader models).

    EDIT: this was a false positive. The crash still happens regularly, altough slightly less often.
     
    Last edited: Jun 13, 2023