Search Unity

  1. Unity Asset Manager is now available in public beta. Try it out now and join the conversation here in the forums.
    Dismiss Notice

Official Strict shader variant matching

Discussion in '2022.1 Beta' started by aleksandrk, Sep 7, 2021.

  1. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,023
    Hello Unity alpha users!

    By default, when a shader variant is not found in the player build, Unity silently picks a "close enough" variant from the available ones. This has a potential to hide bugs in the keyword setup.
    Unity 2022.1 has a new checkbox in player settings, "Strict shader variant matching", that can be used to alter this behaviour. If this setting is enabled and a variant is not found by the player, an error is printed in the console, specifying the shader, subshader index, pass and keywords that were used for variant search, and error shader is used for rendering.
    This setting is intended as a simple validation layer that reveals keyword setup mistakes. By default it's disabled.

    Happy debugging! :)
     
  2. JohnAustinPontoco

    JohnAustinPontoco

    Joined:
    Dec 23, 2013
    Posts:
    283
    Hah! I discovered the 'fallback' earlier this year while tracking down a particularly nasty shader bug.
    https://uninomicon.com/shaderkeyword

    It'd be nice if the fallback mechanism were documented. How does it choose? Even better to have a warning like "Shader [Keywords] is missing. Falling back to Shader [Keywords]. Renderer [objectpath]". As is it's rather hard to identify when it happens.
     
    JesOb likes this.
  3. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,023
    That's not a fallback, it's a consequence of having global keywords.

    This is the second keyword system improvement (the first one is here). We want to introduce strict rules for resolving keyword conflicts, and make fixing keyword issues easier, but this will come later.
     
  4. leozzyzheng2

    leozzyzheng2

    Joined:
    Jul 2, 2021
    Posts:
    63
    Great! Is there any plan to give this feature to 2020.3.x LTS?
     
  5. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,023
    No.
     
  6. hippocoder

    hippocoder

    Digital Ape

    Joined:
    Apr 11, 2010
    Posts:
    29,723
    Here's a thought - can we specify an "error shader" of our own? Having that control over default fallback is an even safer way to manage this new option. A use case scenario is that the developer wants to manage it strictly but one falls through the net. The dev can identify it but the gamer just thinks "oh, generic wall".

    Revenue unaffected, everyone happy and it's silently fixed by the dev in the know.
     
    Peter77 and ImpossibleRobert like this.
  7. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,023
    I tried this recently for Hybrid... turns out it's not as easy as it sounds :)
    Error shader is quite special, and is treated by the code as such.
     
    hippocoder likes this.
  8. paulatwarp

    paulatwarp

    Joined:
    May 17, 2018
    Posts:
    135
    I am getting "not found" messages even when I don't strip any shaders. What could cause that?
     
    Bovine likes this.
  9. paulatwarp

    paulatwarp

    Joined:
    May 17, 2018
    Posts:
    135
    For example:
    Code (CSharp):
    1. Shader Standard, subshader 0, pass 2, stage vertex: variant <no keywords> not found.
     
  10. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,023
    @paulatwarp This often means the variant doesn't exist at all.
    In your particular case, are there any directives that don't have an underscore? These need one of the keywords to be enabled. It can also happen if two keywords from the same directive are enabled simultaneously.
     
  11. paulatwarp

    paulatwarp

    Joined:
    May 17, 2018
    Posts:
    135
    In the Standard shader?

    Does that mean I have a malformed Material somewhere?
     
  12. paulatwarp

    paulatwarp

    Joined:
    May 17, 2018
    Posts:
    135
    Okay, I've found the model / material. It's an FBX file, imported with a default material and None set in the import settings. When that model is in the scene, the material set is Default-Material. And I get these errors in the console:
    Code (CSharp):
    1.  
    2. Shader Standard, subshader 0, pass 2, stage vertex: variant <no keywords> not found.
    3. Shader Standard, subshader 0, pass 2, stage pixel: variant <no keywords> not found.
    4.  
    The only keyword set on the Material is _EMISSION as far as I can tell.
     
  13. paulatwarp

    paulatwarp

    Joined:
    May 17, 2018
    Posts:
    135
    Scratch that. It's simpler than that. Create a new project, add a cube and leave the default material on it. Then build. I get the same errors.
     
  14. paulatwarp

    paulatwarp

    Joined:
    May 17, 2018
    Posts:
    135
    Also if I create a new Material using the standard shader. This new material does not have the _EMISSION keyword on it by default, but the result is the same two errors as above.
     
  15. paulatwarp

    paulatwarp

    Joined:
    May 17, 2018
    Posts:
    135
  16. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,023
    Yes, please. It will be tracked properly in this case.
    Thank you!
     
    paulatwarp likes this.
  17. paulatwarp

    paulatwarp

    Joined:
    May 17, 2018
    Posts:
    135
    IN-53232

    Thanks
     
    aleksandrk likes this.
  18. paulatwarp

    paulatwarp

    Joined:
    May 17, 2018
    Posts:
    135
    Bug closed as won't fix.

    Unless there's some workaround for this, I can't see this being a useful feature.
     
  19. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,023
    URP and HDRP should work properly with this feature enabled.
     
  20. sebas77

    sebas77

    Joined:
    Nov 4, 2011
    Posts:
    1,643
    Hello, I have a problem with this on Android device.

    In some particular situation if I have it off, the GPU driver just crashes on some devices .

    if I have it on, the game then works, ofc with a ton of pink shaders. And I get weird errors like

    2023/11/08 18:52:39.488 2014 2111 Error Unity Shader Shader Graphs/UnityTerrain_simple_minSpec, subshader 0, pass 0, stage all: variant <no keywords> not found.


    but I prewarm shaders from a shader collection that has:

    upload_2023-11-8_18-55-28.png

    please don't tell me is a bug pleeeeeease.

    I also don't exactly get why it say that is not found instead than just compile a new variant
     
  21. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,023
    When this setting is turned off, it automatically picks some "close enough" variant. If it were to select it when this feature is on, you'd get exactly the same.

    Are there any keyword directives that don't have a
    _
    variant allowed in the shader?

    We don't bundle the shader compiler with the player. Only the Editor can compile new variants.
     
  22. sebas77

    sebas77

    Joined:
    Nov 4, 2011
    Posts:
    1,643
    OK I have more information so I can be more precise and use the same terminology:

    I am aware of this, however on my android device this was resulting, for some reason I couldn't understand because no logs were produced, in the samsung s10 drivers crashing. Only once I have enabled the strict option, the crash disappeared, although obviously the pink shaders appeared. This sent me to the next bit of investigation and to the real solution of the problem.

    the problem resulted to not have anything to do with variances, but it's related to what's written in this post:
    https://forum.unity.com/threads/asset-bundles-and-shaders.806331/#post-8696241

    it is absurd that this is not common knowledge and the users like me have to spend a ton of time to investigate and learn.

    you do, but not what you think I am talking about. The editor compiler generates the final code that is then used by the driver to actually compile the shader for the single GPU, like explained here (random link that explains it):

    https://docs.nvidia.com/jetson/arch...er,control binary program management directly.

    This is what ShaderVariantCollection warmup actually does. It compiles the shaders on the GPU and cache them (on specific devices). However it's not clear to me yet what's the current state of this since on Vulkan it appears it's not enough anymore (https://docs.unity3d.com/ScriptReference/Experimental.Rendering.ShaderWarmup.WarmupShader.html)

    So this is what I gathered so far:

    * our pipeline expects shaders to be bundled in the addressables
    * We put in a shader variant collection the variants we need to be used, so that the client knows what to warmup
    * the bulding process strips out the CODE of these variants that we need. For this reason the client is not able to compile them.
    * to avoid this from happening, I have either to follow what explained in the forum OR fundamentally KEEP ALL the shader variants and disable all the strip options. The use this code https://gist.github.com/guycalledfrank/567df882922d83f3dfe3c19aeeea8cd1 to strip the variants that are not in our shader variants collection
     
  23. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,023
    Ah, sure. Normally when we talk about variant compilation, we mean the Editor. This is compilation by the driver.
    The process here is simple - we first find a variant and then send it to the driver. If we cannot find a variant, you get the pink shader and an error :)

    I see. AFAIK the team that's responsible for asset bundles is doing some improvements :)

    You mean the source assets?

    Avoid what exactly? Prevent the client from compiling variants?

    Yes, the warmup API is there since DX11/OpenGL days. It doesn't provide enough information for modern graphics APIs (Vulkan included) to perform warmup properly.
     
  24. sebas77

    sebas77

    Joined:
    Nov 4, 2011
    Posts:
    1,643
    currently the exact shader variant CODE generated by the editor is not found in the addressables once built, hence the pink shaders. I want to find a definitive solution to be 100% sure that in the bundles there is the code necessary for the player to compile the exact shader necessary and not the closest. I have also ZERO clue why it actually worked for weeks using the closest variance found, but it's now not working anymore. I suspect that simply I cannot trust the process if I don't have totally control over it (hence I need to use the strict version)

    what's the current suggested solution then:? I am thinking about creating a dummy scene with a quad for each shader and load it before to start the game, would it be enough? I have not enough knowledge about modern API, so I am not sure if I should just ignore the problem.
     
  25. npatch

    npatch

    Joined:
    Jun 26, 2015
    Posts:
    247
    I suspect this has to do with Addressables and how the SVC and the shader assets themselves (referenced in the SVC) are bundled in bundles that will eventually be loaded and warmed up on runtime through code. Somehow they are getting stripped when you have the SVC referencing them (but not referenced in a scene that's also addressable) and you want to load the SVC on runtime through the bundle and warm up the shaders so you can then spawn e.g. prefabs that need those shaders.

    I've seen forum posts here where one Unity employee says it's easy, just create an SVC, it references the shader assets, make it addressable and then build the bundle, but obviously this doesn't include stripping guidelines. unity_bill, in another forum thread, said that any shaders not referenced in an addressable scene, will, by default, get stripped, unless you Keep All, or likely do some proper Addressable Group configuration which is not clear what proper actually means.

    Another suspicion of mine is that if you don't put SVC and shader assets in the same group so the group becomes a dependency on other groups, then duplicates will happen so the prefab in another group will not reference the same asset as the SVC's variants.
     
  26. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,023
    This is close. You'd need the exact same vertex layout as the mesh(es) that you're going to use the shader variants with.
     
    sebas77 likes this.
  27. sebas77

    sebas77

    Joined:
    Nov 4, 2011
    Posts:
    1,643
    just an update:

    Removing ALL the kind of stripping and putting shader variant collection, shaders and material in the same bundle fixed the issue.

    The client log is now very clean, showing all the shader compiled correclty with expected variances (strict ones)

    this is a great starting point to now work on the filtering.
     
    Walkinger, JesOb and npatch like this.
  28. thebarryman

    thebarryman

    Joined:
    Nov 22, 2012
    Posts:
    130
    Love this feature. One thing that could make it even more useful (though I'd imagine it might be tough) is if the "missing material/shader asset" shader were a different color than the "missing variant from build" shader.
     
    JesOb likes this.
  29. Walkinger

    Walkinger

    Joined:
    Oct 27, 2022
    Posts:
    5
    How do I handle missing variants of URP internal shaders? I'm using strict shader variant matching and deferred rendering and my player console gets this recurring exception:

    Code (csharp):
    1. Shader Hidden/Universal Render Pipeline/StencilDeferred, subshader 0, pass 3, stage pixel: variant _ADDITIONAL_LIGHT_SHADOWS _DEFERRED_FIRST_LIGHT _DEFERRED_MAIN_LIGHT _DIRECTIONAL _MAIN_LIGHT_SHADOWS _SCREEN_SPACE_OCCLUSION _SHADOWS_SOFT not found.
    2. Shader Hidden/Universal Render Pipeline/StencilDeferred, subshader 0, pass 4, stage pixel: variant _ADDITIONAL_LIGHT_SHADOWS _DEFERRED_FIRST_LIGHT _DEFERRED_MAIN_LIGHT _DIRECTIONAL _MAIN_LIGHT_SHADOWS _SCREEN_SPACE_OCCLUSION _SHADOWS_SOFT not found.
    I tried including the variants using a shader variant collection and adding it to the preload shaders, but that doesn't work:
    ShaderVariants.PNG
    I also tried using a explicitly referenced material that uses the shader and has the keywords enabled. That didn't work either.

    Is there something I'm missing?

    ---

    I replicated the same behavior using the "3D URP" template. I enabled "strict shader variant matching" and swapped the renderer to deferred. Using a development build, I get the same recurring errors.
     
  30. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,023
    @Walkinger Which version of Unity are you using? This may have been fixed already if you're not on the latest patch. If not, please submit a bug report so that relevant people can take a look.
     
    Bovine likes this.
  31. Walkinger

    Walkinger

    Joined:
    Oct 27, 2022
    Posts:
    5
    @aleksandrk

    My project is using 2022.3.11f1.

    I managed to replicate it using 2022.3.13f1. The project is included in the bug report IN-61020.
     
    aleksandrk likes this.
  32. Bovine

    Bovine

    Joined:
    Oct 13, 2010
    Posts:
    195
    I'm seeing this error for a shadergrah shader I created. Do I need to set any keywords anywhere to ensure the right variant exists? Could this be a bug that's been fixed - is this in the latest LTS do you know?

    Silly question, but how are the variants decided upon? When picking a shader variant, what is the engine looking at to determine which variant to pick?
     
  33. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,023
    It could be.
    We take the keywords from the material and enabled global keywords (the engine may enable some as well) and do a logical OR on those and mask to the pass being rendered. If strict variant matching is enabled, that's the variant we're looking for.
     
    npatch and Bovine like this.