Search Unity

  1. How can our website serve you better? Give us your feedback. Take our survey and let us know.
    Dismiss Notice

Unity Shader keyword system improvements in 2021.2 alpha

Discussion in '2021.2 Beta' started by aleksandrk, May 5, 2021.

  1. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,012
    Hello Unity users!

    TL;DR:

    Unity 2021.2.0a16 changes the way shader keywords work and removes fixed limits on the number of keywords used both per shader and globally.

    First, the long-awaited part: fixed keyword limits are gone. There can now be up to 4.294.967.294 (2^32 - 2) global keywords per project. Local keywords are limited to 65534 (2^16 - 2) per shader or compute shader.

    Next, we introduce a concept of keyword spaces.

    Each shader has its own local keyword space, with no distinction between local or global keywords that existed before. All keywords declared in a shader are local. [EDITED on 08.06.2021] This, in turn, means that #pragma multi_compile and #pragma multi_compile_local do the same thing; the same is true for #pragma shader_feature and #pragma shader_feature_local. We will deprecate the pragmas with “_local” suffix in the future. The "_local" suffix in #pragma multi_compile and #pragma shader_feature directives determines whether the global keyword state can override the local shader keyword. The first appearance of a shader keyword wins in case of a conflict. [END OF EDIT]
    Local keyword space is formed from all keywords declared in the shader, all keywords declared in passes that are added with UsePass functionality and all keywords from the shaders in the fallback chain. If a keyword in a fallback or in UsePass has the same name as a keyword in the main shader, it’s treated as the same keyword and counts only once. Unity automatically adds several builtin keywords to each local keyword space (UNITY_SINGLE_PASS_STEREO, STEREO_CUBEMAP_RENDER_ON, STEREO_MULTIVIEW_ON and STEREO_INSTANCING_ON). Any keywords added by shader compiler access plugins are also automatically added to each local keyword space.
    For compute shaders the local keyword space is formed from all keywords declared in the compute shader.
    The local keyword limit mentioned above applies to keywords in a single local keyword space.

    Global keyword space is completely separate. Prior to rendering keywords from the global keyword space are converted to local keyword space. The conversion is based on keyword names. Enabling a global keyword FOO means that any shader or compute shader that has FOO in its local keyword space will have this keyword enabled.

    A note on performance: local keyword spaces maintain a mapping from the global keyword space. When a keyword is added to the global space, the mapping gets updated on the first conversion from the global space to the local space. For best performance, add all global keywords as early as possible.

    C# API changes.

    We keep the current string-based C# API intact with some minor improvements: Shader.DisableKeyword and Shader.IsKeywordEnabled no longer create a global keyword if it doesn’t exist.

    We also added a new API that is faster and has a clear separation between local and global keywords. Material, ComputeShader, Shader, CommandBuffer and ShaderKeywordSet classes have been extended to work with the new API - methods that accept instances of LocalKeyword and GlobalKeyword structs. Shader and ComputeShader also got a new member keywordSpace that can be used to query the local keyword space of a particular Shader or ComputeShader.
    GlobalKeyword also has an explicit static method to create a new GlobalKeyword. The constructor does not create a new keyword if it doesn’t exist.
    Shader.EnableKeyword and CommandBuffer.EnableShaderKeyword create a new global keyword if it doesn’t exist.
    Shader.DisableKeyword, CommandBuffer.DisableKeyword, Material.DisableKeyword, Material.EnableKeyword and ComputeShader counterparts have no effect if a keyword doesn’t exist. Shader, Material and ComputeBuffer IsKeywordEnabled methods return false for keywords that do not exist.
    We also added Shader.enabledGlobalKeywords and Shader.globalKeywords to query the enabled global keywords and all existing global keywords, respectively.

    Stay tuned for more updates!

    (EDITED on 04.06.2021)
    We added new API to set shader keyword state directly. Material.SetKeyword, Shader.SetKeyword, ComputeShader.SetKeyword and CommandBuffer.SetKeyword are available from 2021.2.0a19 onward.
     
    Last edited: Jun 8, 2021
    DrummerB, eizenhorn, ccjay and 21 others like this.
  2. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,012
    Known issues
    • Filtering objects in the scene hierarchy or opening Prefab variants from the hierarchy spams errors in the console (EDIT: fixed in 2021.2.0a17)
    • GI debug views are not rendering anything (EDIT: fixed in 2021.2.0a18)
    • Draw calls are more expensive (EDIT: fixed in 2021.2.0b3)
    • Material is marked as dirty if enabling a keyword that is already enabled or disabling a keyword that is already disabled (EDIT: fixed in 2021.2.0a18)
    These issues are already fixed and will appear in a later alpha version.
     
    Last edited: Jun 29, 2021
    darger likes this.
  3. jbooth

    jbooth

    Joined:
    Jan 6, 2014
    Posts:
    4,967
    Ah, only 65534 local variants? 640k is enough for anyone, huh?

    (Rushes off to design a shader for 65535 local keywords)
     
  4. Lurking-Ninja

    Lurking-Ninja

    Joined:
    Jan 20, 2015
    Posts:
    7,069
    Word!
     
    Alic likes this.
  5. Invertex

    Invertex

    Joined:
    Nov 7, 2013
    Posts:
    1,146
    Glad to see this has been addressed, it's been showing its age for quite a long while now.
     
    aleksandrk likes this.
  6. Jes28

    Jes28

    Joined:
    Sep 3, 2012
    Posts:
    739
    Good news thanks :)

    Please add ability to disable keyword in editor so trying to enable it for material will lead into some pinky-cyan shader.
    Thus we can control usage of shader variants right in editor
     
  7. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,012
    Not sure I understand. Do you mean "some list of keywords in the Editor that are not allowed to be turned on / show a warning when they get turned on"?
     
  8. Jes28

    Jes28

    Joined:
    Sep 3, 2012
    Posts:
    739
    Mostly right :)

    Show warning is good but most important is pink shader to understand that this option is not allowed
    e.g. LitUberShader I will disable NormalMaps and if someone add normal map in any material it will become pink, or pinky-cyan to distinguish from error shader, and ok there can be warning in log or/and in material inspector that this option was disallowed in project.

    With this tool we can easily create set of shaders with variants that allowed in project and easily manage project ShaderVariant Count. For now we only can write strip scripts make build to test amount of shader variants used (usually 180000 just for Lit) and to see pink shaders that try to use stripped variants. With this tool this can be easily done in editor and may be strip Lit Uber Shader to just 10 variant used in project and all in Editor.
     
  9. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,012
    @Jes28 we'll think about this
     
    Jes28 likes this.
  10. jbooth

    jbooth

    Joined:
    Jan 6, 2014
    Posts:
    4,967
    I think, in practice, these changes mean naming conventions should change from what most people use.

    If my understanding is correct, there really aren't global keywords from the shaders perspective anymore- only local. And when some code calls Shader.EnableKeyword, it sets this as a global keyword and sets this for every shader, regardless of if the intention was for that feature to be local or global.

    Personally I would have preferred that these remain distinct spaces, so someone calling Shader.EnableKeyword to set a global keyword would not affect my local shader keywords. While global keywords are useful, most keywords in the shaders I write and use tend to want to be local, and are expressed as options on a per material basis for the user, not a game wide basis.

    But that said, if this is the new system, I think it means wanting to prefix your keyword naming conventions to avoid collisions, which is kind of the opposite of what people did before local keywords (reusing common keyword names to avoid bloating the number of keywords). In fact, I'd suggest we simply start putting LOCAL or GLOBAL in the name of the keyword to mark it's intension. Sure, if someone sets _LOCAL_NORMALMAP via Shader.EnableKeyword, they will still be setting it as global and changing it for every material anyway, but at least they can infer they are breaking the rules that way, instead of accidentally doing it because the intention is not clear.

    This does make me wonder, if I have something like:

    Code (CSharp):
    1. #pragma shader_feature _ _FOO _BAR
    And I set _BAR on the material, but call Shader.EnableKeyword("_FOO"), who wins? The code is only compiled for _FOO or BAR, not both, and the mixing of global and local space means that either the code has a priority feature, or both will end up getting set..
     
  11. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,012
    Yes, that's correct.

    This would leave some interesting side-effects. Suppose you have a shader that declares a keyword FOO inside a shader_feature_local and it has a fallback that has the same keyword inside a shader_feature. Should it be affected by the global keywords then? Should only the keyword from the fallback be affected?

    That said, we'll consider this, as it's a behaviour change indeed.

    Both get enabled and then either variant can be picked. I think it's mentioned in the manual, but I can't find it right now :) The keyword system worked this way for ages, and I'd like to improve that as well.
     
  12. jbooth

    jbooth

    Joined:
    Jan 6, 2014
    Posts:
    4,967
    Yes, because you explicitly declared that it's control is provided by the material, not the global namespace. To me this seems more sensible, because it's exactly what you told it to do. While the new system is clearly workable, it actually treats global keywords as local keyword overrides, NOT it's own global keyword registry.

    I'm also guessing that it doesn't work bidirectionally - for instance, you might think you could use this new system in your game to disable all normal maps with Shader.DisableKeyword("_NORMALMAP"); But since that just removes it from the global registry, it's going to default back to whatever the material says instead of disabling the keyword globally, because there's not really a clear on/off in the keyword system, rather defined or not. So with this system, you're not really enabling and disabling keywords, but rather overriding them and not overriding them, globally. That is not at all clear from the function naming or description of global/local used. If global vs. local is explicit, then both the code in the shader and in the C# layer is clear and explicit about what it means.
     
    Alic, Prodigga, firstuser and 3 others like this.
  13. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,012
    That's correct as well. But that's how it used to work before the introduction of shader_feature_local :)
     
  14. jbooth

    jbooth

    Joined:
    Jan 6, 2014
    Posts:
    4,967
    wait, so even if you declare a feature as local it gets overwritten by global setting?
     
  15. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,012
    yes, that's how the new system currently works.
    But you said it already:
     
  16. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,012
    Or are we already talking about different "local"? :)
    Right now (2021.2.0a16+) the behaviour is like this: enabled global keywords will enable a keyword with the same name in all local keyword spaces, regardless of whether it's declared as shader_feature_local, multi_compile_local, shader_feature or multi_compile.
     
  17. pvloon

    pvloon

    Joined:
    Oct 5, 2011
    Posts:
    591
    Great news! One request, I think nearly every project has this function:

    Code (CSharp):
    1. void SetKeywordEnabled(string keyword, bool enabled) {
    2.   if (enabled) {
    3.     Shader.EnableKeyword(keyword);
    4.   } else {
    5.     Shader.DisableKeyword(keyword);
    6.   }
    7. }
    And variations for materials etc. Would be good to have as an actual official API
     
    richardkettlewell and Peter77 like this.
  18. firstuser

    firstuser

    Joined:
    May 5, 2016
    Posts:
    138
    Noob question and forgive me if I missed this somewhere but, does this have any impact on materials and how they handle unused keywords?

    For example the current LTS default behavior where if you try one shader and then switch to another very different shader on the same material, you need to manually clean up the disabled/inapplicable fields to avoid issues in some cases.

    Would these changes also mean all local keywords are applied again from scratch or something like that?
     
  19. jbooth

    jbooth

    Joined:
    Jan 6, 2014
    Posts:
    4,967

    Right; that’s the change I am not fond of, because I only see it causing issues. Ideally I’d prefer to define where a keyword is expected to be set, so that a local keyword is actually local, and not local only until some script somewhere that I didn’t write sets it globally but didn’t realize my shader uses the same named keyword.

    otherwise I’ll start naming everything v defensively to minimize this chance “_BS_LOCAL_MYFEATURE”. Workable, certainly, but we have modern conventions like namespaces so we don’t need to do stuff like this anymore. If we are going to keep the overriding behavior than I would suggest we change terminology to match; keywords, and global keyword overrides, or something like that.
     
    Alic, firstuser and GliderGuy like this.
  20. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,012
    @pvloon it's coming later :)

    @firstuser it will keep the keywords that exist in the new shader and drop those that don't.

    @jbooth many projects rely on global keywords overriding material state, so we won't change that.
    Thanks for bringing this to my attention - we didn't consider this particular behaviour (being able to define keywords that cannot be overridden by global keywords) important.
     
    pvloon and firstuser like this.
  21. Baste

    Baste

    Joined:
    Jan 24, 2013
    Posts:
    5,495
    There's a lot of this "just doesn't do anything if it doesn't exist" language in the things you write. Are these silent failures, or will we get a warning that "the keyword you're trying to do something with doesn't exist!".

    Silent failures are always annoying! Shaders gets complex, and it can be hard to spot exactly what's going wrong. If we're not getting told that we misspelled a keyword name, that's going to make debugging worse than it has to be.
     
  22. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,012
    When you create an instance of a GlobalKeyword struct, it will print an error in the console if the keyword doesn't exist.
     
    firstuser and Baste like this.
  23. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,012
    @jbooth We'll fix it - keywords declared as local will not be affected by global keyword overrides. We'll also improve the documentation around global keywords to make it more explicit that these are keyword overrides.
    Thanks for the feedback! :)
     
    Alic, jorikito, Invertex and 5 others like this.
  24. jbooth

    jbooth

    Joined:
    Jan 6, 2014
    Posts:
    4,967
    Awesome!
     
    Alic likes this.
  25. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,012
    @Jes28 regarding your request: I don't it would be beneficial to have this built into Unity. Essentially what you're asking for is a system to validate that certain keywords are not used in the project. I think this can be achieved using the scripting API that exists already (with the exception of rendering stuff in different color).
     
  26. Jes28

    Jes28

    Joined:
    Sep 3, 2012
    Posts:
    739
    Thanks for response :)

    Sad :(

    Do you have plans to address uncontrolled growth of ShaderVariants in project in some another way?
    Because now we dont have any API to do this and you just said that my idea not good enough to be implemented :)

    Do say that shader stripping is API because it is not. It work only in build time and if Build took Night and then we need go through entire game just to know that we dont have pick shaders dont looks like, a solution to issue :)
     
  27. Jes28

    Jes28

    Joined:
    Sep 3, 2012
    Posts:
    739
    If you can add Callback in editor that will be called just before compiling/using shader variant so we can return false or error string and shader will not be compiled than every thing else we can create ourselves.

    Something like this:
    Code (CSharp):
    1.  
    2. #if UNITY_EDITOR
    3. Shader.ShaderVariantCheck += OurChecker;
    4.  
    5. String OurChecker( String[] defines )
    6. {
    7.    //some our check logic
    8.  
    9.   //return default if everything correct
    10.   if( VariantCorrect )
    11.     return defult;
    12.  
    13.   //return error string if variant incorrect so unity will treat this shader variant as shader with errors
    14.   return "Incorrect Variant";
    15. }
    16. #endif
    17.  
    One callback and our life will be a lot easier
    All tooling we can create by ourselves :)

    May be this can be already done in some way in Unity?
     
  28. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,012
    @Jes28 you could build a script that goes over all materials and checks their keywords, and have this script run manually or in CI.
    No, I don't think this is feasible. It will not fit architecturally, but, even more importantly, it will have a detrimental effect on performance.
     
  29. Jes28

    Jes28

    Joined:
    Sep 3, 2012
    Posts:
    739
    Thanks again :)

    Going through all materials can help for start and for build thanks, but for WIP it will not.
    I have found API that can help with WIP part:
    Code (csharp):
    1.  Undo.postprocessModifications += MyPostprocessModificationsCallback;
    and it works perfectly fine in most cases but not with Materials.
    Changing anything on material dont trigger callback :( It this by design or this is Bug and I can report it? If this is by design where can I found similar API for Materials?

    Sorry for questions barely related with topic :)
    This is last one for sure :)
     
  30. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,012
    I'm not sure, what should be reported there. My guess it should work and that you can report this as a bug :)
     
    Jes28 likes this.
  31. Kolyasisan

    Kolyasisan

    Joined:
    Feb 2, 2015
    Posts:
    364
    Will we be able to declare those structs as public static readonly and inline-initialize them? One of the most helpful ways of organizing work for me is to declare various shader stuff in local static classes and have them be there, inline-initialized with Shader.PropertyToId for shader properties and have const strings for keywords.

    If the new system will require us to use this new way of working with keywords, but won't allow for such workflow, then I think it would be a massive downgrade. But I guess that clear separation between global and local keywords on both the C# and shader sides is a nice feature. How would local and global keywords be declared on the shader side?

    And I hope the issue with drawcall cost will be seriously addressed because some systems are already knee-deep in performance issues with them.
     
  32. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,012
    Yes, you can use that with GlobalKeyword.
    LocalKeyword requires an instance of a Shader or ComputeShader class as a parameter.
    If you really want this workflow, you can use the string-based API, but it performs less error checking and is potentially slower.
    The same way there were declared previously.

    I have a branch with a fix already :)
     
  33. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,012
    It's now available in 2021.2.0a19, SetKeyword :)
     
    GliderGuy, firstuser and pvloon like this.
  34. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,012
    @jbooth the fix will be available in 2021.2.0a21.
    Note: if there's a conflict (for example, if a shader declares "shader_feature_local FOO_ON FOO_OFF" and the fallback declares "shader_feature FOO_ON FOO_OFF"), the first one encountered wins.
    The directives are traversed in the order of appearance in a single shader, then UsePass, then fallback.
     
    Invertex likes this.
  35. jbooth

    jbooth

    Joined:
    Jan 6, 2014
    Posts:
    4,967
    Seems great to me!
     
    aleksandrk likes this.
  36. bran76765

    bran76765

    Joined:
    Jul 14, 2018
    Posts:
    21
    I know it would probably be a while out but will this specific feature/fix be backported to other unity versions? I've run into a lot of shader errors now where a lot of stuff gets cut off in the editor now (I assume this doesn't happen in the build though) and while most of it isn't needed, having like 3 pages of the "Maximum shader keywords exceeded" is starting to get annoying. I did try looking into my shaders but it seems almost all of it is needed? I have things like

    KriptoFX
    Crest
    Aura
    Enviro
    CTS/Procedural Worlds

    and more. That's just off the top of my head but all these shaders seem to be getting used for something so I can't just delete half my shaders/take out keywords.

    So will I have to do a deep dive into shaders in my project or will it be eventually getting backported? (I'm currently using 2020.1.6)
     
  37. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,012
    No. The change it too large and risky.

    We increased the global keyword limit for older versions from 256 to 384. You can update to 2020.3 :)
     
    Alic likes this.
  38. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,012
    The draw call performance fix will be available in 2021.2.0b3 :)
     
unityunity