Search Unity

  1. Good news ✨ We have more Unite Now videos available for you to watch on-demand! Come check them out and ask our experts any questions!
    Dismiss Notice

Shader keyword limit

Discussion in 'Shaders' started by LukasCh, Aug 15, 2018.

?

Which solution works best for you?

  1. 1

    63.0%
  2. 2

    4.0%
  3. 3

    33.0%
  1. holyfot

    holyfot

    Joined:
    Dec 16, 2016
    Posts:
    35
    I'm also still having the famous and horrible 256 shader keyword limit issue. Even in 2019.3 HDRP :(
     
    GameJob2017 likes this.
  2. holyfot

    holyfot

    Joined:
    Dec 16, 2016
    Posts:
    35
    Create a HDRP project with PostProcessing, it uses 189 keywords (91 global). Install a few assets and you're already over the limit! This issue has been going on for years and is even worse in HDRP, unity really needs to bump it up to 1024 keywords.
     
    Last edited: Mar 27, 2020
  3. darger

    darger

    Joined:
    Aug 30, 2012
    Posts:
    12
    I purchased a lot of assets at Assetstore for large scale game development.
    I imported them into my project and ran into a "Maximum number (256) of shader global keywords exceeded" error!
    In short, Unity cannot effectively use many third-party assets at the same time! Such issues have not been sufficiently announced in advance and are unexpected to me.
    I was pushed into hell from my dream game development. What did the money I paid for so many assets mean?

    How can I solve the problem?
    I can't determine which third-party assets use Shader.EnableKeyword. So it is difficult to replace global keywords with local keywords.

    The global keyword limit should definitely be increased from 256 to 10,000.
     
    GameJob2017 likes this.
  4. sunseeker1988

    sunseeker1988

    Joined:
    Jul 10, 2014
    Posts:
    455
    Good luck, I have 3 threads and none of the solutions worked. I created a new project. I think unity 2019 addresses this issue. You might have to create a new project in unity 2019 like I did and export your entire project as an asset from your current project and u;load to your new one.
     
    GameJob2017 likes this.
  5. holyfot

    holyfot

    Joined:
    Dec 16, 2016
    Posts:
    35
    I've tested the new unity 2019.3.7, they still haven't increased the limit sadly.
     
  6. Bartolomeus755

    Bartolomeus755

    Joined:
    Jun 20, 2013
    Posts:
    257
    The same problem for me after upgrading to HDRP. Had only a few assets form NM installed and now after the upgrade I get the annoying error.
     
    holyfot and sunseeker1988 like this.
  7. cordmeister

    cordmeister

    Joined:
    Apr 1, 2018
    Posts:
    1
    Shader Keywords are by far my biggest headache at the moment, so I feel everybody's pain. If you attempt to use a collection of amazing third-party visual assets from the app store, it seems you're DOA. I use Dissolve shader, curved world, wireframe, living particles, and others in my current project, and it's become impossible to craft a build. I'm utilizing Kronnect's Shader Control at the moment to try to tame the beast, but I really feel for those of you who have already gone through this and reached a total impasse. How can Unity NOT address this in a way that removes the blockers that exist for those of us who are obviously purchasing so many assets on the Asset Store? It's like preventing the highest-spending grocery shoppers from eating. If I can't build, what am I doing with all this time and effort? It negates the entire enterprise of game development. As someone who has every opportunity to jump over to Unreal (and was advised to do so long ago), it makes it hard for me to recommend Unity or justify continuing my current project in such a fickle architecture.
     
  8. ftejada

    ftejada

    Joined:
    Jul 1, 2015
    Posts:
    614
    Totally agree ... This problem is a real Hell. I only have three or four assets from the store with an HDRP project and I already got the problem.
    And the thing is that there doesn't seem to be a solution to fix this ...
    And frankly between this and other big problems that HDRP is dragging, the loss of the pre-calculated Global GI is enough to think about moving to Unreal
     
  9. FeastSC2

    FeastSC2

    Joined:
    Sep 30, 2016
    Posts:
    893
    @LukasCh Can you please explain how to detect which shaders are using the Shader Keywords?
     
  10. holyfot

    holyfot

    Joined:
    Dec 16, 2016
    Posts:
    35
    FeastSC2 likes this.
  11. FeastSC2

    FeastSC2

    Joined:
    Sep 30, 2016
    Posts:
    893
    @LukasCh I think it still would be nice to bump up the amount to 512 or 1024 seeing as not every shader keyword can be made local.
    Not only that but with HDRP, URP, etc... many assets have variants of the same shader and it drastically increases the amount of global shader keywords.

    This includes the packages provided by Unity. What's worse is that the Unity packages are not modifiable. You can't delete their HDRP shaders if you're not in HDRP just because it's in the Packages folder.
     
  12. LukasCh

    LukasCh

    Unity Technologies

    Joined:
    Mar 9, 2015
    Posts:
    84
    Problem is that increasing cap will solve problem only temporary, we did this many times before and it only delayed issue. We just need a higher adoption of local keyword in asset store and we trying to figure out how to archieve that.
    For now you can always try to convert ure downloaded asset store shader shader_feature into shader_feature_local (In most cases its safe conversion, but not in all. For this reason we can not do this automatically on our side)
    However we do not disregard posbility of increasing keyword limit especially with HDRP now.

    We are currenlt aware of this. This is even a bigger issue if project contains URP and HDRP at the same time.

    Actually there is many ways to modify it. You can find more info https://docs.unity3d.com/Manual/Packages.html. As we have SRP as public repo https://github.com/Unity-Technologies/Graphics.

    Currently we do not have a way to show which global keywords are used and its something we are working on. It is not as trivial as it migh look from first look, because global keyword can be simply crearted via script without having any connection with shaders.
     
    FeastSC2 likes this.
  13. darger

    darger

    Joined:
    Aug 30, 2012
    Posts:
    12
    You don't have to limit the number of global keywords in shaders.
    What happens if you remove the restrictions? If there are too many global keywords in the shaders, the editor performance will be slightly degraded. It's a small problem.

    In exchange, the current global keyword limit is causing serious problems that users cannot solve! You wrote that it's not safe for editor to automatically convert global keywords to local keywords. So is there a way we can do it safely?
    Importing the HDRP package into an existing project and exceeding the keyword limit will cause a mysterious crash of editor when upgrading materials!

    While it's great that many Asset Store assets adopt local keywords to improve editor performance, it can be done even after the global keyword limit have been relaxed or removed.
     
  14. sunseeker1988

    sunseeker1988

    Joined:
    Jul 10, 2014
    Posts:
    455
    I know everything you guys work on requires resources and time. I feel like somethings IF possible, give users the ability to take a different route. I tried to fix the keyword issue but it did not work as many people who have this issue run into. We have many different levels of people experienced with c#. Its sometimes possibly forgotten when you have such good developers writing fixes, coming up with solutions and long term solutions that many developers using unity have programming experience entirely learned from unity tutorials and they find great difficulty in doing things you might see as easy to address.

    Sometimes for your less experienced group of unity users, if it wont wreck stuff, give them an option to turn on an ability(higher keyword limit, or some work around). if it degrades performance, give them the option to choose, like "hey, if your having this problem, you can turn on this feature but forewarned it will or possibly degrade performance. People who are dead in the water or are totally stuck could use these features. This might not be a realistic idea, but it would help alot of people who are like "ok, im not experienced enough with fixing these problems that are fixable by me, so Ill choose the easier option at the expense of blah blah...
     
  15. holyfot

    holyfot

    Joined:
    Dec 16, 2016
    Posts:
    35
    Right now the "local keywords" feature doesn't make a difference. You cannot change multi_compile keywords to local without messing up the whole shader (stuff starts glitching out, crazy artifacts). Changing shader_feature keywords to local increases compile time exponentially. What used to take 10-20min to compile now takes 14+ hours. I reverted the shaders and took out the local keywords, back to 10-20min compile times. We need a legitimate solution: to increase the global shader keyword limit to 1024 atleast!
     
    Last edited: May 11, 2020
  16. HerVap

    HerVap

    Joined:
    Oct 22, 2019
    Posts:
    5
    I now run into this problem too, please we need a solution , why put a assetstore to buy stuff and limit us to use just a few things , when there is a lot of things to be used, Unity have us tied here.
     
  17. darger

    darger

    Joined:
    Aug 30, 2012
    Posts:
    12
    It's surprising that Unity hasn't solved this problem, which is more serious than any nice new feature.
    No one wants to limit the number of shader global keywords.
    This limitation plays no role other than ruining everyone's project one after another.
    For example, Does anyone want to limit the number of 3D models in a project because it takes a long time to import assets?
    If anyone on the Unity team tries to make a game using HDRP, he will immediately notice the problem. He just imported some major assets from the Asset Store and the project crashes.
    "Oh, this is a problem. Why didn't anyone ever notice this problem?"
    He modifies a line of Unity Engine source code. that's it.
     
    naby7u likes this.
  18. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    1,660
    Bumping the keyword limit will solve it only temporarily, at the expense of additional memory cost and lower performance of some functions.
    Don't worry, we'll figure it out.
     
  19. kraihd

    kraihd

    Joined:
    Sep 10, 2015
    Posts:
    20
    If the limit is not a internal constant but a ProjectSettings variable, we can easily avoid the extra memory cost.
    There already seems to be some values like that, for example, the audio voice count, the async upload buffer size, or the input manager keys.

    LukasCh said:
    > I fear that users would set it always to maximum to avoid headache in future (Even though if we had default like 256).

    I don't think so because none of the above values get into such a situation.
     
  20. darger

    darger

    Joined:
    Aug 30, 2012
    Posts:
    12
    Unity has chosen to push the project's destruction switch to keep memory costs down.
    Unity is aware of the problem but does not fix it. Perhaps they think it's an asset publisher's problem, not theirs. They may think that all assets correspond to local keywords as a fundamental solution to the problem.
    Recall that local keywords are only available since 2019.3 and asset publishers must continue to support earlier versions as well. All assets may use local keywords in the distant future, but that is a very long term solution.
    We need to wait long-term until the problem is resolved. Meanwhile, we will continue to pay for Unity subscription.
     
  21. Invertex

    Invertex

    Joined:
    Nov 7, 2013
    Posts:
    1,078
    A nice temporary solution for now that should be an easy implementation for the Unity team would be to have the keyword limit be auto-adjusted instead of manually set by users if there is a worry about them setting it higher than needed. Unity is already able to check how many keywords are in use, so it should be able to just set its internal limit to the next tier in accordance. If a project is growing too big in keywords, all you have to do is pop a little warning in the console so that users at least know why their performance has been degraded, taking any blame out of your hands.

    There's no downside to this solution really, the only thing I could see being argued is that it will take some pressure off asset developers to actually move to local keywords, which I guess I could agree with a bit, but I'm not sure that's really worth the downsides of not doing this to alleviate this issue for users. Asset devs switching to local-keywords should be a store enforced rule not a rule pushed by the complaints of users, that's kind of bad PR.
     
  22. IN_Yves

    IN_Yves

    Joined:
    Apr 22, 2020
    Posts:
    2
    Hey guys... i'm trying to do some sort of tool to track all used keywords so we will know how far or how close we are of consume all available keywords but all my ideas had failed so far. Any ideas on how to do that?
     
    Last edited: May 19, 2020
  23. holyfot

    holyfot

    Joined:
    Dec 16, 2016
    Posts:
    35
    Unity blames it on asset store devs, asset store dev's say it's unity's issue. I agree with the asset store devs, it is Unity's fault. Honestly after all this I might as well just go to Unreal. Unreal doesn't have these shader keyword limitations, and soon they won't have geometry limitations.
     
    Last edited: May 23, 2020
    naby7u likes this.
  24. HerVap

    HerVap

    Joined:
    Oct 22, 2019
    Posts:
    5
    Like when? it's been a problem for years.
     
    Sabso likes this.
  25. cloverme

    cloverme

    Joined:
    Apr 6, 2018
    Posts:
    57
    Any reasonable workarounds on this? This is awful, there's got to be someone on the dev team there that can help at least write something to go through the shaders and change them all from global to local or something. Imagine our frustration in watching all the Unity HDRP propaganda then trying to build a game and get these engine (yes, engine stop blaming other developers) errors in our consoles? Unity needs to step up to the plate on this shader situation, this is bonkers and highly disappointing.
     
    Sabso likes this.
  26. vlery

    vlery

    Joined:
    Jan 23, 2017
    Posts:
    1
    Hi, I really appreciate this useful tickets that people can discuss something about shader management here :)
    I think the decription of the unique index assignment is very helpful and really want more details. I believe the assignment happens at runtime. So when a shader is compiled or a material instance is created, the counter increases. while I have notice some tools help to remove unused keywords from one material which doesn't declare in the its shader. Does this really help to control the keyword used at runtime? If it is, the I suppose 'Material.EnableKeyword' will be called when a material is used and the unused keywords will take up the limit positions. And I actually notice something even worse, when artist change shaders for a material, the keywords won't be clean up (I mean the 'm_ShaderKeywords' attribute in a material file).
     
  27. DavidBVal

    DavidBVal

    Joined:
    Mar 13, 2017
    Posts:
    128
    This issue was a real pain for us, until we realized in our case it was connected to the 4Gb scene limit of Unity (pre-2020.1). Once we fragmented our assets into some "loader" scenes, the max keyword errors went away. I wish Unity offered more informative error messages sometimes...
     
  28. BilginSahin

    BilginSahin

    Joined:
    Jul 2, 2012
    Posts:
    45
    Same Problem here. is there any fix?
     
  29. holyfot

    holyfot

    Joined:
    Dec 16, 2016
    Posts:
    35
    "#pragma shader_feature" you can change to local by just making them "#pragma shader_feature_local". The Shader Control asset can help with this an finding all your shaders, if you don't need them you can uncheck them. Make sure to close unity before making these edits. In HDRP there is a DeferredTile.shader in 2 places:

    C:\Users\YOUR_USER\AppData\Local\Unity\cache\packages\packages.unity.com\com.unity.render-pipelines.high-definition@7.4.1\Runtime\LightLoop

    and:

    Project\Library\PackageCache\com.unity.render-pipelines.high-definition@7.4.1\Runtime\Lighting\LightLoop

    Open both DeferredTile.shader and find the line "#pragma multi_compile VARIANT0 VARIANT1 VARIANT2"

    and change it to: #pragma multi_compile_local VARIANT0 VARIANT1 VARIANT2"

    That will free up 29 shader keywords. You have to manage your shaders & shader keywords to keep the count down. I have successfully gotten my project's keywords down enough that it is feasible.
     
    Last edited: Sep 11, 2020
    Rowlan likes this.
  30. Bechmann

    Bechmann

    Joined:
    Nov 5, 2017
    Posts:
    8
    So yeah I have the same problem "Maximum number (256) of shader global keywords exceeded" any fix yet? I have a handfull of asset form the store in my project. And plan was to use even more, but this is kind of a showstopper. I have been working on this project for quite a while, and for a single developer that relies on third party assets this is a serious limitation.
     
    Rallix likes this.
  31. LukasCh

    LukasCh

    Unity Technologies

    Joined:
    Mar 9, 2015
    Posts:
    84
    Hey, I am sorry to hear that. Seems like the problems your having comes mainly from HDRP, so you should try reaching out help there too. In case they added more global keywords that now causing the limit.


    As I do not have you project to check, I am guessing you still have your global keywords set in material. So once the unity is opened again it sets those keywords.
    I recommend:
    - Replace shader_feature -> shader_feature_local and don't touch multi_compile.
    - Restart Unity.
    - Check if you still getting these errors.
    - If you do, you might still have too many multi_compile from your additional asset store packages. This is not something we can fix on our end yet.
     
    Rallix likes this.
  32. LukasCh

    LukasCh

    Unity Technologies

    Joined:
    Mar 9, 2015
    Posts:
    84
    Good catch, I showed this to HDRP team and they will fix it.
     
  33. DavidBVal

    DavidBVal

    Joined:
    Mar 13, 2017
    Posts:
    128
    It has been months and no solution seems to be in sight from Unity, @LukasCh . Right now our project is at risk, please do something! Worse performance is bad, but right now we just CAN'T build our product, which is far worse.
     
    naby7u and ftejada like this.
  34. Leniaal

    Leniaal

    Joined:
    Nov 7, 2012
    Posts:
    77
    This only makes me worried more. This just means it will take a long time. This issue is such a PITA and especially annoying if the issue is caused by shaders that won't even be in the final build of the game, but just for editor functionality.
     
  35. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    1,660
    Well, it's not an easy task :). We are aware of the problem and are going to fix it ASAP.
     
    Sabso, DavidBVal, Rallix and 2 others like this.
  36. Tech-Byte

    Tech-Byte

    Joined:
    Dec 5, 2014
    Posts:
    33
    Started a new HDRP project and w/ just a couple assets from the asset store, HDRP I'm at 340 keywords and I can't reduce the count any lower. 2019.4 LTS latest version. States the keywords are set to READONLY when I try to automate them to local.

    I really would love to be able to just have the option to with 2019.4 to choose 256, 512, 1024 keywords and deal with it on my end, with the understanding of a performance decrease, just to not have a memory leak bog down my client to a crawl.

    Why is this not just an option for those of us who do not want to move to a later version of Unity until the newer versions are LTS? You can see threads all the back from 2014 complaining about the 256 keyword limit, why has this not been solved already? Something can only be so complex to the point where it seems like it is outright being ignored.
     
    Akshara likes this.
  37. jorikito

    jorikito

    Joined:
    Apr 20, 2018
    Posts:
    1
    I'm in the exact same situation as you are.. I cannot remove any more keywords since they're read-only.

    Why push so much to make developers buy more and more assets on the asset store that they're not going to be able to use anyways?

    So frustrating.
     
  38. Emiles

    Emiles

    Joined:
    Jan 22, 2014
    Posts:
    38
    https://bitbucket.org/ArtIsDarkGames/shader-keywords-tool-unity3d/src/master/
    This was shared on another post. might be usefull for people. My project, which fortunately isn't critical, uses Gaia Pro and VegetationsStudio and its basically borked.

    HDRP is such a loverly idea, wish i hadn't botheedr though as its been one massive setback from the get go. Its definately not LTS, unless your a studio that writes everything for themselves and isn't dependent on assets from the store.
     
    Tech-Byte and Sabso like this.
  39. julianr

    julianr

    Joined:
    Jun 5, 2014
    Posts:
    1,152
    upgraded to 2020.x and still an issue. Lots of admin changing them to local then.. or create a workable fix. Please address it Unity :)

    Shader Control is a quick fix for now. But we shall see, as not all shaders can be switched to local.
     
    Last edited: Oct 4, 2020
    Tech-Byte and Sabso like this.
  40. TJHeuvel-net

    TJHeuvel-net

    Joined:
    Jul 31, 2012
    Posts:
    535
    It actually is quite easy.

    The important part, for us at least, is thats its any keyword enabled on the material.Whether it exists or not in the current shader is irrelevant, it counts to this global maximum. That, combined with keywords hardly ever being cleared is a recipe for disaster.

    Our own materials, and materials from packages, have had lingering keywords enabled which quickly got us to the global maximum, without even actually using them. For example; i assign the standard shader and assign a normal map. Several keywords are enabled, including `NORMAL_MAP`. Then i switch the shader to unlit, now the `NORMAL_MAP` keyword is still there, counting towards the global limit despite it not being used at all.

    We've had materials from packages that came with all sorts of keywords enabled, and for me it was non-obvious that even keywords that dont have a multi_compile or shader_feature count towards this maximum. I made a quick utility to strip these ghost-keywords, though to be honest this should be baked in somehow. I find all shaders in our project, analyse the keywords defined and then find all materials and strip keywords not defined in their particular shader. It wasnt (easily) possible to open the shader source of built-in shaders, but luckily most of our shaders are custom.

    We need much more information about this limit, and how to avoid it. For example:

    1. Expose an API to get the global shader keywords, instead of having to open the `EditorEncounteredVariants` file in a hex editor
    2. Introduce a way to find the keywords defined per shader, the variants that are known
    3. Automatically strip out keywords that dont exist anymore in the Material Inspector
     
  41. DavidBVal

    DavidBVal

    Joined:
    Mar 13, 2017
    Posts:
    128
    What Unity should do today better than tomorrow is to buy Kronnect's Shader Control component and integrate it into Unity like they did with TextMesh Pro, then enhance it into a built-in tool that can also clean up all those materials with unused keywords.
     
  42. firstuser

    firstuser

    Joined:
    May 5, 2016
    Posts:
    51
    The lack of cleanup of unused keywords when switching shaders in particular has been such a difficult battle, even with third party tools it just feels really broken that it works like this by default... hopefully there will be a solution one day :(
     
  43. Sabso

    Sabso

    Joined:
    Sep 13, 2019
    Posts:
    1
    Unity should have a solution for this. It has been 2 years.

    This is a nightmare for people that use the asset store. The fact it is still an issue is like a slap in the face from Unity for spending money on the asset store. It's like 2 slaps in the face if you buy an asset that has anything to do with shaders.

    You're probably buying the asset because you don't have great technical skill with shaders and then it puts you over the limit for shader keywords, leaving you with the diabolical problem of needing to learn about shaders to fix the problem. Why did you buy the asset? Oh, it was for the shaders, of course.

    The asset makers probably won't listen to some random weirdo asking them to chill out on the keywords, they're not responsible for other devs assets so the solution is what? Keep waiting? Stop using the asset store? Learn all about shaders so I can fix the error myself? They all seem pretty unreasonable when you consider the reasons why people buy assets.

    To ask a perfectly reasonable question: Why can't I use all the assets I bought in my project together? I have been waiting for 2 years to be able to, and this is the #1 issue interfering with what I can do with Unity.

    Infuriating.
     
    naby7u, firstuser and DavidBVal like this.
  44. cloverme

    cloverme

    Joined:
    Apr 6, 2018
    Posts:
    57
    If Unity made games using shaders from the asset store instead of one-off custom shaders made using teams of 10 people for 5 minute tech demos, this would have been fixed by now.
     
  45. Tech-Byte

    Tech-Byte

    Joined:
    Dec 5, 2014
    Posts:
    33
    It's been 6+ years. People have been complaining about this for years. It was a warning for the future that if people with large projects are hitting it, a new rendering pipeline will probably hit it and they were right. These new Pipelines along with basic Unity assets use up close to half your allotted keywords, add some very needed assets such as Gaia and CTS, Vegetation Studio and you've pretty much hit it or already hit the keyword limit.

    They state they can increase the keyword count to 512/1024 but it will cause performance issues and people have been saying, fine, make it optional, let us increase keyword counts, get rid of this memory leak, and just deal with a decrease in performance by working our project around it until a full solution is added.

    But they won't, I'm assuming there is a huge mess of spaghetti code that prohibits them from going from 256 to 512-1024 keywords and causes the client to crash and they've not wanted to touch that bit of Unity since they went from 128 keywords to 256.

    Honest, if I wasn't this invested in Unity, I'd have jumped ship to Unreal long ago.
     
    cloverme likes this.
  46. DavidBVal

    DavidBVal

    Joined:
    Mar 13, 2017
    Posts:
    128
    Pretty much this. After two years and a half of pouring money and time into development I'm not going to switch engines, but that's the only thing keeping me here.
     
    Tech-Byte likes this.
  47. Tech-Byte

    Tech-Byte

    Joined:
    Dec 5, 2014
    Posts:
    33
    I've been using Unity since late 2013, I've invested learning a lot about Unity. I mainly sought Unity due to my original setup not being able to handle Unreal and Unity being free to use. Unity also had a ton of tutorials whereas Unreal did not and still doesn't have many resources.

    I really do hope Unity gets its crap together with HDRP, it looks amazing. Porting my project to HDRP was a game changer in terms of graphics, very much felt like an upgrade from a PS3 game to a PS4 game. The memory leak/keyword issue really needs to get solved, I'll be working my project around it for now in the hopes they fix it.

    I assume since major projects such as Rust uses Unity and is reworking their whole game in HDRP, it'll get fixed eventually.
     
    cloverme likes this.
  48. iMer

    iMer

    Joined:
    May 21, 2013
    Posts:
    15
    The biggest issue, as other posters have mentioned, is that keywords on materials just.. linger for no reason whatsoever
    My current solution is to literally search&replace dead keywords across all files in the project (after searching all files to see if they arent alive still) with an empty string, surely this cant be how unity wants us to do things

    Best part is you can do the whole S***-show again if you need to update any assets since the newest render pipeline update broke compatibility :)
     
    firstuser likes this.
unityunity