Search Unity

  1. Unity 6 Preview is now available. To find out what's new, have a look at our Unity 6 Preview blog post.
    Dismiss Notice
  2. Unity is excited to announce that we will be collaborating with TheXPlace for a summer game jam from June 13 - June 19. Learn more.
    Dismiss Notice

Compiling Shader Variants Taking Ages

Discussion in 'Shaders' started by Nigey, Apr 20, 2018.

  1. iGoA

    iGoA

    Joined:
    Aug 23, 2020
    Posts:
    22
    Ok, I wasn't aware, that shader coding might be so important. So, in my example I should have a shader code that just does one color (as base) and one normal map and ideally no other features, right?

    EDITED: I don't have experience with shader coding and I'm doing all this solo in my spare time. Would you agree, that this asset could help me out?:
    https://assetstore.unity.com/packages/vfx/shaders/shader-control-74817
     
    Last edited: Nov 12, 2021
  2. zhuchun

    zhuchun

    Joined:
    Aug 11, 2012
    Posts:
    433
    This asset looks good, but generally, it's not a good idea to tweak keywords you don't know, as it's likely to break something. If you're not familiar with code, the best option is to use the latest ShaderGraph. Amplify Shader Editor is my favorite tool as it offers more fine-tuning options, but those do require code experience.

    In your case, if the stripping process is really taking "ages", I mean, any time longer than 10 min, it's most likely a bug, submit them a bug report and cross your fingers. Otherwise, just let it be.
     
    Last edited: Nov 19, 2021
  3. iGoA

    iGoA

    Joined:
    Aug 23, 2020
    Posts:
    22
    Thanks for your advise. Meanwhile, I've been able to reduce the build time to about 5 minutes. I've migrated the project to URP and installed shader control (but didn't tweak much, only keyword I was sure about).
     
  4. ibyte

    ibyte

    Joined:
    Aug 14, 2009
    Posts:
    1,049
    Does anyone know how to enable Shader Collections that work with Unity Cloud build?
     
  5. lyniaspost

    lyniaspost

    Joined:
    Apr 8, 2019
    Posts:
    1
    I found that one of the possible solutions to this is going to Edit > Project Settings > Graphics > and clicking Clear at the bottom of the menu.

    Hope this helps. It's 4 years late, though.
     
    pbrito_unity likes this.
  6. khatristudio798

    khatristudio798

    Joined:
    Feb 26, 2022
    Posts:
    4
    i solved this problem when you Internet On then work properly
     
  7. laf3rs

    laf3rs

    Joined:
    Jan 30, 2014
    Posts:
    5
    I fixed it by removing the Standard shader that I had previously added to the list of shaders in Project Settings>Graphics>Always Included Shaders (dropdown list)
     
    P_Jong likes this.
  8. djarcas

    djarcas

    Joined:
    Nov 15, 2012
    Posts:
    248
    I wish any of the solutions listed in this thread made any difference.

    upload_2022-4-16_20-1-51.png

    301 million? Madness.
     
  9. zhuchun

    zhuchun

    Joined:
    Aug 11, 2012
    Posts:
    433
    301M variants is a LOT, perhaps the most I've ever seen by far. Only 2 possible ways come to my head, first that's a bug, it could lead to anything. Second, your project is generating shaders everywhere, like what Unreal Material Editor do.
     
  10. djarcas

    djarcas

    Joined:
    Nov 15, 2012
    Posts:
    248
    Yup. I've not done anything particularly different or interesting, it's got multi-pass VR (as single-pass URP) is broken and a tiny handful of shadergraphs, but I've got 20-40 minute build times and insane numbers like this!
     
  11. Voronoi

    Voronoi

    Joined:
    Jul 2, 2012
    Posts:
    593
    Same. It did help the build time to uncheck Mesh Optimization and some of the other suggestions, i.e. Strip All and create a shader variant collection. Went from about an hour to more like 10 minutes. This is building for Android using the 2022.1 beta. There was a similar problem when building for iOS, but that seemed to have been solved.

    buildtimes.png
     
    DavidZobrist likes this.
  12. djarcas

    djarcas

    Joined:
    Nov 15, 2012
    Posts:
    248
    That looks like a suspiciously familiar number of shaders.
     
  13. RoyBarina

    RoyBarina

    Joined:
    Jul 3, 2017
    Posts:
    98
    I second that. too familiar
     
  14. unnanego

    unnanego

    Joined:
    May 8, 2018
    Posts:
    203
    I have 440M...
    None of the suggestions from the post are helping. The exact same project compiles in 60 seconds on latest unity 2020...
     
    Last edited: Apr 27, 2022
    DavidZobrist and EvOne like this.
  15. iMer

    iMer

    Joined:
    May 21, 2013
    Posts:
    29
    I'll raise to 1000? :)

    Code (csharp):
    1.  
    2. Compiling shader "Universal Render Pipeline/Lit" pass "ForwardLit" (vp)
    3.    [48.26s] 100M / ~1082M prepared
    4.    [95.61s] 200M / ~1082M prepared
    5.    [141.80s] 300M / ~1082M prepared
    6.    [193.80s] 400M / ~1082M prepared
    7.    [246.51s] 500M / ~1082M prepared
    8.    [354.62s] 600M / ~1082M prepared
    9.    [409.65s] 700M / ~1082M prepared
    10.    [464.94s] 800M / ~1082M prepared
    11.    [516.46s] 900M / ~1082M prepared
    12.    [559.97s] 1000M / ~1082M prepared
    13.      0 / 12681216 variants left after stripping, processed in 628.71 seconds
    14.  
    (server build so all variants are forcibly stripped)

    Edit: found some more screenshots before optimizing with 1300

    CI build:


    Edit 2:
    The only solution is to fork URP and modify shaders to just remove any keywords you dont use, for us that means we just toss out decals, light cookies and the other new features to make it work
     
    Last edited: May 2, 2022
    Magasenakwa and unnanego like this.
  16. djarcas

    djarcas

    Joined:
    Nov 15, 2012
    Posts:
    248
    I wonder if we have anything in common; I'm using Decals, Screen Space Shadows and Screen Space Ambient Occlusion.
     
  17. iMer

    iMer

    Joined:
    May 21, 2013
    Posts:
    29
    We've got fairly extensive graphics settings in the hopes people on toasters can make the game playable for themselves somehow - so we got no lights, only main light, additional lights per vertex/pixel, shadows/no shadows, soft/hard shadows, that kinda thing - also have some scenes with baked/mixed lighting, some with only realtime (no realtime GI so far). So that of course causes more variants that you might have with only one or two light quality settings - but we've been doing this in 2020 without issues

    We are using the screen space ambient occlusion, tried screen space shadows but those were too glitchy (back to non-screen space cascades for the time being)
    Not using decals or light cookies currently (would like to in the future, but not if it means build times take forever or shaders wont build at all since light cookies use additional samplers causing too many to be used)
     
  18. IvyKun

    IvyKun

    Joined:
    Sep 28, 2013
    Posts:
    132
    Unity 2020.3.29, small mobile project where I use just like 5/7 shaders and I got 46k variants and I've been stuck in building the first time for more than 1 hour... this is absurd. Really can't understand this process or how 5/7 shaders can become 46k.
     
  19. Peyot

    Peyot

    Joined:
    Mar 27, 2013
    Posts:
    6
    That's a bug somehow related to the lighting settings. Sometimes I'm getting 1400M (!! for fox sake how's that possible??) variants after changing some settings in the Lighting tab and adding/moving lights.
    I couldn't track what settings exactly yet but will try to investigate. To fix it I reset to the last commit and getting back to my 3 minutes Build and Run.
     
  20. kogi_rc

    kogi_rc

    Joined:
    Apr 23, 2019
    Posts:
    39
    I also have this problem where suddenly in 2021 LTS progress bar shows 125M shader variants and build time increased dramatically beyond reasonable usability.

    URP, Android Vulkan backend. No decals, no screenspace shadows. Mobile SSAO. 5 quality presets. Addressables in project. Few scenes with different lighting settings. It was fine on 2021.1, it is ~10 times slower on 2021 LTS.

    Supposedly there's a fix coming in this area (see: https://forum.unity.com/threads/build-time-taking-almost-a-day-urp.1133605/page-2#post-8107979) but it's hard to tell if it's an acknowledgement of the problem we share because there's a bit of a "you're holding it wrong" vibe in next posts.

    IDK, maybe we are "holding it wrong", hopefully someone figures out what triggers the 10-fold slowdown of 2021 LTS.
     
    Last edited: May 12, 2022
  21. djarcas

    djarcas

    Joined:
    Nov 15, 2012
    Posts:
    248

    I'd be happy if I'm holding it wrong, as long as the tool tells me a) that I'm holding it wrong and b) how to fix it!
     
  22. DriesVrBase

    DriesVrBase

    Joined:
    Mar 24, 2020
    Posts:
    66
    Having the same issue on 2021.3. 390M variants. It's a really simple project with around 3 to 4 indoor scenes. I don't get it. Before the upgrade to 2021.3 the build took around 7-10 minutes max. Currently It takes at least 50 minutes. Up to 1h15m
     
    unnanego likes this.
  23. LucasForcinaSD

    LucasForcinaSD

    Joined:
    Sep 13, 2021
    Posts:
    18
    We're facing the same problem with the LTS in its last patch
    When using 2021.2.10 we used to build our project in like 20 minutes, now it can go up to something between 10 and 18 hours...
    This is making testing things that are build-only really difficult.
    We're trying different solutions from this thread but nothing seems to work so far and we don't have enough free machines to farm builds.
    Hope we get a fix soon enough..
     
  24. kogi_rc

    kogi_rc

    Joined:
    Apr 23, 2019
    Posts:
    39
    In case there are people visiting this thread and not the other one that tracks this issue:

    https://forum.unity.com/threads/build-time-taking-almost-a-day-urp.1133605/page-2#post-8142635

    There is a fix coming and also there's an explanation in next posts how to to check if you are possibly affected by the problem that is getting fixed. If Unity follows regular LTS schedule it should be out by the end of this week or the next one. That's not confirmed release date but a guesstimate based on previous releases.
     
  25. LucasForcinaSD

    LucasForcinaSD

    Joined:
    Sep 13, 2021
    Posts:
    18
    If you're facing an issue about compiling shaders variant in 2021.3 and have things enabled in your Renderer such as URP Decals or SSAO, try disabling them.

    Fact is Decals were not working in 2021.2.10f, so we removed them at that time and that's why we were building fast. Since we upgraded to LTS we also re-enabled Decals, and disabling them seems to save us a lot of build time.
    URP features seems to add so much keywords to the shaders compilation we're reaching unbelievable variants count.
    Hoping it helps and that Unity can address this in the future.

    Linking this thread that talks about shader stripping improvments for URP features since 2021.2 for anyone curious:
    https://forum.unity.com/threads/shader-stripping-improvements-in-urp.1193215/

    I wish we could have control over which keyword is enabled on the URP features. We're using Shader Control for our project but it doesn't let you modify everything.
     
  26. XR-Phil

    XR-Phil

    Joined:
    Jun 13, 2019
    Posts:
    19
    Have the same issue. Working on 2021.3.1f1 HDRP. I have 3 Scenes, used Gaia and Nature Manifacture Assets for 2 Mid Sized Terrain Scenes...Project File is about 90GB, Building takes Hours. Compiling Shaders x of 198.452 . I tried the options above, still not ready with my 1st Built. After 1h and 17min it compiled ~ 20.000 of those 198.000 so round about 10% :D Is that normal? I already tried the tips above but none of them really helped.
     
    unnanego likes this.
  27. dnach

    dnach

    Unity Technologies

    Joined:
    Mar 9, 2022
    Posts:
    90
    As some folk mentioned in this thread, it is advised to disable any rendering features that are not needed in the URP/HDRP assets used in the build's quality settings, and ensure you enable the shader stripping settings in the URP/HDRP global settings.

    That being said, we understand that the shader build times are a massive pain - and are working on some immediate solutions to alleviate this. The most important one being an optimization to Unity's automatic shader variant stripping, to greatly reduce shader build times via variant keyword prefiltering.

    Check the official forum post more detail:
    https://forum.unity.com/threads/improvements-to-shader-build-time-and-runtime-memory-usage.1305615/
    This post also outlines some best practices and other useful information regarding shader variants logging and stripping.

    Thank you all for your feedback, and please keep it coming!
     
    Last edited: Jul 11, 2022
    DavidZobrist likes this.
  28. unnanego

    unnanego

    Joined:
    May 8, 2018
    Posts:
    203
    But the problem doesn't exist in unity 2020, the build that takes 20 minutes now took 1 minute in 2020
     
    DavidZobrist likes this.
  29. dnach

    dnach

    Unity Technologies

    Joined:
    Mar 9, 2022
    Posts:
    90
    The problem is a lot more prevalent for 2021 and later versions, as the URP shaders introduced more multi_compile keywords and thus generate more variants.
     
  30. matheus_inmotionvr

    matheus_inmotionvr

    Joined:
    Oct 3, 2018
    Posts:
    63
    I just upgraded from Unity 2019.4 to 2021.3 and the first build took forever, which is understandable. But then subsequent builds are also taking forever, and most of the time the build progress bar displays "Preparing variants for compilation (XM / 40M)".

    Shouldn't these shaders be cached somewhere, and not get compiled every time I build the project?
     
    DavidZobrist and unnanego like this.
  31. kogi_rc

    kogi_rc

    Joined:
    Apr 23, 2019
    Posts:
    39
    It's been like that for a few months now since the LTS released. If your URP build is taking 10 times longer, it's expected.

    You can follow these topics, too:
    https://forum.unity.com/threads/build-time-taking-almost-a-day-urp.1133605/
    https://issuetracker.unity3d.com/issues/shader-variant-build-preparation-does-not-scale
    https://forum.unity.com/threads/improvements-to-shader-build-time-and-runtime-memory-usage.1305615/

    Improvements are supposed to be coming in version "23.1" with backports to earlier versions but no amount of promises will restore the hours losts over the last few months, so I recommend downgrading back to your stable Unity version before you commit into this LTS with changes that are harder to revert.

    I don't mean to throw shade at Unity team here because they are apparently dealing with quite a lot lately so I understand that sometimes improvements in one engine part may be detrimental in some other engine module but yeah... if you value your time, don't use this LTS.
     
  32. matheus_inmotionvr

    matheus_inmotionvr

    Joined:
    Oct 3, 2018
    Posts:
    63
    That's a shame. I hope they are able to fix this as soon as possible.

    In my case, disabling "Optimize mesh data" did the trick and builds that were taking between 30 and 40 minutes now take less than 4. But still, I'd rather have this fixed. Let's hope.
     
    DavidZobrist likes this.
  33. unnanego

    unnanego

    Joined:
    May 8, 2018
    Posts:
    203
    you're lucky, nothing helps my case(
     
  34. DavidZobrist

    DavidZobrist

    Joined:
    Sep 3, 2017
    Posts:
    234
    Great to have found this thread and not beeing alone on that one.
    Updated from LTS 2019 ( URP old version) to 2021.3.11f1 with the official URP included package.
    Reached only 140 million shader variants. (increased the buildtime by 3x -4x)
    I assumed not all old urp mats properly updating via the new converter (it also times out on trieng to update read only or/and the global effects sections)

    Will try to disabling "Optimize mesh data".
    Graphics Shader Strip All
    And create a shader variant collection

    tomorrow. But this was an exhausting day dealing with that.

    @dnach
    Any official updates on fixing that from Unitys side?

    Regarding
    "That being said, we understand that the shader build times are a massive pain - and are working on some immediate solutions to alleviate this."


    Or are the tips above the solution for us moving forward?
     
  35. DavidZobrist

    DavidZobrist

    Joined:
    Sep 3, 2017
    Posts:
    234
  36. flintcheeze

    flintcheeze

    Joined:
    Sep 10, 2015
    Posts:
    80
    Fast forward to Nov 18 I am still facing the issue for a 125mb size project lol spent 1hr 19mins already and still counting will try out the above solution cause what the elll
    Unity 2022.1.23f1
     
    ynaoto likes this.
  37. Meceka

    Meceka

    Joined:
    Dec 23, 2013
    Posts:
    423
    Have any of you tried disabling "Asynchronous Shader Compilation" available under Project Settings window > Editor?
     
  38. flintcheeze

    flintcheeze

    Joined:
    Sep 10, 2015
    Posts:
    80
    I am about to try that now
     
  39. Magasenakwa

    Magasenakwa

    Joined:
    Oct 13, 2018
    Posts:
    91
    Just tried it. Still takes me only 35 seconds to compile 174M shaders. On a whole I build very fast compared to you guys but I would still like to know WHY I have 173M + 46M shaders and how to get that down to a more sensible number like the 10 shaders we ACTUALLY use.

    98% of the project is UGUI. The rest is a single scene with a character standing inside an inverted cube, viewed in VR. 200+ million shaders? I think not :(
     
    EvOne likes this.
  40. djarcas

    djarcas

    Joined:
    Nov 15, 2012
    Posts:
    248
    Why would we want to do that? Surely that'll just make it happen in series, ie take even longer?
     
  41. Meceka

    Meceka

    Joined:
    Dec 23, 2013
    Posts:
    423
    Yes in theory it could take longer. I have no idea of the reason behind it but that was increasing my shader compilation time by perhaps 1000 fold so I disabled it long time ago.

    For me not it didn't affect the build time though. Just the shader compilation time after modifying a shader in the project. It got reduced from minutes to a second when I disabled it.

    I just thought I would share it as I believe it's a relevant toggle. People here are sounding desperate to try anything.
     
    Munchy2007 likes this.
  42. bilalakil

    bilalakil

    Joined:
    Jan 28, 2018
    Posts:
    78
    The problem popped up from me after upgrading from Unity 2021.3 to Unity 2022.2. Builds went from (quite) a few minutes, to a few hours, and my CI started failing due to 6 hour timeouts Some of the techniques listed in here may have benefits, but we're using a smorgasbord of assets / plugins so I'm worried about side-effects.
     
  43. ph0b0s

    ph0b0s

    Joined:
    Dec 18, 2013
    Posts:
    19
    After months of hard work we convinced our bosses to use CI/CD in GitHub. This weeks previous Christmas all people saw the benefits from this but last week we hit the 6h build time barrier that makes the system totally unusable and force us to make the builds manually in our machines. I don't know what's really going on but this is a big issue for companies like mine and probably makes us change the engine in the future, we don't have the people and infrastructure to manually check for every shader variant to make our application works with the short schedules we have to accomplish. I love Unity and always was a huge defender of what they make possible for the community of developers but sometimes having to deal with this kind of limitations makes the engine almost unusable in a professional environment.
     
    andreiagmu and Meceka like this.
  44. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,046
    @bilalakil @ph0b0s please double-check that you don't have unnecessary quality settings differences in the render pipeline assets as described here under "Build-time shader variant stripping".
     
  45. FredTuna

    FredTuna

    Joined:
    Dec 28, 2016
    Posts:
    16
    We went crazy trying to figure out how to reduce shader variants in our HDRP . In our case, our HDRenderPipelineAsset all had
    allowShaderVariantStripping
    set to false!! The default value is true but back in 2018, shader variant stripping was an experimental feature that was exposed in the HDRenderPipelineAsset assets. We set it to false then since it was experimental and never thought more of it. Later in 2018, it was made no longer experimental and at the same time, it was no longer exposed in those assets but remained serialized in the asset files and the
    allowShaderVariantStripping
    field is still used to determine whether stripping should happen in HDRP code. We modified that bool to true in the HDRenderPipelineAsset files manually and now shader passes that had 10k+ variants have 44.

    Since going to Unity 2021 and then 2022, our cloud builds could not finish because it would time out after 24 hours. One shader pass compilation of 5k+ variants took 8 hours and there were many many passes like this, some worse. In our previous project on Unity 2019.3, it was long but at least our cloud builds could finish so we figured it was fine. Later Unity versions made the problem so bad that we had to debug and break into HDRP code until we saw that
    allowShaderVariantStripping
    field skipping all possible variant stripping.
     
    kmowers likes this.
  46. NoctisFatehart

    NoctisFatehart

    Joined:
    Jul 30, 2014
    Posts:
    28
    Boy do I wish I knew what exactly was causing this...




    It's almost comical how many variants there are that are causing this to take so much longer to build.
     
  47. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,046
    @NoctisFatehart I recently added shader name to this dialog to help identify problematic ones.
     
    futurlab_peterh, RoyBarina and Meceka like this.
  48. DavidZobrist

    DavidZobrist

    Joined:
    Sep 3, 2017
    Posts:
    234
    ehhhhh.. All was fixed for me in 2021 LTS just switched to 2022 LTS
    I wait since 4 hours for the build..

    150 k variants ..
     
    Last edited: Jun 2, 2023
  49. bigbrainz

    bigbrainz

    Joined:
    Jul 21, 2015
    Posts:
    177
    @DavidZobrist, I know when we tried switching to 2022, we found that there's a new functionality in the renderer. As I understand it, to get comparable build times to 2021, you will have to go in and choose the new feature in the Build Settings called something like "Faster Build Times". This was for WebGL.

    We switched back to 2021 because of some show-stopping bugs, so I don't have the screenshot anymore. And maybe you're also seeing other things, but this could be part of the issue.

    So in 2022, under Build Settings, you will see a 3rd option that's something like "Faster Build Times". Apparently the "new" "normal" build does some crazy optimizations that made our build times about 6x slower. And they didn't improve our performance in any measurable way. So "Faster Build Times" appears to be the equivalent of what "Faster runtime" was back in 2021. Maybe it's just a bug, but this was at least a reasonable workaround for us.

    upload_2023-6-2_11-0-53.png
     
    EvOne and DavidZobrist like this.
  50. DavidZobrist

    DavidZobrist

    Joined:
    Sep 3, 2017
    Posts:
    234

    Thanks, I give it a try when this one is finished x6 times slower would mean my build will be finished in 1 hour max :D

    The big junk so far is the shader variants.. ( 5 hours so far)


    Update: the option war correctly selected.
    Still its a 5-6 hours to build.

    Also the finished / completed builds dont start up anymore on the device.
    Thinking about going back to 2021 LTS

    Update2:

    After 2 days. We decided to go back to 2021 LTS.
    All works fine again..
     
    Last edited: Jun 5, 2023
    EvOne likes this.