Search Unity

compiling shader variants takes forever when building for Android

Discussion in 'Android' started by strongbox3d, Aug 5, 2018.

  1. strongbox3d

    strongbox3d

    Joined:
    May 8, 2012
    Posts:
    860
    Hello guys,

    I have spent several days googling and reading about this problem and I couldn't find any definitive answer to this problem.

    When I build my project to test it in my Android phone, the compiling shader variants takes a lonnngggg time. Mainly in "standard (specular setup).

    Is anyone else here have had the same problem and if so, has anyone found a clear and real solution for this?

    Regards,
    Carlos
     
    Last edited: Aug 5, 2018
    tigerleapgorge likes this.
  2. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,025
    How many graphics APIs do you have enabled for the build? Each will generate an additional shader variant.
     
    tigerleapgorge likes this.
  3. strongbox3d

    strongbox3d

    Joined:
    May 8, 2012
    Posts:
    860
    I used to find this before 2018, now I don't know where to find the settings where I can select the apis I want to use to remove the other ones. So I don't know now if there are some apis enabled that I don't need.

    Regards,
    Carlos
     
    tigerleapgorge likes this.
  4. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,025
    afaik the place is still the same - player settings
    you can access this either via the build window -> player settings or edit -> settings -> player settings
     
    tigerleapgorge likes this.
  5. strongbox3d

    strongbox3d

    Joined:
    May 8, 2012
    Posts:
    860
    Got it. I only have Vulkan and openGLES3 enabled.
     
    tigerleapgorge likes this.
  6. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,025
    How many shader variants does it build?
     
    tigerleapgorge likes this.
  7. strongbox3d

    strongbox3d

    Joined:
    May 8, 2012
    Posts:
    860
    When I use Standard shader(specular) build thousands of those. I have 13 scenes in my game, and I had to disable the last 7 scenes from the build in order to spend less time since I am only testing, and despite this it took around 40 to 45 minutes compiling shaders. I only use that standard specular shader and a couple of other, which compile very fast and very small amount of variants.
     
    tigerleapgorge likes this.
  8. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,025
    This just means you have a lot of shader features used in your scenes.
    If this is just for testing, you can remove one of the gfx APIs from the list, this alone should the shader build time in half.
     
    tigerleapgorge likes this.
  9. strongbox3d

    strongbox3d

    Joined:
    May 8, 2012
    Posts:
    860
    What do you mean with having a lot of shader features? I use the diffuse and normal map slots only. Is there a way to remove features that don't have texture slots from the standard specular shader?
     
    tigerleapgorge likes this.
  10. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,025
    This is not only about using textures. Do you have fog in any of the scenes? Lightmaps with different parameters? Standard shader has support for many things, and each of those you use in your scenes might double the amount of variants.
    The variants that are not possible are excluded, though.
     
    cxode and tigerleapgorge like this.
  11. Internetpolice

    Internetpolice

    Joined:
    Oct 16, 2017
    Posts:
    60
    I am actually experiencing this same issue..... I use only standard shaders, no fog, no light maps, one single light-source. Are you speaking about how many actual standard shaders ("variants") we are using?
     
  12. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,025
    Yes. You can right click on a material, select "Select shader", then click on the arrow on the drop-down "compile and show code". It will show how many variants are actually used.
     
    EirikWahl likes this.
  13. Internetpolice

    Internetpolice

    Joined:
    Oct 16, 2017
    Posts:
    60
    368 this is overkill as hell as I have maybe 12 in the scene...... is this really just due to selecting multiple API tiers? >.>
     
  14. strongbox3d

    strongbox3d

    Joined:
    May 8, 2012
    Posts:
    860
    I know for certain that this didn't happened before I started using Unity 2018 and I was using then the same amount of materials and all the other usual stuff, or even more.
     
    mgsvevo and M-Hanssen like this.
  15. strongbox3d

    strongbox3d

    Joined:
    May 8, 2012
    Posts:
    860
    Here I have a couple of screenshots, the first one demostrates that the Standard Specular shader is only being compiled for Android, the seconds shows the usage of the shader in the material. And only one character is using this material

    StandardSpecularShaderDemostration.jpg CharacterMaterial.jpg
     
    tigerleapgorge likes this.
  16. M-Hanssen

    M-Hanssen

    Joined:
    Sep 8, 2014
    Posts:
    34
    I agree with @strongbox3d Before Unity 2018 the shader variant compilation build time was at least 90% faster then the current build time. This is also becoming a serious issue in our company! For the same project with target windows (mono) i'm compiling shaders for 20 minutes per build, while in previous versions of Unity this was like 1 or 2 minutes!
    @aleksandrk Can you please elaborate ways to decrease this or the reason why Unity does this? It seems like a MAJOR regression in build time!
     
  17. strongbox3d

    strongbox3d

    Joined:
    May 8, 2012
    Posts:
    860
    I wonder why I didn't get any other answer from the Unity folks in relation to this issue. I believe that this issue is "kind" of important and unacceptable when trying to test the game in the target platform.

    Regards,
    Carlos
     
    tigerleapgorge likes this.
  18. merpheus

    merpheus

    Joined:
    Mar 5, 2013
    Posts:
    202
    Indeed, this issue is effecting production workflow so much, in bad way. There suppose to be a solution a selection, because it wasnt like that before 2018.1. I know there are major render pipeline changes in that version but, there suppose to be a way, at least the sake of quick testing and there are situations that you cant use unity remote for testing so an actual build will be necessary in that kind of situations. Dear Unity team, please find a way to shorten or remove that "dead block" of time.
     
    tigerleapgorge likes this.
  19. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,025
    As new features are added (and there were some new features in 2018.x), some shaders get additional "#pragma multi compile BLA" or "#pragma shader_feature BLA" lines. The first variant always doubles (or more than doubles, depends on the amount of parameters) the amount of shader variants, the second does that only if the actual thing is used.
    Another thing is that it's possible to load scenes additively. This means that if you have a material with a certain set of features in one scene, and you have another scene that adds something that can affect this material (e.g. fog), the material has to support fog, although it's not in the actual scene where it's used. This tends to increase the amount of permutations as well.

    We are well aware of this issue, and add additional parameters only if that's unavoidable.
     
  20. M-Hanssen

    M-Hanssen

    Joined:
    Sep 8, 2014
    Posts:
    34
    I get the explanation why unity does this variant compilation. But still I don't get it why it takes AGES since Unity 2018.x for the same project that took 30 seconds in Unity 2017.x?? To me whatever the explanation is, this should be fixed, because build time is currently unacceptable!
    Might it be part of the solution to actually expose the usage of shader keywords and the impact this has on shader variant compilation to the user? Unity is literally compiling 100.000+ shader variants for just using the "standard shader" In our world! I'm very willing to understand why this happens!?
     
    tigerleapgorge likes this.
  21. strongbox3d

    strongbox3d

    Joined:
    May 8, 2012
    Posts:
    860
    I didn't and don't load scenes additively, so that is not a reason for my project at all.
     
    tigerleapgorge likes this.
  22. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,025
    It doesn't matter whether you do or not. It is possible to do that, and there is logic to support that.
    You can also do things from code, like changing the fog type or adding more lights. Since Unity supports that, the Standard shader does as well.
    Just an example: you have a single material in a single scene in a project. This material uses the Standard shader, and just has an albedo texture. You have a couple of models, each using this material. You also have two lights, one is directional and one is point. You don't bake lightmaps. And then there's some fog, and let's say it's linear. You're building for Windows standalone, DX11 only, to simplify this a bit.
    In this case the shader needs to support:
    Base pass:
    * Unlit, lit by directional light, lit by point light, lit by both directional and point light geometry (the last case can be eliminated if we say that only one light per pass is supported); = 3 variants
    * Geometry affected by fog and not affected by it; = 2x3 = 6 variants
    Shadow casting pass:
    * Shadow from directional light, shadow from point light = 2 variants
    Shadow receiving pass:
    * Unlit, shadow from directional light, shadow from point light = 3 variants
    This gives us 6 + 2 + 3 = just 11 variants.

    Now, enable instancing on one of the models.
    Suddenly, everything has to support the case when instancing is on and when instancing is off. 22 variants.

    You can create a new light in C#. Even several. We need an additional pass to handle more than "some fixed amount" lights. That's 12 more variants, 34 total.

    You can create a spotlight using C#. That's one more light type to handle, so everywhere where we had 3 variants, we now have 4. That's 68.

    As new features are added, more and more things have to be supported. The amount of variants grows exponentially. Fortunately, some of those features are mutually exclusive. This allows us to reduce the amount of variants to more sane numbers.

    As I said, we're well aware of this problem, and we're working on making things faster.
     
  23. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,025
    Speaking of a lot of variants... What are your graphics tier settings @strongbox3d?
     
  24. harryvzhvr

    harryvzhvr

    Joined:
    Jul 12, 2018
    Posts:
    9
    Hi all, same happening here... Whats wrong with Unity!? My scenes are clean and I have only one shader in my project file. I have 8 scenes with spherical cubemaps but it exports 8 sharedasset shaders... Then the export ends up at 30GB wth.

    Any luck anybody ?
     
    Last edited: Sep 19, 2018
  25. strongbox3d

    strongbox3d

    Joined:
    May 8, 2012
    Posts:
    860
    My tiers settings are:

    1 and 2: Default.

    The problem became so bad, that I switched the use of standard shader to legacy shaders/diffuse only and even doing this, compiling takes at least 15 minutes.

    3: See screenshot: TierSettings.jpg
     
    tigerleapgorge and JasonDaze like this.
  26. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,025
    OK, can you submit a bug report so that we can take a look at what's taking so long?
     
    tigerleapgorge likes this.
  27. Thiago-Crawford

    Thiago-Crawford

    Joined:
    Jan 7, 2013
    Posts:
    92
    Hello, any updates on this?? It is a major issue for me...

    Reproduction Steps:
    Create a simple project with Lightweight RP for VR
    Set the build target to Android (Gradle)
    Open sample scene (one with the DIY equipment)
    Build takes forever....
    THOUSANDS upon THOUSANDS of shader variants being built...

    Unity 2018.2.6f1

    Edit: ok maybe not forever, but much longer than a small scene like this should I think (maybe 30min). On a more complex scene, like those from ArchVizPRO Interior Vol. 3 and it will take forever.
     
    Last edited: Sep 21, 2018
  28. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,025
    This scene uses different shaders. There are many shader features enabled, and it uses postprocessing as well.
    The amount of shader variants compiled does not depend on the scene complexity, only on the features the materials use.
     
  29. Thiago-Crawford

    Thiago-Crawford

    Joined:
    Jan 7, 2013
    Posts:
    92
    I know I said 30min before, because that's what it looked like as the build was finalizing when I made that statement. Turns out it is longer than that, it actually never finished it because I just force quit unity after 1 hour of Unity trying to finalize the build.
    Well my apologies, let me rephrase it...
    On a more complex scene, which in a lot of cases uses more materials and variants, like those from ArchVizPRO Interior Vol. 3 it will take forever.

    Something must be wrong, please find out what it is and fix it.
     
  30. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,025
    Please submit a bug report.
     
  31. M-Hanssen

    M-Hanssen

    Joined:
    Sep 8, 2014
    Posts:
    34
    Can anyone please post the bug report here, this is one I want to keep my eyes on!
     
    ojelibalon likes this.
  32. nachofrades

    nachofrades

    Joined:
    Dec 11, 2013
    Posts:
    2
    I have reduced the compile time a lot with low quality shadows. But it must be fixed, for 4 materials compile tens of thousands of shader variants...
     
  33. nachofrades

    nachofrades

    Joined:
    Dec 11, 2013
    Posts:
    2
    Updated! After tons of shaders variants, the build do not have textures... bravo! :eek:
     
  34. MB6

    MB6

    Joined:
    Oct 31, 2018
    Posts:
    1
    Any solution for that? I've also encountered this issue and there is like thousands of shader variants (also mainly standard (specular setup)) during build.
     
  35. MadeFromPolygons

    MadeFromPolygons

    Joined:
    Oct 5, 2013
    Posts:
    3,982
    how is this still not fixed yet?!
     
  36. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,025
    So, did anyone actually report a bug? If you did, please post the bug number here.
     
  37. Deleted User

    Deleted User

    Guest

    To report a bug you need the editor, and you know what? The editor has been hooked up by the shader compiler variants for the past 1h and a half. Now I am reporting a bug for this issue, even we all know this is not a bug strictly speaking. In the mean time, since it is taking its time to build the package for the report let me make a simple consideration.

    This has been reported here since August 5th, it has to count somehow. Otherwise, if every time we come here to seek for help what we get is "report a bug", close the support forum and put a big text on the first page:

    "Report a bug, because on our computers it works!"

    So, instead of procrastinating, avoiding to tackle this issue, why don't you open any recent 2018.1.2+ Unity, create a new project with Lightweight Pipeline VR, for Android, Report a bug yourself at this point, and then BUILD?.

    Create a simple project with Lightweight RP for VR
    Set the build target to Android (Gradle)
    Open the example scene (one with the DIY equipment)
    Build takes forever....
    THOUSANDS upon THOUSANDS of shader variants being built...
    Unity 2018.2.6f1


    We are not talking about big projects, we are talking about the first example scene to test the LWRP for VR ¬_¬.

    Oh, since we are here? Even you can't reduce the number of variants the first time you build, can't you, at least, figure out what actually has changed since last time and filter out those variants that don't need to be recreated?

    Thanks,
    Fabio
     
  38. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,025
    @p2s-fabio You have some wrong assumptions in your reply.
    First, I'm frequently building (a fresh build) of LWRP template (but not the VR one), and it doesn't take that long for me. I didn't measure this exactly, but it's probably around 10 minutes between hitting "build" and seeing it on the device, running. Since I don't see this problem, I ask to make a bug report, because your environment is likely to be different from mine. Also, this is about the VR LWRP template, and it's likely that it's going to be picked up be the people who are actually working with VR, and know, what's happening there and what's normal or not. I'm not sure whether they are checking this part of the forum, which brings me to the second point:
    This is a community forum. This is not a support forum. Unity employees in general are not required to participate in this in any way.

    Unity recompiles only the shader variants that changed (e.g. if you change a single shader source file, it will recompile just this shader. If you change an include, it will only recompile the shaders that used this include). If there have been no changes, it should use the variant from the cache in Library folder. If you see different behaviour, you should report this as well.
     
    anattress likes this.
  39. Deleted User

    Deleted User

    Guest

    @aleksandrk Well, at least one bug has been reported now by me. As far as I know, it happens by simply start a new project as described above (LWRP VR Android), that's why reporting a bug sounded overkilling, but it was not reproducible on your side, which apparently is the case. Also, I didn't know Unity employees were not ment to participate in the conversation.

    Anyway, here is the bug report https://fogbugz.unity3d.com/default.asp?1099293_2p70l1noer6l9ini
     
    tigerleapgorge likes this.
  40. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,025
    Thank you!
    The process of reporting a bug exists for very simple reasons. For example, it's easier to track, what's happening, not duplicate work and get the issues solved faster (they get assigned to the people that work in an area specific to the bug).

    This is not what I said. It's "not required to", not "not meant to".
     
  41. demonixis

    demonixis

    Joined:
    Aug 20, 2013
    Posts:
    185
    Hey same issue here, I've opened the issue 1090892.
     
  42. tlovidiu

    tlovidiu

    Joined:
    Nov 13, 2012
    Posts:
    27
    Same issue here... I hope someone comes with a solution.
     
  43. demonixis

    demonixis

    Joined:
    Aug 20, 2013
    Posts:
    185
    Hey, I switched back to the legacy forward renderer and compilation time as decreased a lot. It's not exactly what I would like but I can work again on my project so it's fine for now.
     
  44. ASIM-SENYUVA

    ASIM-SENYUVA

    Joined:
    Apr 29, 2013
    Posts:
    90
    We upgrade our code from 2017 to 2018. compiling shader variants takes too much time during the build and we can not test our built version. we have 52 scenarios into code. I can not build all of them but I try to build twscenariosio. they take around half an hour. we can not continue like this. this is a very big bug. Unity team should be found solution or workaround this.

    upload_2018-11-29_2-21-18.png
     
    JasonDaze likes this.
  45. FloBeber

    FloBeber

    Joined:
    Jun 9, 2015
    Posts:
    166
    Last edited: Dec 3, 2018
  46. John-Lisenby

    John-Lisenby

    Joined:
    Nov 8, 2013
    Posts:
    122
    This hasn't been fixed yet? It's taking me 6+ hours to compile my game? This makes no sense that when I make one code change, and I have to wait a day to build and test it.

    I upgraded my project from 2017 to 2018 what a mistake. At this rate it will take forever to finish my game! :-(

    John
     
    tigerleapgorge likes this.
  47. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,025
    If you change a shader which has a lot of variants used in your game, it has to recompile them to be able to use them.

    It might be a good idea to make a small test scene and do your iteration on it.
     
  48. John-Lisenby

    John-Lisenby

    Joined:
    Nov 8, 2013
    Posts:
    122
    Hi,

    When I do a code change (script) not shader script. It seems that the it just recompiles the shaders every time I rebuild my game. I haven't changed or added any new shaders in months. However, I did check the ShaderCache folder in my library folder of my project, and It looks like it just keeps rebuilding each shader each time (date) I rebuild my game. I deleted the folder it was over 10gb in size. I'm not sure if this normal.

    Thanks,

    John
     
  49. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,025
    Hi!

    ShaderCache contains all variants that have ever been built, so it's fine if it's 10Gb in size.

    It shouldn't rebuild the variants it has in the cache, but reading each of them from the cache takes time as well. If you have a lot of variants or a very slow drive, this may take long and seem like it's compiling the shaders. You can check in the shader compiler log files in your project's Library folder if it's recompiling any variants. If you see that it's actually recompiling the shaders even if you didn't change them, please submit a bug report.

    We're working on reducing iteration time. Unity 2019.1, for example, has an option to patch the script changes to an app that already exists on the device without updating anything else.
     
  50. FloBeber

    FloBeber

    Joined:
    Jun 9, 2015
    Posts:
    166
    I'm currently working on a project with Unity 2018.3 and LWRP 4.6.0 (targeting mobile VR) and I wanted to share my experience.
    • The very first time I build (or when I clear the Library folder), building takes a long time (~45 mins).
    • Modifying any LWRP setting would recompile all Shaders (~45 mins)
    • Modifying any Shader takes less time (as far as I understand, because only the modified Shaders get recompiled)
    • Modifying scripts does not increase build time (@John-Lisenby)
    • But building without changing anything still takes MUCH longer than using Legacy. In my opinion, that is a major drawback using LWRP. I'd be curious to know why that happens and if you guys are planning to improve this in 2019.x

    Your feedback is much appreciated @aleksandrk :)
     
    JasonDaze and M-Hanssen like this.