Search Unity

  1. Improved Prefab workflow (includes Nested Prefabs!), 2D isometric Tilemap and more! Get the 2018.3 Beta now.
    Dismiss Notice
  2. The Unity Pro & Visual Studio Professional Bundle gives you the tools you need to develop faster & collaborate more efficiently. Learn more.
    Dismiss Notice
  3. Improve your Unity skills with a certified instructor in a private, interactive classroom. Watch the overview now.
    Dismiss Notice
  4. Want to see the most recent patch releases? Take a peek at the patch release page.
    Dismiss Notice

compiling shader variants takes forever when building for Android

Discussion in 'Android' started by strongbox3d, Aug 5, 2018.

  1. strongbox3d

    strongbox3d

    Joined:
    May 8, 2012
    Posts:
    552
    Hello guys,

    I have spent several days googling and reading about this problem and I couldn't find any definitive answer to this problem.

    When I build my project to test it in my Android phone, the compiling shader variants takes a lonnngggg time. Mainly in "standard (specular setup).

    Is anyone else here have had the same problem and if so, has anyone found a clear and real solution for this?

    Regards,
    Carlos
     
    Last edited: Aug 5, 2018
  2. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    318
    How many graphics APIs do you have enabled for the build? Each will generate an additional shader variant.
     
  3. strongbox3d

    strongbox3d

    Joined:
    May 8, 2012
    Posts:
    552
    I used to find this before 2018, now I don't know where to find the settings where I can select the apis I want to use to remove the other ones. So I don't know now if there are some apis enabled that I don't need.

    Regards,
    Carlos
     
  4. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    318
    afaik the place is still the same - player settings
    you can access this either via the build window -> player settings or edit -> settings -> player settings
     
  5. strongbox3d

    strongbox3d

    Joined:
    May 8, 2012
    Posts:
    552
    Got it. I only have Vulkan and openGLES3 enabled.
     
  6. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    318
    How many shader variants does it build?
     
  7. strongbox3d

    strongbox3d

    Joined:
    May 8, 2012
    Posts:
    552
    When I use Standard shader(specular) build thousands of those. I have 13 scenes in my game, and I had to disable the last 7 scenes from the build in order to spend less time since I am only testing, and despite this it took around 40 to 45 minutes compiling shaders. I only use that standard specular shader and a couple of other, which compile very fast and very small amount of variants.
     
  8. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    318
    This just means you have a lot of shader features used in your scenes.
    If this is just for testing, you can remove one of the gfx APIs from the list, this alone should the shader build time in half.
     
  9. strongbox3d

    strongbox3d

    Joined:
    May 8, 2012
    Posts:
    552
    What do you mean with having a lot of shader features? I use the diffuse and normal map slots only. Is there a way to remove features that don't have texture slots from the standard specular shader?
     
  10. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    318
    This is not only about using textures. Do you have fog in any of the scenes? Lightmaps with different parameters? Standard shader has support for many things, and each of those you use in your scenes might double the amount of variants.
    The variants that are not possible are excluded, though.
     
  11. Internetpolice

    Internetpolice

    Joined:
    Oct 16, 2017
    Posts:
    60
    I am actually experiencing this same issue..... I use only standard shaders, no fog, no light maps, one single light-source. Are you speaking about how many actual standard shaders ("variants") we are using?
     
  12. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    318
    Yes. You can right click on a material, select "Select shader", then click on the arrow on the drop-down "compile and show code". It will show how many variants are actually used.
     
  13. Internetpolice

    Internetpolice

    Joined:
    Oct 16, 2017
    Posts:
    60
    368 this is overkill as hell as I have maybe 12 in the scene...... is this really just due to selecting multiple API tiers? >.>
     
  14. strongbox3d

    strongbox3d

    Joined:
    May 8, 2012
    Posts:
    552
    I know for certain that this didn't happened before I started using Unity 2018 and I was using then the same amount of materials and all the other usual stuff, or even more.
     
    M-Hanssen likes this.
  15. strongbox3d

    strongbox3d

    Joined:
    May 8, 2012
    Posts:
    552
    Here I have a couple of screenshots, the first one demostrates that the Standard Specular shader is only being compiled for Android, the seconds shows the usage of the shader in the material. And only one character is using this material

    StandardSpecularShaderDemostration.jpg CharacterMaterial.jpg
     
  16. M-Hanssen

    M-Hanssen

    Joined:
    Sep 8, 2014
    Posts:
    16
    I agree with @strongbox3d Before Unity 2018 the shader variant compilation build time was at least 90% faster then the current build time. This is also becoming a serious issue in our company! For the same project with target windows (mono) i'm compiling shaders for 20 minutes per build, while in previous versions of Unity this was like 1 or 2 minutes!
    @aleksandrk Can you please elaborate ways to decrease this or the reason why Unity does this? It seems like a MAJOR regression in build time!
     
  17. strongbox3d

    strongbox3d

    Joined:
    May 8, 2012
    Posts:
    552
    I wonder why I didn't get any other answer from the Unity folks in relation to this issue. I believe that this issue is "kind" of important and unacceptable when trying to test the game in the target platform.

    Regards,
    Carlos
     
  18. m3rt32

    m3rt32

    Joined:
    Mar 5, 2013
    Posts:
    7
    Indeed, this issue is effecting production workflow so much, in bad way. There suppose to be a solution a selection, because it wasnt like that before 2018.1. I know there are major render pipeline changes in that version but, there suppose to be a way, at least the sake of quick testing and there are situations that you cant use unity remote for testing so an actual build will be necessary in that kind of situations. Dear Unity team, please find a way to shorten or remove that "dead block" of time.
     
  19. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    318
    As new features are added (and there were some new features in 2018.x), some shaders get additional "#pragma multi compile BLA" or "#pragma shader_feature BLA" lines. The first variant always doubles (or more than doubles, depends on the amount of parameters) the amount of shader variants, the second does that only if the actual thing is used.
    Another thing is that it's possible to load scenes additively. This means that if you have a material with a certain set of features in one scene, and you have another scene that adds something that can affect this material (e.g. fog), the material has to support fog, although it's not in the actual scene where it's used. This tends to increase the amount of permutations as well.

    We are well aware of this issue, and add additional parameters only if that's unavoidable.
     
  20. M-Hanssen

    M-Hanssen

    Joined:
    Sep 8, 2014
    Posts:
    16
    I get the explanation why unity does this variant compilation. But still I don't get it why it takes AGES since Unity 2018.x for the same project that took 30 seconds in Unity 2017.x?? To me whatever the explanation is, this should be fixed, because build time is currently unacceptable!
    Might it be part of the solution to actually expose the usage of shader keywords and the impact this has on shader variant compilation to the user? Unity is literally compiling 100.000+ shader variants for just using the "standard shader" In our world! I'm very willing to understand why this happens!?
     
  21. strongbox3d

    strongbox3d

    Joined:
    May 8, 2012
    Posts:
    552
    I didn't and don't load scenes additively, so that is not a reason for my project at all.
     
  22. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    318
    It doesn't matter whether you do or not. It is possible to do that, and there is logic to support that.
    You can also do things from code, like changing the fog type or adding more lights. Since Unity supports that, the Standard shader does as well.
    Just an example: you have a single material in a single scene in a project. This material uses the Standard shader, and just has an albedo texture. You have a couple of models, each using this material. You also have two lights, one is directional and one is point. You don't bake lightmaps. And then there's some fog, and let's say it's linear. You're building for Windows standalone, DX11 only, to simplify this a bit.
    In this case the shader needs to support:
    Base pass:
    * Unlit, lit by directional light, lit by point light, lit by both directional and point light geometry (the last case can be eliminated if we say that only one light per pass is supported); = 3 variants
    * Geometry affected by fog and not affected by it; = 2x3 = 6 variants
    Shadow casting pass:
    * Shadow from directional light, shadow from point light = 2 variants
    Shadow receiving pass:
    * Unlit, shadow from directional light, shadow from point light = 3 variants
    This gives us 6 + 2 + 3 = just 11 variants.

    Now, enable instancing on one of the models.
    Suddenly, everything has to support the case when instancing is on and when instancing is off. 22 variants.

    You can create a new light in C#. Even several. We need an additional pass to handle more than "some fixed amount" lights. That's 12 more variants, 34 total.

    You can create a spotlight using C#. That's one more light type to handle, so everywhere where we had 3 variants, we now have 4. That's 68.

    As new features are added, more and more things have to be supported. The amount of variants grows exponentially. Fortunately, some of those features are mutually exclusive. This allows us to reduce the amount of variants to more sane numbers.

    As I said, we're well aware of this problem, and we're working on making things faster.
     
  23. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    318
    Speaking of a lot of variants... What are your graphics tier settings @strongbox3d?
     
  24. harryvzhvr

    harryvzhvr

    Joined:
    Jul 12, 2018
    Posts:
    7
    Hi all, same happening here... Whats wrong with Unity!? My scenes are clean and I have only one shader in my project file. I have 8 scenes with spherical cubemaps but it exports 8 sharedasset shaders... Then the export ends up at 30GB wth.

    Any luck anybody ?
     
    Last edited: Sep 19, 2018 at 5:23 PM
  25. strongbox3d

    strongbox3d

    Joined:
    May 8, 2012
    Posts:
    552
    My tiers settings are:

    1 and 2: Default.

    The problem became so bad, that I switched the use of standard shader to legacy shaders/diffuse only and even doing this, compiling takes at least 15 minutes.

    3: See screenshot: TierSettings.jpg
     
  26. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    318
    OK, can you submit a bug report so that we can take a look at what's taking so long?
     
  27. Thiago-Crawford

    Thiago-Crawford

    Joined:
    Jan 7, 2013
    Posts:
    92
    Hello, any updates on this?? It is a major issue for me...

    Reproduction Steps:
    Create a simple project with Lightweight RP for VR
    Set the build target to Android (Gradle)
    Open sample scene (one with the DIY equipment)
    Build takes forever....
    THOUSANDS upon THOUSANDS of shader variants being built...

    Unity 2018.2.6f1

    Edit: ok maybe not forever, but much longer than a small scene like this should I think (maybe 30min). On a more complex scene, like those from ArchVizPRO Interior Vol. 3 and it will take forever.
     
    Last edited: Sep 21, 2018 at 12:39 PM
  28. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    318
    This scene uses different shaders. There are many shader features enabled, and it uses postprocessing as well.
    The amount of shader variants compiled does not depend on the scene complexity, only on the features the materials use.
     
  29. Thiago-Crawford

    Thiago-Crawford

    Joined:
    Jan 7, 2013
    Posts:
    92
    I know I said 30min before, because that's what it looked like as the build was finalizing when I made that statement. Turns out it is longer than that, it actually never finished it because I just force quit unity after 1 hour of Unity trying to finalize the build.
    Well my apologies, let me rephrase it...
    On a more complex scene, which in a lot of cases uses more materials and variants, like those from ArchVizPRO Interior Vol. 3 it will take forever.

    Something must be wrong, please find out what it is and fix it.
     
  30. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    318
    Please submit a bug report.