Search Unity

Why does Unity build Shader Variants on each build?

Discussion in 'Shaders' started by tinto22, May 1, 2019.

  1. tinto22

    tinto22

    Joined:
    Mar 28, 2017
    Posts:
    6
    I have a few different questions here. Compiling shader variants has practically ground progress on our project to a halt. I've added a script via the Shader Variant API (not sure of official name - but the IPreprocessShaders interface), to inspect what variants it is building. Through this, it has become apparent that it is building over 1 million variants. Sometimes, this takes hours. Strangely, sometimes it takes about 20-30 minutes. Either way, it's killing us on this project.

    I've investigated in more depth and have some findings:

    a) There are many many instances of the same shader+keyword configuration - sometimes over 900. I printed out all values I could and everything matches on each of those unique instances.
    b) In a sample project, I was able to reproduce this by simply adding a new material referencing the Standard shader - as long as the material was being used in a scene. With 1 material, I got 2 instances of each unique configuration. With 2 materials, I got 3 instances, and this then grew linearly as more materials were added.
    c) I tried using the shader variant API to ignore all unique variants after the first one it saw. This didn't work - faster builds, but everything in-game was magenta.

    I have 3 main questions:

    1) Why does Unity increase the number of variants linearly with each material use? Surely it only needs one copy of each possible configuration of the shader+keywords at runtime, so why is it building many many times more than it needs to?
    2) Why does Unity build shader variants EACH time the game is built. I'm building for PS4 (in this example), and each time I build the window pops up saying it's building shader variants. Nothing changed since last time I built so why isn't this work cached and reused for subsequent builds on the same platform?
    3) What, if anything, can we do to fix this so we're no longer basically blocked on this project. Unity was meant to save us money by improving our iteration times but at this point it has ground us to a halt.

    P.S. I have read through this thread already (https://forum.unity.com/threads/compiling-shader-variants-taking-ages.527724/) and tried everything suggested. Nothing works and there's no response from Unity in the thread.

    Also, I am running 2018.3.13f1.
     
  2. bgolus

    bgolus

    Joined:
    Dec 7, 2012
    Posts:
    7,005
    Are you using a shader variant collection asset? If not, Unity will iterate over every material in the project and ask to compile every single variant. My understanding is it is not actually recompiling them every time, but checking the disk first and skipping if found. However on modern PCs it probably takes longer to access the disk than to just compile the variant and re-save it every time, especially if you're not running your project off of a fast SSD.

    I do believe this is an area of active work over at Unity. They've been seeing issues with shader compile times for LWRP and HDRP builds, in large part because they've internally finally moved to having internal groups building larger projects that better reflect real world use cases than the hyper focused demos of the past.
     
  3. tinto22

    tinto22

    Joined:
    Mar 28, 2017
    Posts:
    6
    I have tried using a shader variant collection asset - unfortunately it made no difference whatsoever...
     
  4. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    862
    Each time a material is encountered, we need to make sure the shader variant is actually compiled. What you see in the dialog that says "I'm building shader variants" is not necessarily actually building shader variants. If a variant has been compiled already, it's getting it from the shader cache in the library folder.
    I suppose what's happening when you do your own IPreprocessShader and remove variants that you think are identical, it will think that it doesn't need to reference this variant, and doesn't tell the material, where to look for its shader (or something similar).

    When you see a duplicate, it doesn't mean it gets recompiled.

    When your build takes hours instead of 20-30 minutes, this means that something triggered shader recompilation, for all shaders or just some shaders. This can happen if you update Unity (but it's not always the case), if you modify an include file, or modify shader source.
     
  5. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    862
    This depends on too much factors. Rule of thumb is that it's still faster to read from the disk cache.
     
  6. tinto22

    tinto22

    Joined:
    Mar 28, 2017
    Posts:
    6
    Ok, good to know that it's not actually compiling them - but it's mostly a moot point as it's still taking an extremely long time, regardless - so I'd still like advice on how we can improve it.

    Furthermore, the numbers we are seeing just seem pretty insane. I'll give an example - prior to doing some pretty aggressive stripping in my IPreprocessShader, the profiler was showing memory usage on PS4 under the "Unity" category as 1.4GB. After some aggressive stripping, the profiler now shows 0.6GB under the "Unity" category. This implies, of course, that 800MB of memory was being used up by shaders, or some other bookkeeping that is tied to shaders. This is pretty crazy, especially given that it's the "default" behaviour and I've had to write this IPreprocessShader to get it back under control.

    Does this seem out of whack to you, @aleksandrk? Does it sound like a bug, or is this what people should be expecting to see? For the record, this is a VR project with Single Pass Stereo on - so I understand that increases the number of shader variants... but... 800MB??

    Thanks for any help you can offer.
     
  7. asperatology

    asperatology

    Joined:
    Mar 10, 2015
    Posts:
    966
    Maybe we have to create a ShaderVariantSettings.asset file, that contains the properties to skip all shaders? (I don't know. All I saw was an option to skip all shaders.)
     
  8. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    862
    @tinto22 I'm not sure about PS4 :)
    I'll let the team that works on it know about this thread.

    Many shaders come with this line:
    Code (CSharp):
    1. #pragma multi_compile _ UNITY_SINGLE_PASS_STEREO STEREO_INSTANCING_ON STEREO_MULTIVIEW_ON
    , which quadruples the amount of shader variants. There is some code that strips unnecessary variants, but I'm not sure, what the end result is. So, in theory, it could mean a lot of extra memory.
     
  9. sambickley

    sambickley

    Unity Technologies

    Joined:
    Jan 9, 2014
    Posts:
    247
    @tinto22 Hey, do you think this is a PS4 specific issue, or does it happen when targeting other platforms (such as PC/Standalone)?

    If it's PS4 specific, please open up a thread on the Unity forum on PS4 DevNet and we'll provide support for you over there.
     
  10. tinto22

    tinto22

    Joined:
    Mar 28, 2017
    Posts:
    6
    Hi @sambickley, we are experiencing really slow build times on PC too, so it's not PS4 specific.

    For the memory issue I mentioned above, though, I'd need to confirm on PC as I was working on PS4 when I discovered it. The problem is that to determine that, I'll need to do hours-long builds. I'll try to do that today anyway.
     
  11. tinto22

    tinto22

    Joined:
    Mar 28, 2017
    Posts:
    6
    Hi @sambickley ,

    While the issue is more pronounced on PS4, it does still happen on PC. Here are the results of my tests today, all PC/Standalone builds:

    Build 1:
    1. Graphics Settings - Built-in Shader Settings. All dropdowns set to "Built-in shader"
    2. Graphics Settings - Shader Stripping Lightmap Modes. All checked/ticked.
    3. IPreprocessShader - No stripping at all
    Memory Profiler Results
    • GfxDriver - 1.02GB
    • Unity - 1.06GB

    Build 2:
    1. Graphics Settings - Built-in Shader Settings. All dropdowns set to "No support", except Screen Space Shadows
    2. Graphics Settings - Shader Stripping Lightmap Modes. All checked/ticked, except "Realtime Non-Directional" and "Realtime Directional"
    3. IPreprocessShader - No stripping at all
    Memory Profiler Results
    • GfxDriver - 0.57GB
    • Unity - 0.88GB

    Build 3:
    1. Graphics Settings - Built-in Shader Settings. Same as build 2
    2. Graphics Settings - Shader Stripping Lightmap Modes. Same as build 2
    3. IPreprocessShader - Aggressive stripping, including stripping out all use of the Standard shader, which we don't use directly (we have a lot of custom surface shaders).
    Memory Profiler Results
    • GfxDriver - 0.57GB
    • Unity - 0.33GB

    Hopefully that helps shed some light on things...?

    Thanks
     
  12. tinto22

    tinto22

    Joined:
    Mar 28, 2017
    Posts:
    6
    Bump - I'd have thought showing a difference in memory usage of around 1GB would warrant a response from someone at Unity...

    Scared this has slid down too far to be noticed.
     
  13. KAJed

    KAJed

    Joined:
    Mar 15, 2013
    Posts:
    7

    Attached Files:

  14. GameDevCouple_I

    GameDevCouple_I

    Joined:
    Oct 5, 2013
    Posts:
    1,762
    You guys with your "Im gonna switch to unreal" :rolleyes:

    Honestly if you did your research properly, this is resolved by creating proper shader variant collections, turning off "optimise mesh data" etc etc. We went from 4 hour shader compilation + build time down to 11 minutes for huge project.

    But to be honest if this is enough to make you switch engine, I guarantee game dev is not for you because you will run into difficult to solve issues in any engine eventually. Thats what game development is, problem solving.

    Good luck but as a user of both unreal and unity, I think your comments about unreal vs unity are optimistic at best and show you have not used unreal in depth or for very long. You will run into a whole set of other problems once you realise how difficult it is to make anything other than a few types of games, and the mess / nightmare that blueprints is and being unable to do certain things without it.

    I am certain I will see both of you back here in a month or so.

    Rubbish. Unity very much is game engine, unless you think 3d rendering frameworks contain physics, audio, pathfinding + AI tools, timelines, scripting etc etc.
     
    Last edited: Jul 19, 2019
  15. AndreasO

    AndreasO

    Joined:
    Sep 25, 2012
    Posts:
    90
    No reason to be so condescending. I have worked for many years with Unity and it's always been a struggle to fix what Unity should solve out of the box. Unreal won't be drag&drop-easy as Unity tries to always imply in their shiny 1-minute trailers but at least I have the source of the engine and many, many more (integrated!) tools Unity is lacking of.

    I don't have time to argue about engines with you. Think whatever you want but I am done with Unity. If it works for you, great. Unity is missing lots of editor tools and basic functionalities for years and they obviously seem to prioritize adding more unfinished features instead of finishing and making existing tools usable on a production level.

    Anyways, good luck and bye. :)
     
    Vagabond_ likes this.
  16. zhuchun

    zhuchun

    Joined:
    Aug 11, 2012
    Posts:
    379
    I'm here because my game takes longer to compile after upgrading, but a few posts here were off-topic. I do agree with @GameDevCouple_I about the point that game dev is about problem-solving. Personally, I use both engines and have spent over 8 yrs on Unity, if there's an engine way better than its competitors, that would be an easy choice. The real problem of Unity is not what features are not good enough, it's lack of docs. When Unity says something is "Preview" or "Not production-ready", they mean it, but people usually don't understand clearly about what prevents them to do so.

    Regarding this topic, I think @aleksandrk explained it well. What we need is transparency and best practice guideline, black box is never OK for enterprise productive.
     
  17. zombiegorilla

    zombiegorilla

    Moderator

    Joined:
    May 8, 2012
    Posts:
    7,783
    (off topic and ranting thread removed)
    Stay on topic.
     
    zhuchun likes this.
  18. Vagabond_

    Vagabond_

    Joined:
    Aug 26, 2014
    Posts:
    876
    You ARE simply reading my mind !