Why does Unity build Shader Variants on each build?

I have a few different questions here. Compiling shader variants has practically ground progress on our project to a halt. I've added a script via the Shader Variant API (not sure of official name - but the IPreprocessShaders interface), to inspect what variants it is building. Through this, it has become apparent that it is building over 1 million variants. Sometimes, this takes hours. Strangely, sometimes it takes about 20-30 minutes. Either way, it's killing us on this project.

I've investigated in more depth and have some findings:

a) There are many many instances of the same shader+keyword configuration - sometimes over 900. I printed out all values I could and everything matches on each of those unique instances.
b) In a sample project, I was able to reproduce this by simply adding a new material referencing the Standard shader - as long as the material was being used in a scene. With 1 material, I got 2 instances of each unique configuration. With 2 materials, I got 3 instances, and this then grew linearly as more materials were added.
c) I tried using the shader variant API to ignore all unique variants after the first one it saw. This didn't work - faster builds, but everything in-game was magenta.

I have 3 main questions:

1) Why does Unity increase the number of variants linearly with each material use? Surely it only needs one copy of each possible configuration of the shader+keywords at runtime, so why is it building many many times more than it needs to?
2) Why does Unity build shader variants EACH time the game is built. I'm building for PS4 (in this example), and each time I build the window pops up saying it's building shader variants. Nothing changed since last time I built so why isn't this work cached and reused for subsequent builds on the same platform?
3) What, if anything, can we do to fix this so we're no longer basically blocked on this project. Unity was meant to save us money by improving our iteration times but at this point it has ground us to a halt.

P.S. I have read through this thread already (https://forum.unity.com/threads/compiling-shader-variants-taking-ages.527724/) and tried everything suggested. Nothing works and there's no response from Unity in the thread.

Also, I am running 2018.3.13f1.

 

Are you using a shader variant collection asset? If not, Unity will iterate over every material in the project and ask to compile every single variant. My understanding is it is not actually recompiling them every time, but checking the disk first and skipping if found. However on modern PCs it probably takes longer to access the disk than to just compile the variant and re-save it every time, especially if you're not running your project off of a fast SSD.

I do believe this is an area of active work over at Unity. They've been seeing issues with shader compile times for LWRP and HDRP builds, in large part because they've internally finally moved to having internal groups building larger projects that better reflect real world use cases than the hyper focused demos of the past.

 

I have tried using a shader variant collection asset - unfortunately it made no difference whatsoever...

 

Ok, good to know that it's not actually compiling them - but it's mostly a moot point as it's still taking an extremely long time, regardless - so I'd still like advice on how we can improve it.

Furthermore, the numbers we are seeing just seem pretty insane. I'll give an example - prior to doing some pretty aggressive stripping in my IPreprocessShader, the profiler was showing memory usage on PS4 under the "Unity" category as 1.4GB. After some aggressive stripping, the profiler now shows 0.6GB under the "Unity" category. This implies, of course, that 800MB of memory was being used up by shaders, or some other bookkeeping that is tied to shaders. This is pretty crazy, especially given that it's the "default" behaviour and I've had to write this IPreprocessShader to get it back under control.

Does this seem out of whack to you, @aleksandrk? Does it sound like a bug, or is this what people should be expecting to see? For the record, this is a VR project with Single Pass Stereo on - so I understand that increases the number of shader variants... but... 800MB??

Thanks for any help you can offer.

 

Maybe we have to create a ShaderVariantSettings.asset file, that contains the properties to skip all shaders? (I don't know. All I saw was an option to skip all shaders.)

 

Hi @sambickley, we are experiencing really slow build times on PC too, so it's not PS4 specific.

For the memory issue I mentioned above, though, I'd need to confirm on PC as I was working on PS4 when I discovered it. The problem is that to determine that, I'll need to do hours-long builds. I'll try to do that today anyway.

 

Hi @sambickley ,

While the issue is more pronounced on PS4, it does still happen on PC. Here are the results of my tests today, all PC/Standalone builds:

Build 1:

1. Graphics Settings - Built-in Shader Settings. All dropdowns set to "Built-in shader"
2. Graphics Settings - Shader Stripping Lightmap Modes. All checked/ticked.
3. IPreprocessShader - No stripping at all

Memory Profiler Results

• GfxDriver - 1.02GB
• Unity - 1.06GB

Build 2:

1. Graphics Settings - Built-in Shader Settings. All dropdowns set to "No support", except Screen Space Shadows
2. Graphics Settings - Shader Stripping Lightmap Modes. All checked/ticked, except "Realtime Non-Directional" and "Realtime Directional"
3. IPreprocessShader - No stripping at all

Memory Profiler Results

• GfxDriver - 0.57GB
• Unity - 0.88GB

Build 3:

1. Graphics Settings - Built-in Shader Settings. Same as build 2
2. Graphics Settings - Shader Stripping Lightmap Modes. Same as build 2
3. IPreprocessShader - Aggressive stripping, including stripping out all use of the Standard shader, which we don't use directly (we have a lot of custom surface shaders).

Memory Profiler Results

• GfxDriver - 0.57GB
• Unity - 0.33GB

Hopefully that helps shed some light on things...?

Thanks

 

Bump - I'd have thought showing a difference in memory usage of around 1GB would warrant a response from someone at Unity...

Scared this has slid down too far to be noticed.

 

According to ShaderControl, there are 29 keywords in the shader I'm using: https://github.com/traggett/UnitySpriteShaders/blob/master/SpriteShaders/SpritesPixelLit.shader

It claims the result is 52 variants... but according to Unity:

 

You guys with your "Im gonna switch to unreal"

Honestly if you did your research properly, this is resolved by creating proper shader variant collections, turning off "optimise mesh data" etc etc. We went from 4 hour shader compilation + build time down to 11 minutes for huge project.

But to be honest if this is enough to make you switch engine, I guarantee game dev is not for you because you will run into difficult to solve issues in any engine eventually. Thats what game development is, problem solving.

Good luck but as a user of both unreal and unity, I think your comments about unreal vs unity are optimistic at best and show you have not used unreal in depth or for very long. You will run into a whole set of other problems once you realise how difficult it is to make anything other than a few types of games, and the mess / nightmare that blueprints is and being unable to do certain things without it.

I am certain I will see both of you back here in a month or so.

Rubbish. Unity very much is game engine, unless you think 3d rendering frameworks contain physics, audio, pathfinding + AI tools, timelines, scripting etc etc.

 

No reason to be so condescending. I have worked for many years with Unity and it's always been a struggle to fix what Unity should solve out of the box. Unreal won't be drag&drop-easy as Unity tries to always imply in their shiny 1-minute trailers but at least I have the source of the engine and many, many more (integrated!) tools Unity is lacking of.

I don't have time to argue about engines with you. Think whatever you want but I am done with Unity. If it works for you, great. Unity is missing lots of editor tools and basic functionalities for years and they obviously seem to prioritize adding more unfinished features instead of finishing and making existing tools usable on a production level.

Anyways, good luck and bye.

 

I'm here because my game takes longer to compile after upgrading, but a few posts here were off-topic. I do agree with @GameDevCouple_I about the point that game dev is about problem-solving. Personally, I use both engines and have spent over 8 yrs on Unity, if there's an engine way better than its competitors, that would be an easy choice. The real problem of Unity is not what features are not good enough, it's lack of docs. When Unity says something is "Preview" or "Not production-ready", they mean it, but people usually don't understand clearly about what prevents them to do so.

Regarding this topic, I think @aleksandrk explained it well. What we need is transparency and best practice guideline, black box is never OK for enterprise productive.

 

(off topic and ranting thread removed)
Stay on topic.

 

You ARE simply reading my mind !

 

I'm having this problem now on an Android Quest project. My build machine just completed its first full build on the platform. It took 3 days.

Any tips for finding out what is bringing in additional variants in a project?

 

Thanks a lot, "optimise mesh data" was the tick I missed. Now it's fine on Oculus Quest.

 

How does turning off 'Optimize mesh data' in Project Settings > Player > Other help shader compilation?
Also, what other tweaks (hinted at by etc etc) helped you reduce your compilation time down to 11 minutes?
I don't have an issue with compilation time personally btw - my PS4 game builds in around 10 - 15 mins.

One of the problems I have with Unity is how often tricks like this are required - switching off an 'optimize' feature to make things more optimal. It's counter intuitive.

Yes, game dev is problem solving. But ideally you want most of those problems to be in the game mechanics and design, rather than technical issues. I spend almost as much time on technical issues as I do on actual game development. I suspect this is true for most devs.

 

Yep !

 

It doesn’t, it just also reduces the build time. That feature will remove unused mesh data (like if a mesh has a second UV, but your project doesn’t use light maps). Mainly it helps reduce the build size, but it generally doesn’t do anything for perf.