slow builds: Can someone help me understand shader compiling?

My experience has been that building is Unity is the most flaky and frustrating part of working with Unity.

It seems that every time I build, 8 processes start up for the Unity Shader Compiler, and the build appears frozen for 6-12 hours while this is happening. It would be nice it Unity could at least provide some kind of progress update because the only way to know Unity isn't frozen is to check the task manager and see these processes.




Can anyone explain why this shader compiling takes so long? Let's say I run a build that completes, and then make 1 (non-shader) coding change, does the entire shader compiling need to re-run. If so, why? Or does it cache some of the previous results. I ask because when I upgrade Unity, it seems like some of the time the first build takes must longer, and then subsequent builds run more quickly. But this doesn't always seem to be the case. Sometimes subsequent builds also take many hours. There seems to be no rhyme or reason to the behavior, so I'm hoping someone can shed some light upon how the process works.

Are there any tricks that can prevent these long shader compile times? Also, it'd be great to know some information about what exactly is the thing that determines how long these shader compiles take to complete? I have a pretty large project so I believe the shader compiling time is somehow related to my project size, but I have no idea what in my project is causing the long compiling.

If I'm only making a coding change between builds, why does Unity even need to do all the full build steps? Couldn't there be a "code changes only" build mode where it doesn't rebuild all the asset files, and doesn't do all the shader recompiling?

Lots of questions, I know, but hopefully someone can help! I'm sure I'm not the only person wondering about this issue.

 

It looks like Library/ShaderCache is where these compiled shaders get stored. And it does seem to be the case the subsequent builds are not recompiling these every time. I've upgrade my Unity many times and don't do a lot of building, and so the behavior for how this works isn't clear. This last build that I made took around 10 hours to compile all the shaders.

 

Same here, building a player on a Win7 rig takes 8+ hours, and I only have a clue on what's happening thanks to TaskManager/processes.
This is a huge issue, as some of us need daily builds to hand out.
So far all I've worked out is that Shader Control 2.1, a dedicated asset that's supposed to solve similar issues, does not work. While it deletes redundant keywords and cleans materials, shader compiling remains incredibly long on my rig.

"I have a pretty large project so I believe the shader compiling time is somehow related to my project size"
Actually it's not necessarily related to project size: my current project is an open-world game with a huge world, and I can confirm that building a player for it used to be a breeze until Unity 5.6.1 or so.

 

Bump! We're getting this on 2017.2. Also tried Shader Control! The only thing that seems to fix it is to delete all the shaders and materials from the project..which let's face it isn't ideal.

Anyone manged to figure this out?

 

bump

 

I've just learned to live with this. Really, like I mentioned, the long build only happens the first time for a new version of Unity. After that builds take about an hour. Doing script only builds can also be helpful (when applicable) because these for me only take around 10 minutes.

 

I'm not really sure this happens only when you install new versions, because the insane build time happened each time I built the game with 2017.4.3f1 (two Windows builds and two OSX builds). You really can't do quick releases for important bug fixes, this way.

 

The lack of reply from unity on this thread is very telling. Sigh...waiting for a compile now...been waiting a long while ALREADY!!!

Does UE4 compile much, much faster than Unity generally for a similar scene?

 

The lack of reply from Unity signifies nothing on these forums.
The lack of reply from higher than normal technical issues is common, just look at the GFX.WaitForPresent threads.
Even basic questions can go unanswered. Unless your are a new user and post a basic question, then a Unity employee will most likely respond. (At least this is what I've observed on the Editor and General Support forum).

 

Try this.

Code (CSharp):

1. using System.Collections.Generic;
2. using UnityEditor;
3. using UnityEditor.Build;
4. using UnityEditor.Rendering;
5. using UnityEngine;
6. using UnityEngine.Rendering;
7. using static UnityEngine.ShaderVariantCollection;
8.  
9. class CustomPreprocessShaders : IPreprocessShaders
10. {
11.     private const string SHADER_VARIANT_COLLECTION_ASSETPATH = "Assets/ShaderVariants_ForOnlyCompileUsedVariant.shadervariants";
12.  
13.     private ShaderVariantCollection m_ShaderVariantCollection;
14.     private ShaderVariant m_ShaderVariantCache;
15.     private List<string> m_CompilerKeywordSet;
16.  
17.     public int callbackOrder { get { return 0; } }
18.  
19.     public CustomPreprocessShaders()
20.     {
21.         m_ShaderVariantCollection = AssetDatabase.LoadAssetAtPath(SHADER_VARIANT_COLLECTION_ASSETPATH, typeof(ShaderVariantCollection)) as ShaderVariantCollection;
22.         m_CompilerKeywordSet = new List<string>();
23.  
24.     }
25.  
26.     public void OnProcessShader(Shader shader, ShaderSnippetData snippet, IList<ShaderCompilerData> shaderCompilerData)
27.     {
28.         m_ShaderVariantCache.shader = shader;
29.         m_ShaderVariantCache.passType = snippet.passType;
30.         for (int iShader = 0; iShader < shaderCompilerData.Count; ++iShader)
31.         {
32.             ShaderCompilerData iterShaderCompilerData = shaderCompilerData[iShader];
33.             ShaderKeyword[] shaderKeywords = iterShaderCompilerData.shaderKeywordSet.GetShaderKeywords();
34.  
35.             m_CompilerKeywordSet.Clear();
36.             for (int iKeyword = 0; iKeyword < shaderKeywords.Length; iKeyword++)
37.             {
38.                 ShaderKeyword iterKeyword = shaderKeywords[iKeyword];
39.                 m_CompilerKeywordSet.Add(iterKeyword.GetKeywordName());
40.             }
41.             m_ShaderVariantCache.keywords = m_CompilerKeywordSet.ToArray();
42.  
43.             if (!m_ShaderVariantCollection.Contains(m_ShaderVariantCache))
44.             {
45.                 shaderCompilerData.RemoveAt(iShader);
46.                 iShader--;
47.             }
48.         }
49.     }
50. }