Compiling Shader Variants Taking Ages

This is radiculas, we upgrade from 2018.1 to 2018.3.0f2，and now it takes me 400 seconds to compile a "Standard Shader".

I am also waiting for the solution.

 

I'VE FOUND A FIX! If your game uses the Unity Post Processing Stack, then if you remove it the time will go much quicker! My shader variants went from 15k to just 120.

 

I think I may have found a way around this, but it only works for a new project. I am using 2018.3 so it's possible they did fix something internally.

Create an empty project. Build it for your platform. It will compile some shader variants. Slowly add all your assets, building every so often. This seems to keep the compile time to a reasonable level. Further changes to the scene don't appear to cause all shader variants to be recompiled.

This is a workaround and I hope that Unity comes up with a better way of handling this.

 

Unfortunately my game doesn't use the Post-Processing stack. Even with 120 shader variants I have seen it repeatedly compile them many times.

 

I was pulling my hair out because of extremely slow shader variants compilation. I'm currently using Unity 2018.3.0f2, this slowness began for me since some versions earlier (from last year).

But when I unchecked "Optimize Mesh Data" (in Project Settings -> Player -> Other Settings -> Optimization), this made a MAJOR difference to the build, far less shader variants were being compiled after I unchecked that option. And in subsequent builds, the cached shader variants compile way faster (as long the shaders aren't modified).

I guess that option was renamed? Is "Optimize Mesh Data" equivalent to "Optimize Meshes" that was mentioned?

 

Another thing I did that seemed to make a positive difference:
In the Player Settings for Android, in "Other Settings -> Rendering -> Graphics APIs" option, I removed Vulkan support (I kept only OpenGLES3), I guess this halves the work from the shader variants compilation.

I had also bought the asset "Shader Control" (by Kronnect), this asset really helped me to identify the shaders that were generating more variant permutations by having lots of keywords.
Then, you can easily identify and deactivate unused/unneeded keywords from the shader, thus lessening the number of generated variants (and finally, lessening the build time).
https://assetstore.unity.com/packages/vfx/shaders/shader-control-74817

 

I just imported this:

After that I wanted a build for my android, but compiling stucked there: compiling shader variants... xx of about 15552

I started the build 10 minutes ago. My computer is lagging, and loud as a vacuum cleaner :S

Using the latest Unity 2018.3.0f2

 

Is there anywhere a response from Unity what is going on here? We used make builds in 10-20 min. Now it takes around 2 hours. All the unused shaders being compiled. We see it as a clear bug.

 

Unchecking "Optimize Mesh Data" made a HUGE difference for me. However I don't understand what the trade off to having this unchecked might be.....

 

If you're able to reproduce this issue, could you please submit a bug-report, including the project to reproduce the issue, as described in this document:
https://unity3d.com/unity/qa/bug-reporting

Submitting a bug-report allows Unity Technologies to take a look at this issue.

After you submitted the bug-report, you receive a confirmation email with a bug-report Case number. You can post this Case number (the number only, not the link) in this forum thread for Unity staff to pick up, in case they see it.

 

@multimediamark If you provide a small repro project instead of your full project, chances are way higher someone is actually looking at it.

 

Unity, please give some kind of progress to the user during Compiling shader variants (HDRP/Lit)!

 

Aren't the LWRP an HDRP still in preview and as such likely to be prone to problems like this still?

 

Did anyone here already submit a repro case?
I do hope Unity has one as they have let this problem become waaay too annoying.
Our project is over 80gb and not the best/easiest repro case.

 

MikeHergaarden, they already know about it, they just can not do anything, because this is how HDRP works. That's just terrible. I'm really not understang WHAT THE HELL did they do to make the shaders compiles SO LONG?! I have not seen such things happen in any other engine. Last time I built my project 24 hours. After that, I'm not even trying, just hope that something will change for the better.

 

A couple of notes on our project using 2018.1.9
* Caching does appear to work for us as subsequent builds were about 40% faster than a build with a cleared cache.
* Unchecking the following reduced our build times considerably ( a further 33% from the above) as we were not needing these particular variants.



*Disabling "Optimise Mesh Data" in the player setting reduced times by a further 25% from the above.

however I don't believe its actually a shader "compilation" issue. Rather I think its reading the shader variants to determine what mesh structures can be removed. I think the progress bar description is just red herring and Unity is doing what it should. I'm guessing its doing some sort of Material to Mesh to Shader cross check resulting in a lengthy process.

My advice would be to enable "Optimise Mesh Data" for release builds and turn it off for Dev purposes.

 

By this time , I read the entire thread , and still not finish this nightmare.(about 40 minutes now).

I will try all stuff I read , and sure will comment results for research.

 

1:05 hours and nothing , stuck in HDRP/Lit 1038 of 1040

 

That moment when he wrote this half a month ago and still no comment from him..

p.s. i have the same problems..
p.p.s Unity 2018.3.4f1

 

Spoiler

 

I seem to have been able to get my builds down to a reasonable time. But it's still adding thousands of shader variants that I don't seem to need and it's roughly doubling the size of all my scene bundles. It seems to me that the Standard Shader is just grossly ineffective at whittling down which shader variants it actually needs.

 

I't would be nice to have a word from Unity staff, at least like: guys it is the way it is.

 

I have had this terrible problem up until now. Reduced from what seemed like forever down to 1 minute build time. Here are my specs:

2018.3.4f1
Build Platform: PC
LWRP
"Graphics Jobs (Experimental)" Disabled
"Optimise Mesh Data" Disabled
"Instancing Variants" Strip All

I also went to each scene and Saved out a shader variant asset after hitting clear button.

It's a fairly big project taking up 9GB for the root folder. Lot's of art assets and even asset store purchases.

So not all hope is lost! Unity team really needs to get on this and fix whatever is broken. They can't see this many complaints and act like there's no problem. This is a big deal breaking so many projects and wasting countless hours of frustration! We need video tutorials showing what Unity expects of us coming from earlier Unity versions which used to build fast! We need a user interface where you can click the "help" button and actually see something other than a 404 webpage.

 

I was encountering the same issues as above, and tried all the suggestions mentioned here with no luck. It was still taking a long time on around 20k shader variants.

So I ripped everything out of the project and added each asset folder back in, one at a time. In my case, I had a folder of obj files that had "use external materials(legacy)" selected as their import settings under the materials tab. Changing that setting to the other option, "use embedded materials" with on demand remap, solved my issue entirely.

My bet is that specific option reeeeeeally doesn't play well with the new unity version builds. Maybe this change, in combo with the suggestions above, will help?

 

If you're able to reproduce this issue, any chance you could you submit a bug-report, including a (mini) project to reproduce the issue, as described in this document?
https://unity3d.com/unity/qa/bug-reporting

Submitting a bug-report allows Unity Technologies to take a look at this issue. Unfortunately, the forum isn't really the place where UT is looking for bug reports (more).

After you submitted the bug-report, you receive a confirmation email with a bug-report Case number. You can post this Case number (the number only, not the link) in this forum thread for Unity staff to pick up, in case they see it.

 

FYI, the "OVR Performance Lint Tool" from the "Oculus Integration" asset keeps a list of FixRecords which contain pointers to game objects, materials and shaders. Apparently, that causes problems, because my build times increased by an hour after I imported the Oculus scripts.

I modified OVRLint.cs around line 142 to add another button:

Code (CSharp):

1.         if (GUILayout.Button("Clear", EditorStyles.toolbarButton, GUILayout.ExpandWidth(false)))
2.         {
3.             mRecords.Clear();
4.             mRuntimeEditModeRequiredRecords.Clear();
5.         }
6.  

clicking that button and then running the "Build and Run" makes my project build within a few minutes again

 

I could try, but problem is more on this being blamed on the users. Lack of documentation isn't a bug. Setting up your projects wrong isn't a bug.

I'll see if I can make a mini project anyway.

 

Oh just to add in. This happens when creating asset bundles too. I will get a bug report asap for it out of work hours.

 

I think this is just a performance problem with old assets. I can't seem to reproduce this issue in a smaller project. In my big project, the loading bar goes backwards. In my reproduction, it progresses as one would expect.

 

I've just opened a new project with the Lightweight-VR Template and hit build for Android Daydream. Now I'm waiting for an hour and it's still compiling 17k shader variants. There's no user content in this project, only the default demo scene with a couple of objects apperently some post-process effect. I'm working on a high-end machine and usually I have build times of only a couple of minutes, even in complex projects. Usually it compiles a few hundreds or maybe two thousand, but not that many.

I haven't tried the variant collection asset, but I already know that even if it works, it's ugly to work with, because it becomes invalid the moment I change any shader or material and then I have to compile them anyway, save the asset to disk manually and manually drag the asset into the graphics settings windows. Also I think that the empty template project should "just work out of the box", meaning I expect to simply hit build and run the demo in a reasonable time.

Edit: Oh no, it's done compiling the 17k shader variants after roughly an hour, but now the loading bar jumps to the start and shows that there are more shaders and 10k more variants to compile. I would report a bug, if it wasn't the project provided by Unity without modifications.

 

I do wonder: What fundamentally changed between 2017 and these current versions that makes this happen? We're trying to work on an Oculus Go project and these build times are nightmarish.
Tinfoil hat time: Is Unity trying to force users to use their new render pipeline by artificially making it so the original pipeline is terrible to work with?

 

Can you post the link from where I can download this project? I would submit the bug-report then.

 

The new pipeline has the same problem.

 

From the Unity Hub create a new project with Unity 2018.3.5 and select the Lightweight VR Template. I then switched to Android and set Daydream in VR Settings. It’s been this taking really long to build because of the shader variants ever since last year.

 

I submitted bug-report Case 1128018, but QA identified it as duplicate today. Anyway, we now know that they're aware of the problem and working on a fix eventually.

The bug-report that handles this case is:
https://issuetracker.unity3d.com/is...render-pipeline-slash-lit-shader-is-compiling

 

That's great news! Thank you for the effort. Will be voting on that issue.

 

The same problem
Unity 2018.3.5f1
Android / Vuforia
Gradle, IL2CPP, .NET 4x, standard renderer

 

I have a project with the post-processing asset from unity imported. Development build times take up more than 30min for each platform. There is not much imported aside from a few models and scripts.

I compiled over 80+ unity projects in the past years and I can assure that none ever felt this slow. I'm gonna try all the alternatives posted above and will notify you guys.

Also sad that Unity didn't respond yet

 

This issue has been dragging around for 6 months, with no helpful response from Unity. Why is it taking so long for a response? I have encountered this problem twice now on two different projects, this is why I am here again. I have a very small project and the slowest part of making an Android build is shader compilation, every time. Now that the project is getting bigger this seem to be growing exponentially. We had this problem on a previous Android project as well. Now, Unity, I would like you to imagine, for a minute, how much this bug is costing developers. Perhaps what you need, is to be building games with your own tools in order to test your tools properly? The current strike rate on projects with this issue here is 2 out of 2.

 

I have the exact symptom. I have a scene that is using Unity's ARKit tools for image tracking.

This is were it gets strange...
My project builds in 4 minutes. But in the build settings, when I position a scene above the image tracking scene under Scenes In Build, the build takes 20 minutes. This happens with any scene, even a completely empty scene (just a camera with light). And it happens 100% of the time. When I place the scene, or any scene, below the Image Tracking scene then the build takes 4 minutes. In short, if the Image Tracking scene is at the top of the list in the 0 position in Scenes In Build, the build time is 4 minutes. If the Image Tracking scene is in any other position other than 0 in that list, then the build takes 20 minutes.

Any ideas about what is going on?

 

For us, Unity appears to be compiling every single shader variant for every material. So if two materials reference the same shader, Unity attempts to compile all variants of that shader twice. Reducing the number of materials and shader references in each scene seems to speed up the compile process slightly (compiling a shader seems to take longer the first time around, and speeds up on subsequent passes), but if a shader has 10,000 variants, like the Standard shader, it still seems like a crazy amount of processing.

 

Unchecking "Optimize Mesh Data" (Settings -> Player -> Other Settings -> Optimization) fixed build times for me. Went from 20 minutes to less than a minute. I also found these ->

Warning to all my friends: BEWARE Optimise Mesh Data!!
https://forum.unity.com/threads/warning-to-all-my-friends-beware-optimise-mesh-data.544735/

OPTIMIZE MESH DATA PLAYER SETTING CAN DRASTICALLY INCREASE BUILD TIMES
https://issuetracker.unity3d.com/is...-setting-can-drastically-increase-build-times

 

I tried another solution. I created a LightweightRenderPipelineAsset and added it to the Scriptable Render Pipeline Settings field. I then had to update materials to LightweightRP Materials. I then checked "Optimize Mesh Data" and then set Instancing Variants to "Strip All" and removed all Preloaded Shaders. My build time was quick, less than one minute. After testing the build I discovered that ARKit does not support LWRP, so I removed it from the field and did another build, that build took 3 minutes...

So I now I have -
- "Optimize Mesh Data" checked
- Instancing Variants set to "Strip All"
- 0 Preloaded Shaders

And my build times are now quick... So it seems that updating my materials to LightweightRP Materials fixed my long build times?

 

This was my worry too. Mines much better after tweaks (pretty much all the ones you have mentioned), but not amazing and I imagine it's because my project has been updated to literally like 20 different Unity editor builds over the past 2 years. I bet if I reimported all materials or something I could get it to go faster.

 

Got and empty project in latest unity beta (2019.2.a11)
Started with template project HD.
Enabled VR.

Created a build, below some stats :
1 edit : Second build almost instant....
2 edit: copied over models from my failing project (extreme long shader compiling), kept building a version, compile time increases much, now its taking very long again.....
3 edit : waited for 2 hours for a build, then ended task.....
4 edit : Unchecked all materials : "Enable GPU Instancing", success build in 10 minutes or so, this seems to have an huge impact!!!!!!!

Code (csharp):

1.  
2. Compiled shader 'Hidden/HDRP/Deferred' in 46.06s
3.     d3d11 (total internal programs: 32, unique: 32)
4. Shader warning in 'Hidden/HDRP/Deferred': 'LIGHTLOOP_DISABLE_TILE_AND_CLUSTER' : macro redefinition at Packages/com.unity.render-pipelines.high-definition/Runtime/ShaderLibrary/TextureXR.hlsl(101) (on d3d11)
5.  
6. Compiling Vertex program with UNITY_SINGLE_PASS_STEREO LIGHTLOOP_DISABLE_TILE_AND_CLUSTER
7. Platform defines: UNITY_ENABLE_REFLECTION_BUFFERS UNITY_USE_DITHER_MASK_FOR_ALPHABLENDED_SHADOWS UNITY_PBS_USE_BRDF1 UNITY_SPECCUBE_BOX_PROJECTION UNITY_SPECCUBE_BLENDING UNITY_ENABLE_DETAIL_NORMALMAP SHADER_API_DESKTOP UNITY_LIGHT_PROBE_PROXY_VOLUME UNITY_LIGHTMAP_FULL_HDR
8. Compressed shader 'Hidden/HDRP/Deferred' on d3d11 from 1.54MB to 0.38MB
9. Compiled shader 'Hidden/HDRP/FinalPass' in 5.05s
10.     d3d11 (total internal programs: 256, unique: 256)
11. Compiled shader 'HDRP/Lit' in 152.55s
12.     d3d11 (total internal programs: 272, unique: 268)
13. Shader warning in 'HDRP/Lit': 32-bit floating-point operations flush denorm float literals to zero, 0.000000 is losing precision (this warning will only be shown once per compile) at Packages/com.unity.render-pipelines.high-definition/Runtime/Material/Lit/LitProperties.hlsl(184) (on d3d11)
14.  
15. Compiling Fragment program with DECALS_3RT SHADOW_LOW DEBUG_DISPLAY USE_FPTL_LIGHTLIST
16. Platform defines: UNITY_ENABLE_REFLECTION_BUFFERS UNITY_USE_DITHER_MASK_FOR_ALPHABLENDED_SHADOWS UNITY_PBS_USE_BRDF1 UNITY_SPECCUBE_BOX_PROJECTION UNITY_SPECCUBE_BLENDING UNITY_ENABLE_DETAIL_NORMALMAP SHADER_API_DESKTOP UNITY_LIGHT_PROBE_PROXY_VOLUME UNITY_LIGHTMAP_FULL_HDR
17. Compressed shader 'HDRP/Lit' on d3d11 from 5.78MB to 1.40MB
18. Compiled shader 'Hidden/HDRP/DebugColorPicker' in 1.31s
19.     d3d11 (total internal programs: 4, unique: 4)
20. Compiled shader 'Hidden/HDRP/DeferredTile' in 153.78s
21.     d3d11 (total internal programs: 248, unique: 248)
22. Shader warning in 'Hidden/HDRP/DeferredTile': 'LIGHTLOOP_DISABLE_TILE_AND_CLUSTER' : macro redefinition at Packages/com.unity.render-pipelines.high-definition/Runtime/ShaderLibrary/TextureXR.hlsl(101) (on d3d11)
23.  
24. Compiling Vertex program with UNITY_SINGLE_PASS_STEREO LIGHTLOOP_DISABLE_TILE_AND_CLUSTER
25. Platform defines: UNITY_ENABLE_REFLECTION_BUFFERS UNITY_USE_DITHER_MASK_FOR_ALPHABLENDED_SHADOWS UNITY_PBS_USE_BRDF1 UNITY_SPECCUBE_BOX_PROJECTION UNITY_SPECCUBE_BLENDING UNITY_ENABLE_DETAIL_NORMALMAP SHADER_API_DESKTOP UNITY_LIGHT_PROBE_PROXY_VOLUME UNITY_LIGHTMAP_FULL_HDR
26. Compressed shader 'Hidden/HDRP/DeferredTile' on d3d11 from 6.37MB to 1.47MB
27. Compiled shader 'Hidden/HDRP/TerrainLit_Basemap' in 149.04s
28.     d3d11 (total internal programs: 240, unique: 240)
29. Compressed shader 'Hidden/HDRP/TerrainLit_Basemap' on d3d11 from 5.41MB to 1.32MB
30. Compiled shader 'HDRP/TerrainLit' in 167.30s
31.     d3d11 (total internal programs: 240, unique: 240)
32. Compressed shader 'HDRP/TerrainLit' on d3d11 from 6.18MB to 1.46MB
33. Compiled shader 'Hidden/Internal-PrePassLighting' in 1.05s
34.     d3d11 (total internal programs: 104, unique: 78)
35. Compiled shader 'Hidden/Internal-ScreenSpaceShadows' in 1.02s
36.     d3d11 (total internal programs: 64, unique: 40)
37. Compiled shader 'Hidden/Internal-DeferredShading' in 1.67s
38.     d3d11 (total internal programs: 108, unique: 108)
39.  

 

Sorry to bump this old thread. But I got a question for experts here

In a standalone player build, does Instancing Shader stripping option takes precedence over ShaderVariantCollection? As in, even if I specify Instancing variant to be included in a ShaderVariantCollection, add it to preloading list, these shaders are still stripped of any instancing shader code?

Reason:
- I use DrawMeshInstanced to render objects, so there might not be a direct reference to shaders in my scene.
- So I thought I would manage this myself by using ShaderVariantCollection.
- But my DrawMeshInstanced simply doesn't render in the build when "Strip All" is enabled.
- So can I use "Strip All" with Instancing material/shader? I thought people have done this with success?

 

Unchecking optimize mesh data made the build time go from 1.5 hours to 7 minutes here.

 

Does this affect performance? Is your game running slower now?

 

I think LWRP team's current suggestion is to disable this option for development build, and only use it for release build. I don't think it will slow your game down much, other than memory usage.

The best answer is from: https://forum.unity.com/threads/fee...t-render-pipeline.562291/page-11#post-4459627

The slowdown is mainly caused by Shader variant stripping and Mesh data stripping each has to account for the other, so you basically run the same process twice (seems like a O(n * m) process)

 

Also back to my previous question, I found out this is an edge case bug caused by a combination of Addressables, DrawMeshInstanced and Shader instancing variant stripping level ("Strip All"), already opened a bug report with repro. Troubleshooting this was not easy...