Standard Shader duplicated in Asset Bundle Build

Hey,

I've been investigating why our asset bundle builds use more memory than our non-bundle builds, and have narrowed it down to the shaders.

The detailed memory view shows us that on an asset bundle build, ShaderLab takes up around 350mb, whilst on a non-asset bundle build it uses < 80mb.

So I dumped out the list of all loaded shaders with Resources.FindObjectsOfTypeAll<Shader>() and the result is that the asset bundle build has numerous copies of some shaders, including 27 copies of the Standard shader, and 31 copies of the Standard (Specular setup) one. (The non-asset bundle build has only one copy of each).

How do I fix this? Some of the shaders that are duplicated are custom shaders which are not being included in the correct asset bundle (so fair enough) but how do I fix this for the standard shader?

 

Hey,

Thanks for that. It was the solution I was thinking of, though I assumed it was more of a hack.

Is the Scriptable Build Pipeline ready for production use?

 

Okay thanks! I'll get sorting this at some point and see if I have any issues

 

Hey,

So I tried it the first way and we're still getting some duplication, think there's still some things using the default material, but it's a large project so tracking them down is hard.

Since we're not (currently) using variants and only load scenes from the bundles, I was going to give the SBP a shot. Can you link me to the sample code that de-duplicates the shaders? @Ryanc_unity

 

Thanks!

 

Tell me there's a way to prevent this from occuring with ALL shaders.
We simply cannot track all the shaders used in all our assets. We have references towards particles. Having two version of ALL standards shaders in our project is also a huge mess.

Bundling all our shaders are ALSO not a solution. Tell me there's a way to tell the AssetBundle pipeline to ignore specific asset or even specific type.

On the same idea, it also duplicate audio asset! It's just insane! I cannot end up putting the whole freaking game in thousand of asset bundle just to prevent everything from duplicating everything else!

 

Aren't standard shaders always included in the game build anyway? Why are they even duplicated to begin with, why not simply store a reference in the AssetBundle?

I need some clarification about this: is this doing exactly what I described - replace duplicated shaders by references?

In any event, I share @LightStriker 's concerns. AssetBundles should not duplicate anything that's contained in the game build, and definitely nothing that Unity ships with (e.g., standard shaders).

 

Ok, this is going beyond my patience.

We managed to find a way to reroute all the material towards shader downloaded from the Unity website.

However, once packaged and loaded in game, they ALL are missing Keywords. A standard shader without the _NORMALMAP _METALLICGLOSSMAP and _EMISSION Keywork is very ugly.

I'm lacking solution here. 1.3Gb Resources folder, or the game simply doesn't work or look like S***.

[EDIT] Found out the keyword issue was related to the shader being unload (somehow), but still being present on the GPU. Look story short, we figured it out. It's still a mess, but at least it's working.

 

Is there any other way (besides the above mentioned way with downloading the source of buildtin shaders), to use the old BuildPipeline.BuildAssetBundles method and not having the duplication of buildin shaders in each assetbundle? We are developing a plugin and do not want to force our users to use the package manager to install the Scriptable-Buildpipline.

Using DefaultBuildTasks.Preset.AssetBundleBuiltInShaderExtraction feels also a bit like a hack, as it looks like that the references to the builtin shaders are just removed afterwarts, as the slots in the assetbundle are just set to null. Maybe we can do this also with the old BuildPipeline.BuildAssetBundles?

 

Thank you for the detailed Feedback!

However, is there any other way of handling this issue, maybe in the future besides DefaultBuildTasks.Preset.AssetBundleBuiltInShaderExtraction?

 

Sure: The Problem is that we do not want that our plugin has a dependency to other plugins and/or packages like the SRP. This would mean, if a user installs our plugin we, he needs to install this package, too. We want to publish our plugin to the Assetstore, therefor we are using the legancy assetPackages.

However, maybe you know a way to automatically install dependecy package, maybe this would be a workaround, so that we can use the SRP in our plugin?

Thank you

 

What version of unity was this happening on? We're updating from 2017.3 to 2019.2 and seeing similar issues

Also, how are we supposed to use this with the normal BuildPipeline.BuildPlayer() funtionality. The method you have provided does not create a manifest file, but the BuildPipeline.BuildPlayer will strip unused files out if a manifest is not provided. The CompatibilityBuildPipeline path returns a manifest, but does not appear to allow us to strip the built-in shaders

 

Would it be possible to flag objects as always being part of the build, so they could be ignored when making AssetBundles?

We already have a "Always Included Shaders" list in the player settings.

 

Bumping thread any answer?

 

After way more working with AssetBundles, I came to the following conclusions:

1) Two pipeline (scripted and build): It's dumb; they don't talk to each others and there's no way to know if one file is in one pipeline or not, or both.
2) Black boxes: No way to know what was actually bundled with a file, no output. The references can be manually found, but not once it's in a bundle. We found it often make for bigger bundles then what would be needed, and Unity tends to package stuff we aren't sure if they are really needed.
3) Duplication of files: Not only shader, but audio mixer, texture, sprite, sprite atlas, animation, animator, etc. Pretty much anything that is reused between bundles get duplicated. It's a massive headache. You have to manually track down those references and manually put them in separated bundles to prevent duplication. (but still duplicate when you build)
4) Load by Type: Similar to the Ressources issue, loading all object by types pretty much makes Unity load ALL objects in the bundle, look at their type, then discard the wrong ones. This is dumb. The headers should contain the information required to prevent this from occurring. Name, Assembly Qualified Name, GUIDs, whatever!

Unless flagged "Include in Build", Sprite Atlases do NOT work in AssetBundles. At all. But still get duplicated!

 

@LightStriker
Thank you for the detailed feedback.

@Unity
Can you/someone give us some insights about the things mentioned above?

 

@Ryanc_unity
How do we avoid custom shaders from being duplicated in the player and into an asset bundle? If we do not put the shader in the Always included shaders it doesn't work correctly with the asset bundle, and if we do put the shader there we get a shader duplication (one in the asset bundle, other in the player).

Thanks!

 

5) In the AssetBundle pipeline, you cannot reference an asset, you pass a path. If for some reason this path leads to multiple asset (say a FBX with animation and a mesh), everything is pulled, even if you only 1 one asset out of many.

I'm adding this because I have animation duplication between weapons bundle. I tried to bundle all the animation, and found out I pulled 200Mb of unwanted mesh. Great.

 

6) Sprite Atlas are duplicated. Always. Unless you script something. I'm not joking. SpriteAtlas don't work out of the box. Took us a LONG time to figure it out why we always had a duplicate of all our SpriteAtlas, and if we unchecked "Include in Build", all our sprites would fail. But unchecking it is actually the way to go if you have anything in an AssetBundle referencing a SpriteAtlas. For some reason, something in a bundle is to dumb to figure out the atlas is packed with the game.

Code (CSharp):

1.         static AssetBundleManager()
2.         {
3.             SpriteAtlasManager.atlasRequested += LoadSpriteAtlas;
4.         }
5.  
6.         private static void LoadSpriteAtlas(string name, Action<SpriteAtlas> callback)
7.         {
8.             AssetBundle assetBundle = LoadAssetBundle<SpriteAtlas>();
9.             callback(assetBundle.LoadAsset<SpriteAtlas>(name));
10.         }

That's dumb. I hate Asset Bundles.

 

Well It is the question I want to ask ,I just rewrite some shader,and what I find is use assetbundle ,in memory profiler I can see more the one reference , and more memory

 

it's so big ,i need test build-in shader

 

this error :
Editor\StandardShaderGUI.cs(5,38): error CS0122: 'BuildTargetDiscovery' is inaccessible due to its protection level
what's wrong with that?

 

I'm getting the same issue - did anyone get this resolved?

 

I wish I could flag an asset as being part of the build, and it would always be in the build, and asset bundle, package or whatever would know it's there and not try to duplicate it.

 

I Drop Unity's StandardShader,Make New PBR_StandardShader,Less than Unity's StandardShader,But achieve the desired results.

 

More than an year later, this seem to still be a problem. If a bundle is marked as "Include in Build", I don't see why it can't look for references in the build itself instead of packing duplicated objects into the bundle.

Anyway, the Addressables system is pretty good to manage memory, so I still use it in my projects. Sadly I added it pretty late into the development so I had to figure something out to deal with this duplicated issue without rebuilding the whole structure of the game. As I stumbled with this post several times while searching the issue, I'm going to leave it here in case it helps someone else.

The idea is pretty simple: if we can't remove it from the bundle... Why don't remove it for the build?
I created a little snippet that you can set via inspector the original shader (in my case, Unity Standard) and a replacement (in my case, Unlit/Texture, as it is pretty small) and then it finds every material in scene and change its shader while keeping a reference to the original one.
The catch is that on Awake the script loads the original shader via Addressable System and change all the materials again. Works like a charm, no more duplicates!
As there might be some materials outside of the scene that still use the shader (it is good to remember that any reference to any object will be kept in memory) I also have set an alwaysInclude array, so you can manually set Materials to have the shader changed as well.



Long Story short:
Add it anywhere in the scene, set the Shader references, and everytime you are about to make a build call the GetAllMats function. If you want to go back, call GoBackAllMats.
It is pretty simple, but saved me days of work!

Code (CSharp):

1. using System.Collections.Generic;
2. using UnityEngine;
3. using UnityEngine.AddressableAssets;
4. using UnityEngine.ResourceManagement.AsyncOperations;
5. #if UNITY_EDITOR
6. using UnityEditor;
7. #endif
8.  
9. public class AddressableShaderForScene : MonoBehaviour
10. {
11.     //This array is set automatically by GetAllMats()
12.     [SerializeField] Material[] allMaterialsUsingShader;
13.     //You can set any material that is not in the scene and you want to change it still
14.     [SerializeField] Material[] alwaysInclude;
15.  
16.     //what is the correct shader to be loaded?
17.     [SerializeField] private AssetReference _addressableShader;
18.     AsyncOperationHandle<Shader> shaderOperation;
19.  
20.     void Awake()
21.     {
22.         if (_addressableShader.RuntimeKeyIsValid())
23.         {
24.             shaderOperation = Addressables.LoadAssetAsync<Shader>(_addressableShader);
25.             shaderOperation.Completed += Shader_Completed;
26.         }
27.     }
28.  
29.     private void Shader_Completed(AsyncOperationHandle<Shader> obj)
30.     {
31.         if (shaderOperation.Result != null)
32.         {
33.             foreach (Material m in allMaterialsUsingShader)
34.             {
35.                 m.shader = shaderOperation.Result;
36.             }
37.         }
38.     }
39.  
40.     private void OnDestroy()
41.     {
42.         if (shaderOperation.IsValid())
43.         {
44.             Addressables.Release(shaderOperation);
45.             if (_addressableShader.Asset != null)
46.                 _addressableShader.ReleaseAsset();
47.         }
48.     }
49.  
50. #if UNITY_EDITOR
51.     //Manually set, via inspector, the original shader and the temporary one, so we can fill the list automatically
52.     [SerializeField] Shader referenceShader;
53.     [SerializeField] Shader substituteShader;
54.  
55.  
56.     [ContextMenu("Get All Materials using shader")]
57.     public void GetAllMats()
58.     {
59.         List<Material> mats = new List<Material>();
60.         foreach(Renderer r in GetAllObjectsOnlyInScene<Renderer>())
61.         {
62.             foreach(Material m in r.sharedMaterials)
63.             {
64.                 if(m!= null && m.shader == referenceShader)
65.                 {
66.                     mats.Add(m);
67.                     m.shader = substituteShader;
68.                 }
69.             }
70.         }
71.         foreach(Material m in alwaysInclude)
72.         {
73.             mats.Add(m);
74.             m.shader = substituteShader;
75.         }
76.  
77.         allMaterialsUsingShader = mats.ToArray();
78.     }
79.  
80.     [ContextMenu("Go Back All Materials to the original shader")]
81.     public void GoBackAllMats()
82.     {
83.             foreach (Material m in allMaterialsUsingShader)
84.             {
85.                 if (m.shader == substituteShader)
86.                 {
87.                     m.shader = referenceShader;
88.                 }
89.             }
90.     }
91.  
92.     //Get all objects of any type, including inactive ones, but ignore if it is not in the current scene
93.     List<T> GetAllObjectsOnlyInScene<T>() where T : Component
94.     {
95.         List<T> objectsInScene = new List<T>();
96.  
97.         foreach (T go in Resources.FindObjectsOfTypeAll(typeof(T)) as T[])
98.         {
99.             if (!EditorUtility.IsPersistent(go.transform.root.gameObject) && !(go.gameObject.hideFlags == HideFlags.NotEditable || go.gameObject.hideFlags == HideFlags.HideAndDontSave))
100.                 objectsInScene.Add(go);
101.         }
102.  
103.         return objectsInScene;
104.     }
105.  
106. #endif
107. }
108.