Search Unity

ShaderLab taking 300mb of memory (IOS)

Discussion in 'General Graphics' started by ohunity, Feb 23, 2019.

  1. ohunity

    ohunity

    Joined:
    Jan 19, 2014
    Posts:
    617
    Not sure what this is, according to the profiler, "shaderlab" is taking up 300mb of memory on my game on iOS.

    We mainly just use the standard shader and a few customs shaders.

    What on earth is causing this?

    upload_2019-2-22_19-37-57.png
     
  2. ohunity

    ohunity

    Joined:
    Jan 19, 2014
    Posts:
    617
    Anyone have the slightest idea? Have searched everywhere online and can't find any details on it.
     
  3. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,176
    Which graphics API are you using?
     
  4. tiancaiwrk

    tiancaiwrk

    Joined:
    Nov 23, 2017
    Posts:
    35
    got the same issue, I just use standard shader to all buildings in the scene, about 500+ buildings
    and build the windows exe, the profiler shows the shaderlab has large memory usage (Unity2017.3)
     
  5. DaveCrowdStar

    DaveCrowdStar

    Joined:
    Nov 13, 2015
    Posts:
    13
    Same issue. Could this have something to do with asset bundles?
     
  6. jtate5

    jtate5

    Joined:
    Jan 13, 2014
    Posts:
    24
    Anyone find a resolution to this? I'm having the same issue with large ShaderLab memory on iOS.
     
  7. RobbyZ

    RobbyZ

    Joined:
    Apr 17, 2015
    Posts:
    36
    I'm having this problem on 2/3 of my platforms, including PC. Two use ~750MB shaderlab, one uses ~250MB shaderlab.

    I'm not targeting iOS though, so seems to be a cross-platform bug. Just comparing settings between them, I haven't yet been able to figure out why.
     
  8. matheus_inmotionvr

    matheus_inmotionvr

    Joined:
    Oct 3, 2018
    Posts:
    22
    Does anyone from Unity have a word on this? This is the second thread (here's the first one) about this same issue and we haven't heard anything about it yet. @aleksandrk ?

    I'm profiling on Android, using Unity 2019.2.15, OpenGLES 2, no SRP. The ShaderLab memory peak is only noticeable when the Standard shader is being used. If I use a Mobile shader instead, ShaderLab's memory drops drastically. If switch to another scene which doesn't use the Standard shader, the memory usage drops as well.
     
    Last edited: Feb 27, 2020
  9. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,176
    @matheusfantazm Standard shader has a lot of variants, so I would expect that it takes a lot of memory.
    Did anyone report a bug? :)
     
  10. richardkettlewell

    richardkettlewell

    Unity Technologies

    Joined:
    Sep 9, 2015
    Posts:
    1,888
  11. AcidArrow

    AcidArrow

    Joined:
    May 20, 2010
    Posts:
    8,431
    Can we somehow reliably exclude the Standard Shader from builds?
     
  12. richardkettlewell

    richardkettlewell

    Unity Technologies

    Joined:
    Sep 9, 2015
    Posts:
    1,888
    AcidArrow likes this.
  13. matheus_inmotionvr

    matheus_inmotionvr

    Joined:
    Oct 3, 2018
    Posts:
    22
    The excessive amount of memory used by ShaderLab was gone after I updated the project to Unity 2019.2.21. I couldn't find anything related to this issue on the release notes from 2019.2.16 until 2019.2.21, but I'm glad it's fixed now.

    Thank you so much for the quick response.

    I understand that the Standard shader has many variants, but I expected that to be registered in the profiler under "Assets>Shaders" (and it is). Although, I don't understand why ShaderLab would take 250Mb of memory, even after the scene load. To be honest, I still don't know exactly what falls under "Other>Graphics>ShaderLab" and I couldn't find any documentation or forum post explaining what's that exactly. For now the problem is solved, but if you could shed some light on that subject. I'd definitely appreciate it.
     
  14. AcidArrow

    AcidArrow

    Joined:
    May 20, 2010
    Posts:
    8,431
    richardkettlewell likes this.
  15. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,176
    @matheusfantazm This covers any memory allocations on the engine side that are marked as "ShaderLab", from shader name and its parameters to whatever else is required there.
     
    matheus_inmotionvr likes this.
  16. CerebralFrost

    CerebralFrost

    Joined:
    Mar 28, 2017
    Posts:
    12
    A little more than a year ago I ran into the same issue: https://answers.unity.com/questions...-grow.html?childToView=1703215#answer-1703215

    I still had this issue on 2018.4.11f1 - I'm not sure about newer versions.

    At the end of the day our problem specifically was shader duplication due to asset bundles - materials that share a shader, are explicitly referenced by bundles, and also implicitly referenced by bundles (i.e. 'pulled in' by other assets) will cause the shader to be duplicated - same shader, but the asset system seems to treat them as different shaders.

    My fix was to do the steps outlined in the answer - a combination of step #2 :"bundling all materials / shaders into a separate asset bundle and putting it into the streaming assets folder, and loading that asset bundle at runtime on demand" and step 0: Remove usage of any and all 'hidden' built-in shaders - replace with 'custom' copies of the shaders from Unity's repo, and make sure those shaders are included in the asset bundle from step 2 (including the standard shader if you are using it) - seems to be the key thing to preventing the duplication.
     
    chrismarch likes this.
  17. tinto

    tinto

    Joined:
    May 5, 2013
    Posts:
    11
    @aleksandrk - I think we all can understand that ShaderLab covers "any memory allocations on the engine side that are marked 'ShaderLab'. The issue is we have no insight into what "whatever else is required there" is.

    I have some questions:
    1. Does the memory covered under "ShaderLab" include only shaders that are being referenced by assets currently loaded, or does Unity cache shaders that might be referenced in the future (i.e. shaders that are included in the build but not yet referenced in memory through loaded assets)?
    2. Depending on the answer to #1, does ShaderLab continue to grow as more and more assets are loaded? Will periodic calls to UnloadUnusedAssets reduce usage?
    3. Does Unity store all variants of a shader in memory when that shader is referenced by an asset, or only the variants currently needed by that asset?
    4. What aspects of a Shader/Material are represented under "ShaderLab"? Presumably there's the shader source itself which will be needed to be supplied to OpenGL - but does the shader source hang around after the OpenGL program is created? If not, are we seeing the memory cost of the compiled OpenGL shader program under ShaderLab, or just the source code? What else?
    5. Is there any way to see what the makeup of "ShaderLab" memory is through the new memory profiler obtained through Unity's Package Manager? From what I can tell, we can see all the allocs, but with no way to determine what they are or where they came from.
    0.6GB for Shaders seems incredibly excessive, and I find Unity's near-silence on this subject a bit alarming - I've yet to work on a Unity project that doesn't suffer from excessive ShaderLab memory usage, but I am unable to find anything online that details what contributes to it or how to deal with it. Have we been doing something wrong on all those projects I've worked on? Probably! But a little more insight into what's going on would help us determine that, and help us avoid making the same mistakes.

    Can someone from Unity please chime in and provide some additional insight here. I know someone mentioned above that it was magically fixed in 2019.2.21, but there are many of us working on projects where we can't just upgrade at the drop of a hat, so it would be really useful for those of us still on 2018.x (or 2019 prior to 2019.2.21) to be given more details of what the issue(s) are and what was changed in that version to address them.

    Thanks
     
    chrismarch likes this.
  18. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,176
    @tinto it's a bit difficult to give a complete answer to a question "what's marked as ShaderLab memory" without spending a lot of time :)
    It makes it much easier when you ask more specific questions.

    1. No pre-loading is done without explicit calls to https://docs.unity3d.com/ScriptReference/Shader.WarmupAllShaders.html or https://docs.unity3d.com/ScriptReference/ShaderVariantCollection.WarmUp.html.
    2. Yes, memory will grow as new shaders are loaded when required by a material.
    3. All variants are loaded at once when loading a shader.
    4. It's difficult to answer this in full. Shader source/binary is unloaded after creating native GPU programs in most cases (we need to keep the source around for cases when we need multiple internal variants of the same shader, e.g. when a shader uses instancing on OpenGL/OpenGL ES). Shader Lab memory will still be used for keeping information around about e.g. shader properties such as uniforms, constant buffers and all other resources and their binding points, along with information identifying the particular variant. A compiled program on OpenGL is represented on the ShaderLab side by a single handle, and there's no way for us to query the memory used on the driver side.
    5. I don't know as I didn't use it yet :)
     
  19. tinto

    tinto

    Joined:
    May 5, 2013
    Posts:
    11
    @aleksandrk - thanks for the more detailed response and for answering the questions.

    For #5 - is there someone on your side you could ask or get to chime in? The built-in memory profiler literally only says, "ShaderLab X MB", which gives us very little to go on. The new memory profiler lists each allocation both on the native and managed sides, from what I understand. After a lot of digging, I have managed to find the allocations tagged as ShaderLab in there (since my first post) - but, they come with no annotations or anything to indicate what causes them or what they are comprised of - we effectively only see memory addresses and the size of the allocations.

    Is there any way at all currently (and by currently I mean in the 2018.x range of Unity) to get more information on what is contributing towards that "X" MB?
     
    chrismarch likes this.
  20. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,176
    I'll ask around :)
     
  21. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,176
    @tinto seems there's no way to find out what exactly comes from where.
    Are you using asset bundles by chance?
     
  22. tinto

    tinto

    Joined:
    May 5, 2013
    Posts:
    11
    We are, yeah. I saw mention in other threads of the same shader being referenced in multiple bundles counting as multiple shaders in the ShaderLab count. Is that what you're referring to? If so, what is the best way to address that?
     
  23. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,176
    You can use https://github.com/faelenor/asset-bundle-analyzer to figure out, what's in your asset bundles. It unpacks all the AB details and dumps them in a SQLite database. The database gives you a good view on Shader subprograms that can help diagnose which variants are being included.
    If you confirm that's really the case, I'll dig into how to solve that :)
     
  24. tinto

    tinto

    Joined:
    May 5, 2013
    Posts:
    11
    This is a very useful tool - thanks for pointing to it. I'll dig deeper with this and see what I can find.
     
    aleksandrk likes this.
  25. LexaFayte

    LexaFayte

    Joined:
    Apr 29, 2017
    Posts:
    4
    I'm having similar issue with ShaderLab on a project I'm part of; I've tried using that tool to analyze the asset bundles we're producing, but the script ends up running into a crash and can't parse through all the bundles (and unfortunately, those are the bundles that contain the helpful information I need).

    Curious if anything has changed to the way Asset Bundles are built in Unity over the last couple of years, since the script for this tool hasn't been updated in about 2 years :(

    Any information and help is super appreciated :)
     
  26. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,176
    The script just runs an internal Unity tool for asset bundles, so this shouldn't be a problem.
    I can let the tool developer know about this thread :)
     
  27. LexaFayte

    LexaFayte

    Joined:
    Apr 29, 2017
    Posts:
    4
    Thanks aleksandrk; that would be very helpful :)
    hopefully that can help shed some light on the issue.
     
  28. joshuacwilde

    joshuacwilde

    Joined:
    Feb 4, 2018
    Posts:
    338
    What if we aren't using asset bundles? Memory from ShaderLab is 237 MB in the profiler on iPhone
    @aleksandrk
     
  29. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,176
    @joshuacwilde in this case you can check the Editor log - it reports shader stats when a shader in total (all variants) adds up to over 1MB
     
  30. joshuacwilde

    joshuacwilde

    Joined:
    Feb 4, 2018
    Posts:
    338
    But the editor log is for disk space, not runtime size, right?
     
  31. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,176
    That's correct. At the same time, it contains info on the amount of variants that went into the build, which directly affects the runtime memory consumption.
     
  32. KospY

    KospY

    Joined:
    May 12, 2014
    Posts:
    143
    I got the same issue on Android and ShaderLab can take more than 800 MB on my project:

    It's the main issue I'm facing and it cause memory crashes as soon as I load more complex scenes.

    I made some progress recently and managed to reduce Shaderlab to 100kb (!) by simply stripping all shaders with an IPreprocessShaders script. Issue is all things are pink now :D, but at least it prove that it's related to stripping shaders.

    Currently I'm working toward finding the shader causing the memory to rise so high by guessing and eliminating them one by one. I already stripped some of them with a minor inpact (around 20mb down), but my next target is the default URP Lit, I have high hope on it as it take ages to create shader variant of it and as I suppose I should use simple lit for mobile.

    Strangely the shaders seem to still be visible in the profiler when I strip them, I don't know why, but when I removed them all they was all pink and ShaderLab was 100kb so it definitely work.

    I'm currently using Unity 2019.4.9, URP/VR with addressables and asset bundles.

    Edit: Ok so I just removed "Universal Render Pipeline/Lit" and ShaderLab is now 67mb :)
    Now I need to find a way to automatically change Lit to Simple lit for Android build...

    Edit 2: After changing all the Lit shaders to simple lit, ShaderLab is now around 375mb!
    Still pretty high, and if I take a look at the memory profiler (on windows, unfortunately doing a capture on Android freeze Unity), it seem that I can't do much better as it's now simple lit taking the most memory (if ShaderLab and shader memory is linked)

    I guess it's because there is many variants, but I don't know what I could do to reduce this.

    Hope this help others!
     
    Last edited: Sep 13, 2020
  33. joshuacwilde

    joshuacwilde

    Joined:
    Feb 4, 2018
    Posts:
    338
    Nice find, 375MB is still insane though! Easily over 1/3 the RAM you will be able to use on any 2GB iOS device, and those are still fairly recent. Let me know what else you find please! I had some luck stripping unused lightmap modes in the graphics settings.
     
  34. joshuacwilde

    joshuacwilde

    Joined:
    Feb 4, 2018
    Posts:
    338
    For future reference :

    I picked up Shader Control : https://assetstore.unity.com/packages/vfx/shaders/shader-control-74817 from the asset store and messed with the settings. I am still messing with them, but so far it has made a massive improvement in runtime shader memory usage.Would definitely recommend for anyone else dealing with this problem.The big thing is removing variants that you know you won't use
     
  35. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,176
    A small update on the topic: we recently changed the way ShaderLab memory is attributed and it now shows per shader asset in the profiler.
    This is (or will be available when the release is done) in 2021.2.0a2, 2021.1.0b3, 2020.2.3f1 and 2019.4.20f1 and newer.
     
  36. joshuacwilde

    joshuacwilde

    Joined:
    Feb 4, 2018
    Posts:
    338
    Wonderful news!! Anything that makes the memory profile better brightens my day.
     
  37. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,176
    @Serhii-Horun I would ask in the profiler subform about this. I think this should work, but I may be mistaken :)
     
    Serhii-Horun likes this.
  38. Serhii-Horun

    Serhii-Horun

    Joined:
    Apr 12, 2015
    Posts:
    125
    Sorry, I have deleted my question(hadn't seen that you had already answered), because it was not anymore needed. But you were so fast! Big Thanks!
     
    aleksandrk likes this.
  39. sacb0y

    sacb0y

    Joined:
    May 9, 2016
    Posts:
    377
    Any news on this? Same issue in android for me. near 300mb lit shader in 2021.2
     
  40. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,176
    @sacb0y The issue discussed in this thread is not about individual shaders taking a lot of memory. It's about having a possibility to figure out, which shaders take a lot of memory in the build.
     
  41. joshuacwilde

    joshuacwilde

    Joined:
    Feb 4, 2018
    Posts:
    338
    Also the problem seems to be that in some cases, all the shader variants get loaded in right up front. I think this is a bug because the unity documentation reports otherwise and it seems that sometimes it does work correctly. From what I have seen it appears to work correct on Android more than iOS.
     
  42. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,176
    All shader variant are always loaded right off. They are not sent immediately to the driver for compilation, though.
     
  43. joshuacwilde

    joshuacwilde

    Joined:
    Feb 4, 2018
    Posts:
    338
    Ah I see. Thanks for the explanation. But why?

    I mean it's not really a problem for us anymore because we wrote our own shader variant kind of system, but it was a huge problem before that. The game used to crash before even hitting something visible in game due to it loading gigabytes of shader memory.
     
  44. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,176
    I have no answer to this question :)
    Perhaps this was done this way to reduce hitching back in the day when 32 variants was enough for everyone.

    I'm working on exposing some control over memory usage when loading shader variants right now.
     
    sacb0y and joshuacwilde like this.
  45. sacb0y

    sacb0y

    Joined:
    May 9, 2016
    Posts:
    377
    I agree this might be some kind of bug, or at the very least a configuration issue associated with addressables.

    I did not have this problem until we started making builds with addressable fully implemented.
     
unityunity