Search Unity

  1. Welcome to the Unity Forums! Please take the time to read our Code of Conduct to familiarize yourself with the forum rules and how to post constructively.
  2. Dismiss Notice

Bug (Case 1291333) Library ShaderCache grows in size over-time until AssetDatabase.Hotreload is incredib

Discussion in 'Asset Database' started by CianNoonan, Nov 13, 2020.

  1. CianNoonan

    CianNoonan

    Joined:
    May 19, 2017
    Posts:
    139
    I'm getting no response from QA on this issue. Hopefully someone from Unity can take a look.

    Library ShaderCache grows in size over-time until AssetDatabase.Hotreload is incredibly slow from searching the cache. This reproducibly destroys the performance of most import actions and flags a question as to why shaders are being touched when importing nonshader assets.
     
  2. Aras

    Aras

    Unity Technologies

    Joined:
    Nov 7, 2005
    Posts:
    4,770
    There's no "indexing" going on there. Yes there's no process in unity to automatically cleanup cache from old cached results of shaders that might no longer be in the project. Shader cache is not related to hot reload at all though, or any sort of "indexing". You can delete Library/ShaderCache any time you want yourself, that will just cause shaders to be recompiled on demand as needed.

    If you do have a repro of how AssetDatabase.Hotreload gets slower and slower over time (either due to shader cache or not), would love a bug report.

    Shaders are not touched when importing non-shader assets. However e.g. importing materials or FBX files does render their preview images, and for that it needs to use the shaders that are used by said assets, and for that it might need to compile said shaders. But that under normal conditions should not cause the cache size to grow out of control, or to lead to excessively long times.

    Anyway, bug report with all the details would be appreciated.
     
    JohngUK and CianNoonan like this.
  3. CianNoonan

    CianNoonan

    Joined:
    May 19, 2017
    Posts:
    139
    (Case 1291333) is the report for this issue. I've attached my shader cache dir, basically, you can see in the profiler when importing a shader that there is an incredibly slow set of calls to Shader.EditorLoadVariantFromCache under AssetDatabase.V2.Hotreload when the cache folder is present. Deleting the cache means the hotreload occurs almost instantly. This manifests as a long freeze of the editor when saving/importing a shader that does not occur when the cache is gone. In my case, the cache is a few GB so it could be that it's accumulated more than normally expected.

    When using ProcMon I can see that many FileIO operations occur on the ShaderCache Dir when importing scripts. This could be valid as maybe they all need to be reloaded after a Domain Reload?

    Attached are two screenshots of the profiler.
     

    Attached Files:

  4. CianNoonan

    CianNoonan

    Joined:
    May 19, 2017
    Posts:
    139
  5. CianNoonan

    CianNoonan

    Joined:
    May 19, 2017
    Posts:
    139
    Some of the members of our team are getting freezes lasting over 30 seconds when saving trivial changes to shader files. These extreme cases are also resolved by deleting the cache
     
  6. CianNoonan

    CianNoonan

    Joined:
    May 19, 2017
    Posts:
    139
    Another note is that originally this was an ADBv1 project so there could be an upgrade related issue here?
     
  7. CianNoonan

    CianNoonan

    Joined:
    May 19, 2017
    Posts:
    139
  8. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,844
    Hi, thank you for the bug report. We'll take a look as soon as QA are finished processing it - I poked them to speed it up.
     
    CianNoonan likes this.
  9. CianNoonan

    CianNoonan

    Joined:
    May 19, 2017
    Posts:
    139
    So there is a missing component to this. AmplifyShaderEditor kicks off the AssetDatabase.Hotreload call, so the underlying issue in within unity but is only triggered by ASE. I'll add this to the ticket and give QA the ASE package in a new proj but it does make me a bit worried that this issue might fall through the cracks between Unity and ASE
     
  10. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,844
    Well, I don't think AssetDatabase.Hotreload should be affected by the size of shader cache.
     
  11. CianNoonan

    CianNoonan

    Joined:
    May 19, 2017
    Posts:
    139
    You can see in the screenshots further up in the thread that AssetDatabase.V2.Hotreload has Shader.EditorLoadVariantFromCache as a child. EditorLoadVariantFromCache does a large amount of File.Open when the shader cache is large.

    Deleting the shader cache will reduce the overhead of EditorLoadVariantFromCache as there are many fewer File.Open's.

    Is it possible that you're looking at more recent implementations that have removed the EditorLoadVariantFromCache call from Hotreload?
     
  12. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,844
    When you clear the cache, does it load less shaders?
    The screenshot above says it's loading over 15K variants.
     
  13. CianNoonan

    CianNoonan

    Joined:
    May 19, 2017
    Posts:
    139
    No its the same runtime case. We dont change anything to do with the shaders or rendering themselves. I suspect its opening 15k files because its traversing the entire directory. Each previous iteration of the shader is left in the directory when the shader is recompiled, so the majority of those are likely just dead old versions.
     
  14. Peter77

    Peter77

    QA Jesus

    Joined:
    Jun 12, 2013
    Posts:
    6,438
    I think he already answered it:
     
  15. CianNoonan

    CianNoonan

    Joined:
    May 19, 2017
    Posts:
    139
    Yes it is true that the overhead is reduced but the actual variants in use is identical
     
  16. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,844
    Can you please reply on the bug report then?
    This should be investigated.
     
  17. CianNoonan

    CianNoonan

    Joined:
    May 19, 2017
    Posts:
    139
    Sure thing, I'll link this thread in too
     
  18. oleg_v

    oleg_v

    Joined:
    Nov 10, 2017
    Posts:
    68
    Probably similar case: unity started to hang everywhere (enter playmode, exit, recompile c# scripts, recompile shaders) without any recent import of assets.
    Found with procmon the most time unity spent on Library/ShaderCache (~40k files), i clean it up. Layouts are crashed one time, but everything started to work w/o any delays :)
     
  19. ROBYER1

    ROBYER1

    Joined:
    Oct 9, 2015
    Posts:
    1,444
    Had my shader cache swell to over 8gb in a project just increasing in size with every build for an IOS app!

    It seems to be connected to a massive hang when opening the shader dropdown on a material to change shader, can take up to 2 mins to open that dropdown sometimes.

    Using Unity 2020.3.0f1
     
  20. Aras

    Aras

    Unity Technologies

    Joined:
    Nov 7, 2005
    Posts:
    4,770
    Yes, and it's fixed in 2020.3.1 https://unity3d.com/unity/whats-new/2020.3.1 "Shaders: Fixed issue where the ShaderCache.db was growing in size after each build"
     
    ROBYER1 likes this.
  21. ROBYER1

    ROBYER1

    Joined:
    Oct 9, 2015
    Posts:
    1,444
    Great I'll bump to that then cheers
     
  22. JannickL

    JannickL

    Joined:
    May 18, 2016
    Posts:
    78
    2021.3.0f1 and the issue still appears BUT i fixed it by moving the project to my HDD (instead of SSD)

    Hope it helps anybody