Search Unity

Compiled shaders aren't cached? Why do they recompile every time I switch platform?

Discussion in 'Universal Render Pipeline' started by crdmrn, Jul 8, 2021.

  1. crdmrn

    crdmrn

    Joined:
    Dec 24, 2013
    Posts:
    152
    Is this the case? Why?
    It really makes no sense and an insane amount of time is lost whenever one needs to switch platform.
    Also, why doesn't the URP Standard Shader come with all the variants already compiled?
    It can't be edited anyway, and that's by far the shader that takes the longest to compile..
    :(
     
  2. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,014
    Compiled variants are cached. When you switch platforms, some inputs change, so we need to compile a different variant. When you switch to this platform next time, it will not compile the same thing again.

    Because there are millions of variants. They would eat a lot of disk space.
     
  3. crdmrn

    crdmrn

    Joined:
    Dec 24, 2013
    Posts:
    152
    Except they do, though. I'm switching between Windows and a console that I can't mention because of NDA, and without changing anything, shaders recompile every time.
    I am perfectly aware of that, but better get 10gb of URP Standard Lit cache, than having to recompile it every time for 2+ hours tbh. There could at least be the option when installing Unity.
     
  4. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,014
    Are you sure they really are recompiled? Perhaps it's reading from the cache? It can still be a long operation if the drive you have the project on is slow.
     
  5. crdmrn

    crdmrn

    Joined:
    Dec 24, 2013
    Posts:
    152
    I have a PCIe SSD disk, reading/writing is not a problem.
    To make it more clear this happens, without changing anything in the project other than the build platform:

    - 1st build Windows -> compiles everything (3+ hours)
    - 2nd build Windows -> reads the cache (~10 minutes)
    - switch platform
    - 1st build Console -> compiles everything (3+ hours)
    - 2nd build Console -> reads the cache (~10 minutes)
    - switch platform
    - 1st build Windows -> compiles everything AGAIN (3+ hours)

    The last thing it's what's annoying, it seems that the cache is cleared every time i switch platform :/
     
  6. crdmrn

    crdmrn

    Joined:
    Dec 24, 2013
    Posts:
    152
    image_2021-07-09_131735.png
    And yes, I am sure ^^"
     
  7. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,014
    OK :)
    I'm pretty sure the cache is not cleared, at least by Unity.
    Which version of Unity are you on?
     
  8. crdmrn

    crdmrn

    Joined:
    Dec 24, 2013
    Posts:
    152
    2021.1.13f1
    I am not sure if it's Unity, or if its the integration for the console provided by it's manufacturer :S
    But compile times seem to have ramped up (for Windows builds too) since we switched from URP 10 to 11 D:
     
  9. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,014
    If you're on 2021.1.13f1, you can check this in the Editor log. When you build your project, it shows stats for each pass compiled, per stage: how many variants it got from the disk cache, how many from remote cache (if it's active), and how many were actually compiled.
     
  10. roundyyy

    roundyyy

    Joined:
    Dec 23, 2019
    Posts:
    112
    I will just join this topic instead of creating new one. Facing similar problem, but without switching platform.
    So on each next build nothing is used form cache. Cache is created, but shader are recompiled again. I make build without any changes, just click build again and it doesn't make any difference. Still 1hour. What are the most common settings I could check? I've tried everything what I found here in forums, but still no difference. Recreated library etc.
    Unity 2019.4.35, builtin rp
     
  11. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,014
    The progress bar doesn't specify whether it's fetching the variants from the cache or compiles them. Is your project on a SSD or on a HDD?
     
  12. roundyyy

    roundyyy

    Joined:
    Dec 23, 2019
    Posts:
    112
    It is on PCiex4 ssd, so very fast, month ago it was exporting in 5-10 minutes after 1st initial build, now it's about 1 hour and it recompiles all shaders always , I removed all not used assets and shaders, stripped unneeded keywords, instancing variants strip unused, always included shaders only few basic ones for UI and diffues (this was always in project like this). Optimize mesh data disabled, deferred disabled, tier settings default, shaders set to current platform only, static, dynamic batching enabled, lightmap encoding normal quality, graphic jobs enabled, prebake collision meshes enabled, managed stripping level disabled not sure what else to list.

    Forgot to add that files in shader cache are modified/created each build, by looking at their date.
    It doesn't matter if I remove this folder or not, build time stays the same.
    Last time shader cache was 40GB
     
    Last edited: Feb 10, 2022
  13. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,014
    Can you please send us a bug report then?
    I'd like to take a look at what's happening there.
     
  14. hineworks

    hineworks

    Joined:
    Nov 1, 2021
    Posts:
    4
    Just submitted a bug report. Here are the steps to reproduce:
    - Create a new project in Unity LTS 2021.3.1f1 with the "3D (URP)" template.
    - Copy the manifest.json quoted below to the Packages directory. This is the manifest.json from the Unity template, plus the "com.atteneder.gltfast (4.7.0)" and its scoped registry.
    - Re-open the project to install the glTFast package
    - In Project Settings > Graphics > Always Included Shaders, add the 3 shaders from Packages/glTFast/Runtime/Shader
    - In build settings, switch platform to iOS
    - Leave all settings at default and run a build. It will take 40-50 minutes on a fast machine to compile all the shaders.
    - When it has finished, run the build again. It will again take 40-50 minutes to compile all the shaders (ignores cached shaders).

    Bug report link: https://fogbugz.unity3d.com/default.asp?1425665_8e8453tjnsmolq17

    Related issues:
    https://github.com/atteneder/glTFast/issues/269

    manifest.json
    Code (JavaScript):
    1. {
    2.   "scopedRegistries": [
    3.     {
    4.       "name": "OpenUPM",
    5.       "url": "https://package.openupm.com",
    6.       "scopes": [
    7.         "com.atteneder"
    8.       ]
    9.     }
    10.   ],
    11.   "dependencies": {
    12.     "com.atteneder.gltfast": "4.7.0",
    13.     "com.unity.collab-proxy": "1.15.15",
    14.     "com.unity.ide.rider": "3.0.13",
    15.     "com.unity.ide.visualstudio": "2.0.14",
    16.     "com.unity.ide.vscode": "1.2.5",
    17.     "com.unity.render-pipelines.universal": "12.1.6",
    18.     "com.unity.test-framework": "1.1.31",
    19.     "com.unity.textmeshpro": "3.0.6",
    20.     "com.unity.timeline": "1.6.4",
    21.     "com.unity.ugui": "1.0.0",
    22.     "com.unity.visualscripting": "1.7.6",
    23.     "com.unity.modules.ai": "1.0.0",
    24.     "com.unity.modules.androidjni": "1.0.0",
    25.     "com.unity.modules.animation": "1.0.0",
    26.     "com.unity.modules.assetbundle": "1.0.0",
    27.     "com.unity.modules.audio": "1.0.0",
    28.     "com.unity.modules.cloth": "1.0.0",
    29.     "com.unity.modules.director": "1.0.0",
    30.     "com.unity.modules.imageconversion": "1.0.0",
    31.     "com.unity.modules.imgui": "1.0.0",
    32.     "com.unity.modules.jsonserialize": "1.0.0",
    33.     "com.unity.modules.particlesystem": "1.0.0",
    34.     "com.unity.modules.physics": "1.0.0",
    35.     "com.unity.modules.physics2d": "1.0.0",
    36.     "com.unity.modules.screencapture": "1.0.0",
    37.     "com.unity.modules.terrain": "1.0.0",
    38.     "com.unity.modules.terrainphysics": "1.0.0",
    39.     "com.unity.modules.tilemap": "1.0.0",
    40.     "com.unity.modules.ui": "1.0.0",
    41.     "com.unity.modules.uielements": "1.0.0",
    42.     "com.unity.modules.umbra": "1.0.0",
    43.     "com.unity.modules.unityanalytics": "1.0.0",
    44.     "com.unity.modules.unitywebrequest": "1.0.0",
    45.     "com.unity.modules.unitywebrequestassetbundle": "1.0.0",
    46.     "com.unity.modules.unitywebrequestaudio": "1.0.0",
    47.     "com.unity.modules.unitywebrequesttexture": "1.0.0",
    48.     "com.unity.modules.unitywebrequestwww": "1.0.0",
    49.     "com.unity.modules.vehicles": "1.0.0",
    50.     "com.unity.modules.video": "1.0.0",
    51.     "com.unity.modules.vr": "1.0.0",
    52.     "com.unity.modules.wind": "1.0.0",
    53.     "com.unity.modules.xr": "1.0.0"
    54.   }
    55. }
    56.  
     
    Last edited: May 6, 2022
  15. funkyCoty

    funkyCoty

    Joined:
    May 22, 2018
    Posts:
    726
    There's definitely some sort of bug with the shader cache lately. I'm on 2021.2.19f1, and it seems it randomly gets cleared. Sometimes even just changing the editor game view's fullscreen state seems to clear it. Sometimes a build clears it. Sometimes asset bundle rebuilds clear it.
     
  16. Nikolai-Games

    Nikolai-Games

    Joined:
    Sep 14, 2019
    Posts:
    6
    I also wonder about shader recompilation ... although iam repeating same build.
     
  17. hineworks

    hineworks

    Joined:
    Nov 1, 2021
    Posts:
    4
    MKGameDev likes this.
  18. SpockBauru

    SpockBauru

    Joined:
    Apr 12, 2021
    Posts:
    23
    Also happening on Android. If you start on Windows and build, go to Android and build, and them back to Windows, it will recompile everything again. It recompiles every time you switch a platform.
     
  19. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,014
    On my machine it takes the compiled variants from the cache. Can you please check the Editor log? It should have the data per shader on how many variants were actually compiled and how many were fetched from the cache.
     
  20. SpockBauru

    SpockBauru

    Joined:
    Apr 12, 2021
    Posts:
    23
    Hello aleksandrk, sorry for the delay, but since each test takes half hour it took longer than expected.

    After several headaches, I tried to update the editor to 2021.3.7, did an Android build, switched to Windows and did a build, and finally went back to Android and built again.

    The new version solved all issues: The first build takes half hour for each platform, but the following ones takes just seconds.

    I suppose that was something related to the workaround for fixing the Android build issue on 2021.3.6 that I took from this post: https://forum.unity.com/threads/cant-build-for-android.1306098/, maybe I missed some step. Fortunately, the issue was solved in 2021.3.7.
     
    aleksandrk likes this.
  21. nasos_333

    nasos_333

    Joined:
    Feb 13, 2013
    Posts:
    13,348
    I have few shaders that make image effects in URP, put in Resources folder so are included in the project

    When insert them in a project, i can see the clouds directly without any delay or compilation time of the shaders.

    In this case, why when i build the game it takes hours to compile each of those shaders ? Should it not be instant, given editor is in Windows and i build for windows.

    Why need to do something extra versus what i already have that shows the effects fine in editor ?

    Thanks in advance for any insight on this issue
     
  22. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,014
    @nasos_333 This is because Editor compiles variants it needs on the fly. Shader compiler is only available in the Editor, so we can't do that in the player, and we precompile the variants that the player may need.
     
  23. nasos_333

    nasos_333

    Joined:
    Feb 13, 2013
    Posts:
    13,348
    hi, thanks for the clarification, i see

    Though if there was a way to tell the system we only need the platform the editor runs in and given we already have compiled this version since it previews in editor, could have in theory an instant build of the game, with zero wait for shader compilation

    I am interested in this as compiling my two cloud shaders for a windows build can take a full day on my laptop, while i already see the clouds in the editor within a few seconds after import the asset in a new project.

    The compilation time difference for the exact same effect is so massive than make me think it would be great to have a "fast build" mode that enables only the platform used in the editor for shaders, which is already compiled.

    That would increase iteration times of testing a complex feature in builds so vastly faster and better.
     
  24. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,014
    Editor compiles only the variants it needs, not all potentially needed variants. The player probably doesn't use all variants that go into the build, but we cannot determine that reliably at build time.
     
    nasos_333 likes this.
  25. nasos_333

    nasos_333

    Joined:
    Feb 13, 2013
    Posts:
    13,348
    I see, thanks