Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Catching and diagnosing runtime shader compilation on Android

Discussion in 'General Graphics' started by DaveCowling, Mar 4, 2020.

  1. DaveCowling

    DaveCowling

    Joined:
    Oct 10, 2017
    Posts:
    7
    We prewarm all of our shaders at app boostrap via the standard preloaded shaders pathway.

    However, on Android (Quest) we see a nasty runtime hitch of up to around 250ms when a certain object is introduced into view for the first time after a fresh install (so compiled shader cache is clear). The cost is coming in the render thread from CreateGpuProgram, so clearly we are still experiencing compilation outside of the warmup phase.

    Is there a way to easily diagnose which shader/variant is the source of the compilation spike? Are there any known issues with the prewarm pathway on Android not actually doing what it is supposed to do?

    Thanks,

    Dave
     
    Last edited: Mar 4, 2020
  2. DaveCowling

    DaveCowling

    Joined:
    Oct 10, 2017
    Posts:
    7
    OK so I discovered that the callstack on the *main* thread in the Profiler window will display the shader details in the Shader.CreateGPUProgram call.

    Unfortunately however, now that we see the shader that is causing the hitch, we can confirm it is one that is *already present* in the shader variants file that we pre-warm, which suggests that Unity's pre-warm functionality is not working as intended...
     
    pravusjif likes this.
  3. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,983
    Hi!
    There's also a checkbox "Log shader compilation" in the Graphics Settings which you can enable. This will print any shader variants that are being compiled on your device.
    Which graphics API are you using?
     
    mahdi_jeddi and Wilfrid_unity like this.
  4. DaveCowling

    DaveCowling

    Joined:
    Oct 10, 2017
    Posts:
    7
    Quest, on Unity 2018.4.10f1. Android-ARM64 + OGLES 3.1 with Standard Render Pipeline.

    Hmm, I don't see the log shader compilation in the graphics settings screen

    The shaders that are getting compiled late are without question included in the shader variants. In the shaderVariants file I see:
    <shader name> ForwardBase UNITY_SINGLE_PASS_STEREO
    <shader name> ForwardBase INSTANCING_ON UNITY_SINGLE_PASS_STEREO

    And the runtime report of compilation (via the profiler) says:
    <shader name> STEREO_MULTIVIEW_ON
    <shader name> INSTANCING_ON STEREO_MULTIVIEW_ON
     
  5. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,983
    Right, it's not there for 2018.4.
    Does Standard Render Pipeline mean built-in?

    Well, this is a different shader variant at runtime, with STEREO_MULTIVIEW_ON instead of UNITY_SINGLE_PASS_STEREO, right?
     
  6. DaveCowling

    DaveCowling

    Joined:
    Oct 10, 2017
    Posts:
    7
    Ah, I assumed that was just different semantics for the same thing - with STEREO_MULTIVIEW_ON being the Android specific flag for single pass stereo, no?

    Interestingly, Unity does not provide me the option to add a variant with STEREO_MULTIVIEW_ON when attempting to edit the variants list. So how do I go about building a variants file with this variant present?

    Looking around, I notice this support thread which also seems to report the same issue... https://forum.unity.com/threads/pos...rogram-to-report-instigating-variants.758042/
    ...so having guidance from Unity on how to mitigate the problem would be great!
     
    Last edited: Mar 6, 2020
  7. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,983
    I will ask someone from XR to answer this :)
     
  8. DaveCowling

    DaveCowling

    Joined:
    Oct 10, 2017
    Posts:
    7
    That would be greatly appreciated, thank you!

    FWIW If I hand edit the variants file in a text editor, I can add the STEREO_MULTIVIEW_ON variant and this seems to solve the issue. Although, of course, hand editing this file is a pathologically bad workflow :)
     
    Last edited: Mar 8, 2020
  9. mpgholden

    mpgholden

    Joined:
    Aug 21, 2014
    Posts:
    38
    @aleksandrk any update on this? We're having this same issue using the Universal Render Pipeline on Oculus Quest. Shader variant files aren't catching variants that use STEREO_MULTIVIEW_ON at runtime.

    Weirdly, I noticed that
    Shader.IsKeywordEnabled(STEREO_MULTIVIEW_ON)
    returns false at runtime even though shaders with STEREO_MULTIVIEW_ON are getting compiled as objects come in view.
     
    DaveCowling likes this.
  10. emrys90

    emrys90

    Joined:
    Oct 14, 2013
    Posts:
    755
    @DaveCowling Did you figure out how to add variants with the STEREO_MULTIVIEW_ON keyword through the UI? I'm running into the same issue where it's not an option for adding to the ShaderVariantCollection.
     
  11. Serhii-Horun

    Serhii-Horun

    Joined:
    Apr 12, 2015
    Posts:
    151
    Hi @aleksandrk !

    We need to read and handle logs about shader compilation from our code(we have tons of scenes, and we need to collect all shader variants which are used by them). Any chance to do that?
    We tried UnityEngine.Application.logMessageReceived, but it doesn't catch shader compilation logging.

    Thanks!
     
  12. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,983
    @Serhii-Horun I suppose you could open the log afterwards and parse it manually.
    We have a feature request for a C# callback for this, but it's not planned for implementation yet.
     
  13. Serhii-Horun

    Serhii-Horun

    Joined:
    Apr 12, 2015
    Posts:
    151
    @aleksandrk Thank you!

    May I ask you also to help understand - how to covert data from parsing logs(shader, pass, stage and keywrods) to shader variant.

    When I try to match pass with PassType, I always get such kind of mistakes:

    ArgumentException: shader NatureManufacture/URP/Lit/Full Triplanar Cover pass type 13 does not have a shader snippet


    It happens even though shader contains proper LightMode. Looks like I miss something when I try to crate shader variant collection in runtime.

    Edit: I just rechecked - opened shader variation collection file, which I made inside Unity Editor, and it has exactly the same pass type 13 and the same keywords.. so, wondering why it throws exceptions in runtime
     
    Last edited: Nov 4, 2020
  14. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,983
    mahdi_jeddi likes this.
  15. Serhii-Horun

    Serhii-Horun

    Joined:
    Apr 12, 2015
    Posts:
    151
    @aleksandrk yeah, but any idea why we can get this exception when we are trying create shader variant in runtime:

    ArgumentException: shader NatureManufacture/URP/Lit/Full Triplanar Cover pass type 13 does not have a shader snippet


    Thank you! You are much faster then our "premium unity support":)
     
  16. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,983
    When exactly are you trying to add this variant? In play mode?

    Sounds like I should pay less attention to the forums ;)
     
  17. Serhii-Horun

    Serhii-Horun

    Joined:
    Apr 12, 2015
    Posts:
    151
    Yes, in play mode.

    We want to collect data about shader variants which is used in our scenes (we want to play scene, collect data from logs and save it in db). Next time, when we play same scenes, we can read data from DB(logs about compiled shaders) and precompile shaders before running the scene(by creating shader variants and shader collection in runtime based on this data and use shader variant collection -> Prewarm()).
     
  18. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,983
    This message means that it cannot find the source code of the pass.
    Do you have fallbacks and/or UsePass in this shader?
     
  19. Serhii-Horun

    Serhii-Horun

    Joined:
    Apr 12, 2015
    Posts:
    151
    Let me attach the compiled code and source shader file.
    What we did -> we run scene first time and noticed such logs:
    Compiled shader: Friend Factory/Nature/NatureManufacture/URP/Lit/Full Triplanar Cover, pass: Universal Forward, stage: fragment, keywords _ADDITIONAL_LIGHTS _MAIN_LIGHT_SHADOWS _MAIN_LIGHT_SHADOWS_CASCADE _USEDYNAMICCOVERTSTATICMASKF_ON

    Then we tried to create shader variant -> load this shader from Asset Bundle, add keywords and set PassType as 13(scriptible rendering pipeline, because we are using URP).
    So, basically when we see "Universal Forward" pass we convert it to PassType.ScriptableRenderPipeline.

    Probably the key is in loading shader from AssetBundle(but I prevented stripping shader variants during asset bundle build..and more over, we try to build shader variant based on logs, which were captured during compilation of shader from that bundle. So, this pass must exist in the shader from the bundle).

    Thank you
     

    Attached Files:

    Last edited: Nov 5, 2020
  20. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,983
    Oh, this one is a shader generated by shader graph.
    I'm not entirely sure, what the flow is there.
    It may well be that this has something to do with the asset bundle, but it might be that it's the way shader graph operates.
    I suppose this is the point where the premium support will have to step in ;)
     
  21. Serhii-Horun

    Serhii-Horun

    Joined:
    Apr 12, 2015
    Posts:
    151
    Ok, will force premium support team then! Anyway, Big thanks!!:)
     
    aleksandrk likes this.
  22. Serhii-Horun

    Serhii-Horun

    Joined:
    Apr 12, 2015
    Posts:
    151
    @aleksandrk That's me again. We have set ticket as "urgent", but haven't response yet. Probably you can help again with our new findings=)

    Basically something wrong happens only if we download shader from bundle, but not if we find it locally via Shader.Find(). We can't use Shader.Find in our main client app, because all shaders comes with bundles, and don't exist locally. So, looks like something going wrong during AssetBundle building process, probably it strips too much from the shader, event though we have added shader collection to the bundle with tested shader varian. I setup "keep all variants" in Graphics settings -> stripping shader.

    Any ideas?
    That's small test code
    Code (CSharp):
    1. public class CreateSVCTest : MonoBehaviour
    2. {
    3.     public bool LoadFromBundle = true;
    4.  
    5.     void Start()
    6.     {
    7.         //we get error if load from bundle, but don't get error if we load just from project by Shader.Find()
    8.         var shader = LoadFromBundle
    9.             ? LoadShaderFromBundle()
    10.             :Shader.Find("Friend Factory/Nature/NatureManufacture/URP/Lit/Full Triplanar Cover");
    11.         var variant = new ShaderVariantCollection.ShaderVariant(shader, PassType.ScriptableRenderPipeline,
    12.             "_ADDITIONAL_LIGHTS", "_MAIN_LIGHT_SHADOWS", "_MAIN_LIGHT_SHADOWS_CASCADE", "_USEDYNAMICCOVERTSTATICMASKF_ON");
    13.          
    14.         var svc = new ShaderVariantCollection();
    15.         svc.Add(variant);
    16.         svc.WarmUp();
    17.         Debug.Log("Loaded shader");
    18.     }
    19.  
    20.     private Shader LoadShaderFromBundle()
    21.     {
    22.         var assetBundlePath = Path.Combine(Application.dataPath.Replace("Assets", string.Empty),
    23.             "AssetBundles/iOS/shaderslibrary_nature");
    24.         Debug.Log(assetBundlePath);
    25.         var assetBundle = AssetBundle.LoadFromFile(assetBundlePath);
    26.         return assetBundle.LoadAsset<Shader>("Assets/ShadersLibrary/Shaders/Standard/Full Triplanar Cover.shader");
    27.     }
    28. }
     
  23. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,983
    Oh, ok, now I got it :)
    A shader loaded from an asset bundle does not have any source associated with it (would be wasteful to keep it around in the data that is loaded in the player).
    When building an asset bundle, only the compiled shader is stored.
    EDIT:
    This means it's impossible to add a shader loaded from an asset bundle to a shader variant collection.
     
    Serhii-Horun likes this.
  24. Serhii-Horun

    Serhii-Horun

    Joined:
    Apr 12, 2015
    Posts:
    151
    OMG!
    Thank you a lot for the help!!
     
  25. alexvector

    alexvector

    Joined:
    Oct 9, 2019
    Posts:
    26
    Is there a way to catch logs generated by "Log shader compilation" in a script?

    While testers are playing the game, I want to capture those messages and generate a report so that I can add them to the shader variant collection. I tried to add a callback to Application.logMessageReceivedThreaded, but it the "Compiled Shader" messages don't seem to go through there. I don't want the testers to run logcat.
    Thanks
     
  26. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,983
    @alexvector no, not at the moment. This is a log that only goes to the logcat on Android.
    You could parse the logs, as I suggested earlier.
     
    alexvector likes this.
  27. alexvector

    alexvector

    Joined:
    Oct 9, 2019
    Posts:
    26
    Sorry @aleksandrk, I had missed a question and your answer to basically the same question I asked...
    Thanks for your quick reply. Oh and that c# callback would be useful if it ever gets implemented.

    But after some parsing experiments, I have some log entries that I don't get how to create
    ShaderVariant for. Some have "Universal Forward", but that was mentioned in a post above so I guess I can use PassType.ScriptableRenderPipeline. Others however are <unnamed> or "Pass", so I can't find a matching PassType.

    For example:
    Compiled shader: Skybox/Cubemap, pass: <unnamed>, stage: all, keywords <no keywords>
    Compiled shader: FX/Flare, pass: <unnamed>, stage: all, keywords <no keywords>
    Compiled shader: Shader Graphs/VFX_basic, pass: Pass, stage: all, keywords <no keywords>
    Compiled shader: Legacy Shaders/Particles/Alpha Blended, pass: <unnamed>, stage: all, keywords FOG_EXP2

    Thanks
     
  28. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,983
    If the pass doesn't have a name, it will show up like that.
    I guess you'll have to check the shader manually.
     
  29. pw_prg_yinchao

    pw_prg_yinchao

    Joined:
    Feb 14, 2020
    Posts:
    18
    Hi @aleksandrk
    I found these logs do not go to logcat on Android. I am using Unity 2019.4.5f1, with adb command like:

    > adb logcat --pid=<pid> *:V

    Is the command incorrect or anything others wrong?

    On the other hand, btw, the "compiled shader" logs do go to Editor's console when the Editor connected to the Android device.
     
  30. ldewet-ct

    ldewet-ct

    Joined:
    Jul 8, 2019
    Posts:
    12
    How can I ensure that the shaders do not get added to assetbundles? I add scenes to the assetbundles, then I add all the shaders to their own assetbundle. Is there a way to say "dont include these shaders"? Or to force include the source for them?

    EDIT: I do have shadervariantcollections for all the shaders in a specific set of scenes, and it is warmed up at loading time for that scene.

    We have noticed that some variants do not get warmed up even though they are p[resent in the collection (like the above issue), so was wondering what we can do to resolve the spikes.
     
  31. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,983
    No idea. Please try searching or asking in the asset bundles subforum.
     
  32. pw_prg_yinchao

    pw_prg_yinchao

    Joined:
    Feb 14, 2020
    Posts:
    18
    If you just want to check the shaders and their variants in an exist bundle, there is a Unity official package AssetBundleBrower. But in fact I recommand the third-party app AssetStudio more.
     
  33. beautifulworld

    beautifulworld

    Joined:
    Jul 24, 2015
    Posts:
    1
    Hi aleksandrk, i just met this problem recently and followed the discussion above. I have used this "Log shader compilation" feature and logcat to collect the log, write some code to parse the logs and get my svc file and it has around 120+ shader variants in total. I warmed up this svc file in the beginning of my app and i can see around 120+ consecutive logs which are just the shader variants i collected in the svc file.

    upload_2022-11-9_21-25-8.png
    but as i kept running my app. log with same shader variant appeared again. for example, the log of "Gun Of Glory/FX/FogTerrain" with FORWARD pass appeared again later while i my app didn`t get restart of i shut it up and opened again.

    upload_2022-11-9_21-27-7.png

    so this makes me a little confused. did i generate the svc currently from the log or its normal that the log will appear both in the warmup and later when gpu load it?
     

    Attached Files:

  34. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,983
    @beautifulworld this means the variant is compiled twice - check that it doesn't get unloaded.