Search Unity

  1. Good news ✨ We have more Unite Now videos available for you to watch on-demand! Come check them out and ask our experts any questions!
    Dismiss Notice

[Asset pipeline v2] Script reimport performance

Discussion in '2019.3 Beta' started by Seb-1814, Nov 27, 2019.

  1. Seb-1814

    Seb-1814

    Joined:
    Nov 23, 2013
    Posts:
    22
    Hi, I upgraded a project to 2019.3b12 and asset pipeline v2. I'm using a bunch of unity packages like Entities, Jobs and Burst.

    Changing one line in a script and going back to unity takes 9 seconds (it was closer to 5-6 seconds in unity 2019.2). Looking at the profiler, I feel like there is a lot of room for optimization. For example ScanAssemblies in EndReloadAssembly spends 375ms scanning all assemblies, of which only 5.5ms is spend on my game assembly, the only one that actually changed. Most of the time is spent scanning UnityEngine and Unity.Entities, and those basically never change in a normal workflow. Couldn't result of the scanning be kept around between reloads somehow?

    That exemple aside, is the performance of scripts reloading being worked on, and can we expect improvements? With unity moving more and more of its engine modules over c#, it becomes especially important that the reimport time of c# code is kept low imo.
     
  2. Seb-1814

    Seb-1814

    Joined:
    Nov 23, 2013
    Posts:
    22
  3. AcidArrow

    AcidArrow

    Joined:
    May 20, 2010
    Posts:
    7,071
    Hey, could be worth to submit a bug report with your findings.
     
  4. Unity_Javier

    Unity_Javier

    Unity Technologies

    Joined:
    Mar 7, 2018
    Posts:
    37
    Hi @Seb-1814

    This is a very valuable capture of the profiler. Thanks for showing it.

    Out of curiosity, how many Assets (roughly) have you got in your project?

    There are some issues which we're addressing right now with regard to the performance (i.e. batching asset creation regressed severely, and we're working on a fix). It would be helpful to at least know how your Assets are distributed.

    I made a Gist which you should just be able to run: https://gist.github.com/Unity-Javier/670752cb02b4c92e451665631fc2ea7d

    And it'll output how many of each asset type you have in your project, so we can see if its more Script heavy or not.

    Additionally, a Bug report could help us see what setup you have more precisely.

    Looking forward to your reply!
     
    phobos2077 and Peter77 like this.
  5. Seb-1814

    Seb-1814

    Joined:
    Nov 23, 2013
    Posts:
    22
    Hi Javier,

    Sure, happy to help where I can

    Our project has quite a lot of heavy textures. I'm curious as to what impact it could have. I'm guessing it shouldn't matter for script compilation and reloading, but could impact the 1.4 sec spent in AssetDatabase.V2.PostProcessAllAssets?

    I'm guessing our project is also rather script heavy, I don't know what's a normal amount of script. I'm also not sure what proportions of scripts are from packages like Entities versus our own project.

    I saw that unity 2019.3f1 was released over night so I switched to it. It seems to be a little faster (around 8 seconds). Or it could be due to a change my method of testing or environnement, I'm not sure. If you need I can send another profile.

    The script you provided ran with an error:
    NullReferenceException: Object reference not set to an instance of an object
    AssetDistribution.GetAssetDistribution () (at Assets/Editor/AssetsDistribution.cs:18)

    By stepping in with the debugger I found that AssetImporter.GetAtPath() returns null for one of the paths: "Packages/com.unity.entities/Unity.Entities.Editor/Mode/default.mode"

    So I added a null-check
    Code (CSharp):
    1.  
    2.             var curImporter = AssetImporter.GetAtPath(assetPath);
    3.             if (curImporter == null) {
    4.                 Debug.LogWarning($"No importer for asset at path {assetPath}");
    5.                 continue;
    6.             }
    7.  
    Here is what the script logged:

    Code (CSharp):
    1.  
    2. UnityEditor.AssetImporter,971
    3. UnityEditor.MonoImporter,2675
    4. UnityEditor.PrefabImporter,85
    5. UnityEditor.TextureImporter,921
    6. UnityEditor.TrueTypeFontImporter,27
    7. UnityEditor.TextScriptImporter,93
    8. UnityEditor.ShaderImporter,102
    9. UnityEditor.ModelImporter,9
    10. Unity.VectorGraphics.Editor.SVGImporter,8
    11. UnityEditorInternal.PackageManifestImporter,56
    12. UnityEditorInternal.AssemblyDefinitionImporter,109
    13. UnityEditor.StyleSheets.StyleSheetImporter,24
    14. UnityEditor.PluginImporter,11
    15. Unity.Build.BuildPipelineScriptedImporter,2
    16. UnityEditor.UIElements.UIElementsViewImporter,11
    17. UnityEditor.AudioImporter,10
    18. UnityEditorInternal.AssemblyDefinitionReferenceImporter,1
    19. Total assets: 5116
    20. UnityEngine.Debug:Log(Object)
    21. AssetDistribution:GetAssetDistribution() (at Assets/Editor/AssetsDistribution.cs:40)
    22.  
    I didn't make a bug report right away, because I wasn't sure if this would be treated like a bug, and reporting bug has proven quite time consuming for me in the past. Also our project is pretty big, would it still be useful for you if I made a bug report without attaching all of the project somehow (project might be in a broken state then) ?
     
    goncalo-vasconcelos and Peter77 like this.
  6. Seb-1814

    Seb-1814

    Joined:
    Nov 23, 2013
    Posts:
    22
    (or are you mainly interested in my machine specs, info on the project like package used and profiler saves, in which case I can submit a bug without the project but with this info?)
     
  7. Unity_Javier

    Unity_Javier

    Unity Technologies

    Joined:
    Mar 7, 2018
    Posts:
    37
    Hey, thanks for the reply.

    I got something similar (in terms of time spent inside a Refresh) by installing 3 packages.

    However, would it be possible for you to share a deep profiler capture, so I can reach out to the teams where the time is being spent at?

    It looks like its not just the AssetDatabase, but the usage of the APIs that is slowing things down.
     
  8. Seb-1814

    Seb-1814

    Joined:
    Nov 23, 2013
    Posts:
    22
    Glad you could reproduce a similar refresh time with just unity packages. It seems there is sill a lot of work to do before reaching the 500ms goal mentioned at unite some time ago ;)

    Here is a link to a deep profile, reload happens at frame number 658.
    https://drive.google.com/file/d/1-Fj7XBLPeMbsgyh_UZTU7cz8sMl4VpEE/view?usp=sharing

    Thanks for the support! I really hope it will help, I'm happy to provide more details if needed
     
    interpol_kun and Lars-Steenhoff like this.
  9. Unity_Javier

    Unity_Javier

    Unity Technologies

    Joined:
    Mar 7, 2018
    Posts:
    37
    Hey, so we tried to open the capture and it didn't work. It looks like the data is incomplete:

    The file size we got is 165742256 bytes, but it should be 2-3x larger according to some of the data in it.

    Could you try uploading it again, and checking if the download opens on your machine?
     
  10. Seb-1814

    Seb-1814

    Joined:
    Nov 23, 2013
    Posts:
    22
    Indeed, I can't load the save either. I did another one and couldn't load it either, it seems to me like the profiler has a bug and doesn't save big profiles correctly. I get this error message when loading one of the deep profiles:

    Screenshot 2019-11-30 at 16.37.09.png
    So I guess that's another bug to log ¯\_(ツ)_/¯
    But in the meantime, since you managed to reproduce the high reload times with a project of your own, maybe you can pass it to the relevant teams, and they can do a deep profile by themselves?
     
    alexeyzakharov likes this.
  11. spiderspy

    spiderspy

    Joined:
    Aug 1, 2013
    Posts:
    10
    I just want to add that I got this too and I cannot efficently edit assets because it wants to autosave and reimport everything that is related, which is the exact opposite what you would expect from nested prefabs, should just be links right? Also why does it have to block anyone's mainthread and usage at all? Also would be nice if it was a package so that we can fix stuff for ourselves.

    Also the Asset pipeline v2 F***s things up for my special editor windows. It, for some god damn reason uses editor window stuff when it import it during a save which apparently causes one of my lists of script objects top be full of null during one stage which is quite weird. Which triggers some of my editor window sanity checks and set things to false because it is null, even though it isnt actually null.

    //Rant

    Edit:
    Wanted to add my current project size

    Code (CSharp):
    1.  
    2. UnityEditor.AssetImporter,2903
    3. UnityEditor.MonoImporter,3397
    4. UnityEditor.TextureImporter,5984
    5. UnityEditorInternal.AssemblyDefinitionImporter,79
    6. UnityEditor.ModelImporter,125
    7. UnityEditor.PrefabImporter,667
    8. UnityEditor.TrueTypeFontImporter,28
    9. UnityEditor.AudioImporter,203
    10. UnityEditor.PluginImporter,40
    11. UnityEditor.TextScriptImporter,128
    12. UnityEditor.ShaderImporter,229
    13. UnityEditorInternal.PackageManifestImporter,59
    14. UnityEditor.StyleSheets.StyleSheetImporter,117
    15. UnityEditor.UIElements.UIElementsViewImporter,59
    16. UnityEditor.ShaderGraph.ShaderGraphImporter,6
    17. MLAgents.DemonstrationImporter,1
    18. Barracuda.NNModelImporter,8
    19. Total assets: 14033
    Edit 2: also I cannot properly debug the save because unity crashes ect, but takes like 24 seconds.
     
    Last edited: Dec 3, 2019
    Seb-1814 and FcsVorfeed like this.
  12. phobos2077

    phobos2077

    Joined:
    Feb 10, 2018
    Posts:
    224
    If only Unity tried making just one full AA+ commercial game on their own, they would've catch 90% of annoying issues and regressions like this, without being surprised every time something like this is brought up by users.
     
  13. laeusnb

    laeusnb

    Joined:
    Jun 10, 2014
    Posts:
    76
    A million times this. The more you structure things using nested prefabs, the more abysmal their usability gets due to all the progress bars anytime you change a variable with autosave (or you disable autosave and just have to grit your teeth every time you're about to save).
     
    ImpossibleRobert likes this.
  14. spiderspy

    spiderspy

    Joined:
    Aug 1, 2013
    Posts:
    10
    Okay I managed to save the dump file at home but it turns out to be 3.3 GIGABYTE, so I took some snippets that I hope will aid you Unity_Javier.

    The context for this save is a fairly empty scene (where we will generate our procedural level) the scene is only about 740kb large so it is suprising that it takes 12-24seconds to save even though in this case I have not even changed anything at all but just pressed save.

    I hope you will solve this, thanks.

    topDuring save.PNG topDuring save2.PNG topDuring save3.PNG
     
  15. Unity_Javier

    Unity_Javier

    Unity Technologies

    Joined:
    Mar 7, 2018
    Posts:
    37
    Hey @spiderspy,
    Thanks for the images. This looks like the Raw Hierarchy view, and that is also very valuable to put the order of operations into perspective.

    We have spent the entire week looking at the performance issues (and will continue doing so for the near future) and have reached out to a number of teams across Unity who are investigating how to fix the issues, and how to make sure they get into 2019.3 so that you all don't have to get hit with these regressions.

    Even if the capture is 3.3GB, a .zip of it would be good to have, since the profiler data compresses really well (i.e. a 3.08GB profile compressed down to 148MB for an investigation related to this).

    You can DM it to me as well if you prefer, or put it in a bug so we get some infrastructure around it and track the progress.

    Either way, thanks for sharing all this info with us! The more info we have the more scenarios we can cover.
     
    optimise, Seb-1814, spiderspy and 5 others like this.
  16. Vitaly_Unity

    Vitaly_Unity

    Unity Technologies

    Joined:
    Sep 4, 2013
    Posts:
    8
    Hi @spiderspy,
    I am Vitaly and I work in the same team with Javier. We have made a lot of performance fixes on AssetPipeline side and are still help other teams to improve performance on their side. That said, we looked at your profiler information (you sent to Javier) and noticed 2 seconds in UnityEditor.Rendering.DebugWindow, we reached to the team owning this, but unfortunately they can't really say what is happening there... and asking for more info.
    Could you please give more context on the project/scene you recorded profiler info with? or ideally file a bug report and attach project (stripped scene) and post bug number here or DM me - that will help us a lot in identifying and fixing the problem
    Thanks in advance :)
     
  17. spiderspy

    spiderspy

    Joined:
    Aug 1, 2013
    Posts:
    10
    Could you explain to me what UnityEditor.Rendering.DebugWindow is? and I will look into it. @VitalyV

    Also concerning context, is that I am saving a small scene that will procedurally generate a larger scene world, but at the point of savig it is a fairly small scene, less than 1mb. We use the nested prefabs a lot, it is a 2D based project.

    Unfortunately I cannot really share the project in itself.
     
    Last edited: Jan 7, 2020
  18. ShockpointGames

    ShockpointGames

    Joined:
    Jul 19, 2018
    Posts:
    26
    Just like to point out that I've noticed these incredibly long script compile and enter play mode times with brand new projects. In 2019.3.0f4, I created a new URP project and without any modifications to the project I'm getting 20+ second play mode entry times. Here's what the profiler produces:

    upload_2020-1-8_22-17-0.png

    Just like to add how odd it is that it's using ~1.8GB GC allocation for a project that is ~1GB in size.
     
    Last edited: Jan 8, 2020
  19. ShockpointGames

    ShockpointGames

    Joined:
    Jul 19, 2018
    Posts:
    26
    I created a new project in 2019.3.0f3, and they also seem to have the exact same issue - this didn’t always happen and has only just started recently. Is there perhaps some corrupted file perhaps in app data that each editor uses globally?
     
  20. ShockpointGames

    ShockpointGames

    Joined:
    Jul 19, 2018
    Posts:
    26
    To continue the campaign of weirdness. Today I loaded up the same fresh URP project from my original post which had 20s+ play mode entry time (profiler screenshot in first post), and now get a completely expected ~3s play mode entry time:
    upload_2020-1-9_19-13-33.png
     
  21. laurentlavigne

    laurentlavigne

    Joined:
    Aug 16, 2012
    Posts:
    3,579
    since I had 4.6.9 reinstalled and tested editor regression I ran a quick and non scientific test of how much slower script reloading is. It is 3x slower compared to 4.6.9

     
  22. Peter77

    Peter77

    QA Jesus

    Joined:
    Jun 12, 2013
    Posts:
    5,135
  23. interpol_kun

    interpol_kun

    Joined:
    Jul 28, 2016
    Posts:
    133
    You can use OBS (it's free) for screen capture without watermarks. Also you can control your audio right there, IDK but maybe your reaction was the part of the comparison...
     
  24. Lorash

    Lorash

    Joined:
    May 6, 2019
    Posts:
    215
    I've sent in a profile of my tiny project, with asmdefs, that takes several seconds both to compile script changes and to enter play mode. Case 1212215.

    Edit: made another one for another thread. Case 1213483.
     
    Last edited: Jan 21, 2020
    LeonhardP likes this.
  25. spiderspy

    spiderspy

    Joined:
    Aug 1, 2013
    Posts:
    10
    Things are better with 2019.3.0f5 but it is not completely gone yet.
     
  26. Vitaly_Unity

    Vitaly_Unity

    Unity Technologies

    Joined:
    Sep 4, 2013
    Posts:
    8
    Hi @spiderspy,
    Thanks for the reply and sorry for the long responding time (I was on vacation and somehow missed your reply). Unfortunately I can't say much myself on what is UnityEditor.Rendering.DebugWindow as I am a QA from a completely different area, but I asked responsible devs and this is what I got:
    " The DebugWindow is a helper window used to check various settings of the scene, it is available in HDRP code and could be disabled but i doubt this is the source of their issue. If user is on 2019.3 they could try avoiding reloading domain assembly (Project Settings > Editor). I am not sure this is HDRP specific though based on the thread. "
    Not much, but if you have some specific questions - don't hesitate to ask :)
    What I can add is that devs looked into that piece of code and found some potential places for improvements (but not something that could explain behavior you experience though), so lets hope that code will be optimized at least :)

    Thanks again
     
  27. Lorash

    Lorash

    Joined:
    May 6, 2019
    Posts:
    215
    The only thing preventing me from using the new scene reload settings is Unity. The new input system dies unless you do a classic full reload every time you enter Play mode.
     
  28. LeonhardP

    LeonhardP

    Unity Technologies

    Joined:
    Jul 4, 2016
    Posts:
    2,188
    This is a known limitation of the new Input System at the moment. We'll add a note about this to the docs. Adding support for this is on the backlog.
     
  29. 00christian00

    00christian00

    Joined:
    Jul 22, 2012
    Posts:
    887
    @Unity_Javier @LeonhardP
    I tried again asset pipeline v2 after some months hoping the slow down would have been fixed but they are still there.
    I noticed something which could affect many here.
    With v2 now whenever I change a script, all the scripts onValidate methods get called, even if these script aren't in the scene.
    Why is that?
    With V1 this didn't happen. I am using the latest LTS as of today.
    Also is there a way to know on the onValidate method if we are being called normally or in some special case like builds(this is also quite new, old Unity versions wasn't doing this) or compilation?
     
  30. Peter77

    Peter77

    QA Jesus

    Joined:
    Jun 12, 2013
    Posts:
    5,135
  31. 00christian00

    00christian00

    Joined:
    Jul 22, 2012
    Posts:
    887
    It doesn't seem to work inside OnValidate, cannot build the player with that method since it doesn't exist in that context.
    Anyway I confirm that assets pipeline v2 is at least 10 times slower than v1, rolling back immediately again.
    Even after a simple build error it did an additional refresh of probably the whole project for another 20 seconds, while with v1 I can work with the project immediately. I think it is nowhere near v1 at the moment, maybe with huge projects and cache server it makes sense, but not for medium projects where the additional overhead is way too much.

    EDIT
    Ok rolled back. Switching from v1 to v2 took me more than 1 hour.
    Switching from v2 to v1, few seconds....
     
    Last edited: Jul 26, 2020
  32. Peter77

    Peter77

    QA Jesus

    Joined:
    Jun 12, 2013
    Posts:
    5,135
    That's odd, I don't see why it shouldn't work. You did put the code inside an
    #if UNITY_EDITOR  #endif
    block, right?
     
  33. 00christian00

    00christian00

    Joined:
    Jul 22, 2012
    Posts:
    887
    I didn't think of it, I thought it would just ignore the code on build then.
    I just tried and it works... does this mean the code is built two times for every build? One with the UNITY_EDITOR macro and one without? Pretty strange.
     
  34. Unity_Javier

    Unity_Javier

    Unity Technologies

    Joined:
    Mar 7, 2018
    Posts:
    37
    Hey @00christian00 ,

    This is actually a regression with how we're managing loaded objects.
    For example, if you restart your editor with V2 and no prefabs are loaded (i.e. no scene with Prefabs open, and you haven't clicked any on the project browser), then you won't get calls to OnValidate, but if the Prefab does get loaded, then calls to OnValidate will happen.

    We're working on a fix to this right now and I hope to be able to get it backported to 2019.4 relatively soon.

    Thanks for bringing this issue up, it is a pretty bad regression.
     
unityunity