Search Unity

  1. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Feedback [ADBV2] Optimizations for the Asset Import Pipeline based on your feedback!

Discussion in '2019.3 Beta' started by Unity_Javier, Jan 28, 2020.

  1. Unity_Javier

    Unity_Javier

    Unity Technologies

    Joined:
    Mar 7, 2018
    Posts:
    188
    We wanted to share some insight into the performance optimizations that we did after a number of regressions were reported during the 2019.3 Beta phase. The list here is not exhaustive, but we believe there is a good balance of low level details and descriptions between the different systems and intentions we had when optimizing the Asset Import Pipeline based on the valuable feedback we got in the forums and various other communication channels. One thing to note is that this info is purely about optimizations, and we'll address known bugs and other issues on their respective threads which we're definitely watching!

    Conservative clearing of internal caches
    When changes to a C# file were detected by the Asset Import Pipeline, a number of caches needed to be cleared as their results could be invalidated. However, we were clearing these caches too frequently and repopulating them during calls to EndReloadAssembly. This meant that we were doing a lot of work that was being thrown away every time a change in a C# script was detected. After inspecting how the caches were cleared, we limited the clearing to only happen when code that could invalidate such caches (i.e. changes to Scripted Importers) occured. Otherwise, we keep the caches around and won’t have to redo the work.

    Early out on Empty Refresh
    When comparing the performance of the old and new Asset Import Pipelines, there was a clear regression that was introduced. This was comprised of performing a full Refresh when nothing changed on the project (i.e. no Asset modifications and no Project Settings changes). This was experienced when going to another application and coming back to Unity then getting a performance hit even though nothing changed. We added a number of states and checks to ensure that when the state of the project is the same, no additional work is required, and as such we were able to make an empty Refresh three times faster than what it was without this optimization.

    Partial Enumeration
    Whenever a Refresh happens, the Assets and Packages folders need to be recursively scanned for changes. This means that we are crawling the entire project directory every time you focus into the editor, and this can be costly. Unfortunately, a number of APIs naively went through this code path, especially when importing a single Asset (i.e. calling AssetDatabase.ImportAsset) and this turned a call that would ideally import a single Asset to an enumeration of your entire project structure. Therefore, we introduced partial enumeration, which essentially enumerates the file or folder that you specify in your API calls, thus allowing these APIs to scale to larger projects. Additionally, this lays the foundation for an upcoming Directory Monitoring solution that was presensented by Jonas Drewsen at Unite Copenhagen 2019, titled Speed Up Your Assets for Big Projects.

    Optimizing how Assets with Importer types are found
    There is a distinction to be made between Native Importers and Scripted Importers. Native Importers are essentially importers written in C++ which ship as part of Unity (i.e. the TextureImporter and the ModelImporter and many more). Since these importers are shipped with the Unity Editor, their implementation is guaranteed to be stable and as such we can build some caches where we can associate a GUID to a Native Importer. As such, we created a fast path for getting a GUID’s associated importer type from a cache instead of getting that GUIDs extension and matching that to the correct importer and returning the type. This helps to speed up Assembly Reloads.

    Improved Profiler markers
    In order to allow for a better profiling experience, a number of new profiler markers were added so that more information about the performance of the Asset Import Pipeline can be extracted. These are especially helpful when a Deep Profiler Capture is taken and shared with us, as that allows us to reason more about the state of the project and the bottlenecks that are experienced. We encourage you all to share profiler captures with us, either via the forum or in a bug report as this can help us see where Unity should be improved to help you with your workflows. Profiler captures compress really well as well, for example a 4GB capture can compress down to a few hundred MBs.


    Limit calls to OnProjectChanged
    OnProjectChanged is a rather useful callback to latch on to. However, a number of Unity’s own windows also register themselves to this callback for various reasons and calling this method can force a redraw of many windows that don’t necessarily need to be redrawn. If this is done multiple times per Refresh, this can lead to noticeable slowdown within the editor. We have added more checks to make sure that OnProjectChanged actually needs to be called, and if it does have to be called, it won’t be called multiple times per frame for the same windows.
     
    phobos2077, grizzly, Rallix and 7 others like this.
  2. TextusGames

    TextusGames

    Joined:
    Dec 8, 2016
    Posts:
    429
  3. iamarugin

    iamarugin

    Joined:
    Dec 17, 2014
    Posts:
    879
    Thanks, this is a great news.
     
  4. MegaMileyStudios

    MegaMileyStudios

    Joined:
    Aug 19, 2017
    Posts:
    72
    Great news! What versions of Unity will this fix be ported to? And when can we see this in action ourselves?
     
    phobos2077 and Lars-Steenhoff like this.
  5. Unity_Javier

    Unity_Javier

    Unity Technologies

    Joined:
    Mar 7, 2018
    Posts:
    188
    Hey, the new Asset Import Pipeline (AssetDatabase V2) is only officially supported for 2019.3 onwards. You can get a copy here :)
     
  6. Lars-Steenhoff

    Lars-Steenhoff

    Joined:
    Aug 7, 2007
    Posts:
    3,509
    For me the texture compression and importing is taking the longest time in my workflow.

    Is the texture compressing in a background process coming any time soon?

    So that I can continue working while assets are importing and getting compressed.

    Thanks
     
    phobos2077 likes this.
  7. Lars-Steenhoff

    Lars-Steenhoff

    Joined:
    Aug 7, 2007
    Posts:
    3,509
    And the other thing I would love to see it that only textures are compressed that I actually use in a scene,
    so when making a build I don't need to wait for all the textures in the entire project to be compressed for build.
     
    phobos2077 and valentingurkov like this.
  8. jdrewsen

    jdrewsen

    Unity Technologies

    Joined:
    Mar 28, 2011
    Posts:
    204
    @Lars-Steenhoff we are working async and ondemand asset importing. We have it working in an alpha state internally - but one thing is doing the imports ondemand, another thing is making the editor experience nice which still needs a bit of work.
     
    phobos2077 and Lars-Steenhoff like this.
  9. Hyp-X

    Hyp-X

    Joined:
    Jun 24, 2015
    Posts:
    421
    Please bring back the status dialogs from ADBV1 (or equivalents)
    - Calculating checksums => in ADBV2 there's no progress dialog on screen during this phase
    - Compiling scripts => in ADBV2 the previous progress dialog left open frozen at 99%
    - Downloading from cache server => in ADBV2 the Importing... line does not progress if the asset is downloaded (asset name not shown, progress not accumulated so never reaches 100%)
    Most of the time we stare at Task Manager to see if Unity is doing anything or locked up
     
  10. Lars-Steenhoff

    Lars-Steenhoff

    Joined:
    Aug 7, 2007
    Posts:
    3,509
    Great to hear its coming along!
    And I imagine the process bar to be moved to the new background processes window?
     
  11. jdrewsen

    jdrewsen

    Unity Technologies

    Joined:
    Mar 28, 2011
    Posts:
    204
    Lars-Steenhoff likes this.
  12. jdrewsen

    jdrewsen

    Unity Technologies

    Joined:
    Mar 28, 2011
    Posts:
    204
    @Hyp-X We are very much aware of this issue. Also for this we have someone actively working on fixing it as we speak.
     
    phobos2077 likes this.
  13. iamarugin

    iamarugin

    Joined:
    Dec 17, 2014
    Posts:
    879
    Could we please have an option to manually disable/trigger compilation?
    Right now if I need to create a bunch of scripts, my workflow looks like:
    - Add script to folder.
    - Wait 20 seconds until it compiles.
    - Add another script.
    - Wait 20 seconds until it compiles.
    ...

    And if I made a mistake and need to rename the file, then, oh well, I need to wait another 20 seconds. It will be much easier to add a bunch of files and then trigger the compilation by pressing dedicated button or hotkey.
     
  14. Lars-Steenhoff

    Lars-Steenhoff

    Joined:
    Aug 7, 2007
    Posts:
    3,509
    Yes and an option to only compile when pressing play or when making a build but not on import would be nice
     
  15. MegaMileyStudios

    MegaMileyStudios

    Joined:
    Aug 19, 2017
    Posts:
    72
    I wish it was possible for scripts to be compiled on a background thread, too bad that's most likely going to be too big of a structural change that the Editor won't be able to support for now
     
    phobos2077 and Lars-Steenhoff like this.
  16. Baste

    Baste

    Joined:
    Jan 24, 2013
    Posts:
    6,266
    You can disable it:
    upload_2020-2-5_12-42-58.png

    Ctrl+R to refresh assets, will cause Unity to recompile.


    This also causes all other assets to not automatically refresh, so if that's an issue, you'll need some other workaround. You could create scripts from whatever tool you use to write code.
     

    Attached Files:

  17. iamarugin

    iamarugin

    Joined:
    Dec 17, 2014
    Posts:
    879
    The whole experience of working with Unity is "you need some workaround" now. I don't want workarounds. I don't want to disable auto refresh, because then I will change some game content (I do it more often than adding scripts), forget to refresh it, then press play, and wait another 20 seconds of domain reloading (disabling it doesn't work atm because a lot of packages currently not support this feature). And then I will discover that my changes was not applied and repeat this cycle again.

    With the scripts you can recompile them on game start or highlight the play button to notify, that they were not compiled.

    Anyway the huge problem with the editor that it completely freezes every time, when recompiling or importing happens. You can not even minimize a window during that time, to hide a console that occupied half of the second screen. If async editor processes will become a thing, may be we will not require such workarounds.
     
  18. mahdi_jeddi

    mahdi_jeddi

    Joined:
    Jul 18, 2016
    Posts:
    245
    I think I've got a problem with sprite atlases that would need the same kind of optimization:

    https://forum.unity.com/threads/hug...h-lots-of-sprite-atlases-case-1243023.880147/
     
    Lars-Steenhoff likes this.