Feedback [ADBV2] Optimizations for the Asset Import Pipeline based on your feedback!

We wanted to share some insight into the performance optimizations that we did after a number of regressions were reported during the 2019.3 Beta phase. The list here is not exhaustive, but we believe there is a good balance of low level details and descriptions between the different systems and intentions we had when optimizing the Asset Import Pipeline based on the valuable feedback we got in the forums and various other communication channels. One thing to note is that this info is purely about optimizations, and we'll address known bugs and other issues on their respective threads which we're definitely watching!

Conservative clearing of internal caches
When changes to a C# file were detected by the Asset Import Pipeline, a number of caches needed to be cleared as their results could be invalidated. However, we were clearing these caches too frequently and repopulating them during calls to EndReloadAssembly. This meant that we were doing a lot of work that was being thrown away every time a change in a C# script was detected. After inspecting how the caches were cleared, we limited the clearing to only happen when code that could invalidate such caches (i.e. changes to Scripted Importers) occured. Otherwise, we keep the caches around and won’t have to redo the work.

Early out on Empty Refresh
When comparing the performance of the old and new Asset Import Pipelines, there was a clear regression that was introduced. This was comprised of performing a full Refresh when nothing changed on the project (i.e. no Asset modifications and no Project Settings changes). This was experienced when going to another application and coming back to Unity then getting a performance hit even though nothing changed. We added a number of states and checks to ensure that when the state of the project is the same, no additional work is required, and as such we were able to make an empty Refresh three times faster than what it was without this optimization.

Partial Enumeration
Whenever a Refresh happens, the Assets and Packages folders need to be recursively scanned for changes. This means that we are crawling the entire project directory every time you focus into the editor, and this can be costly. Unfortunately, a number of APIs naively went through this code path, especially when importing a single Asset (i.e. calling AssetDatabase.ImportAsset) and this turned a call that would ideally import a single Asset to an enumeration of your entire project structure. Therefore, we introduced partial enumeration, which essentially enumerates the file or folder that you specify in your API calls, thus allowing these APIs to scale to larger projects. Additionally, this lays the foundation for an upcoming Directory Monitoring solution that was presensented by Jonas Drewsen at Unite Copenhagen 2019, titled Speed Up Your Assets for Big Projects.

Optimizing how Assets with Importer types are found
There is a distinction to be made between Native Importers and Scripted Importers. Native Importers are essentially importers written in C++ which ship as part of Unity (i.e. the TextureImporter and the ModelImporter and many more). Since these importers are shipped with the Unity Editor, their implementation is guaranteed to be stable and as such we can build some caches where we can associate a GUID to a Native Importer. As such, we created a fast path for getting a GUID’s associated importer type from a cache instead of getting that GUIDs extension and matching that to the correct importer and returning the type. This helps to speed up Assembly Reloads.

Improved Profiler markers
In order to allow for a better profiling experience, a number of new profiler markers were added so that more information about the performance of the Asset Import Pipeline can be extracted. These are especially helpful when a Deep Profiler Capture is taken and shared with us, as that allows us to reason more about the state of the project and the bottlenecks that are experienced. We encourage you all to share profiler captures with us, either via the forum or in a bug report as this can help us see where Unity should be improved to help you with your workflows. Profiler captures compress really well as well, for example a 4GB capture can compress down to a few hundred MBs.


Limit calls to OnProjectChanged
OnProjectChanged is a rather useful callback to latch on to. However, a number of Unity’s own windows also register themselves to this callback for various reasons and calling this method can force a redraw of many windows that don’t necessarily need to be redrawn. If this is done multiple times per Refresh, this can lead to noticeable slowdown within the editor. We have added more checks to make sure that OnProjectChanged actually needs to be called, and if it does have to be called, it won’t be called multiple times per frame for the same windows.

 

Hey, the new Asset Import Pipeline (AssetDatabase V2) is only officially supported for 2019.3 onwards. You can get a copy here

 

@Lars-Steenhoff we are working async and ondemand asset importing. We have it working in an alpha state internally - but one thing is doing the imports ondemand, another thing is making the editor experience nice which still needs a bit of work.

 

@Lars-Steenhoff yes.. for all async imports it will.

 

@Hyp-X We are very much aware of this issue. Also for this we have someone actively working on fixing it as we speak.