[Asset pipeline v2] Script reimport performance

Hi, I upgraded a project to 2019.3b12 and asset pipeline v2. I'm using a bunch of unity packages like Entities, Jobs and Burst.

Changing one line in a script and going back to unity takes 9 seconds (it was closer to 5-6 seconds in unity 2019.2). Looking at the profiler, I feel like there is a lot of room for optimization. For example ScanAssemblies in EndReloadAssembly spends 375ms scanning all assemblies, of which only 5.5ms is spend on my game assembly, the only one that actually changed. Most of the time is spent scanning UnityEngine and Unity.Entities, and those basically never change in a normal workflow. Couldn't result of the scanning be kept around between reloads somehow?

That exemple aside, is the performance of scripts reloading being worked on, and can we expect improvements? With unity moving more and more of its engine modules over c#, it becomes especially important that the reimport time of c# code is kept low imo.

 

Here are some screenshot and a link to my profiler save:

https://drive.google.com/file/d/1-7I_H5m-WvSWcTi22RI73VGYcRZ-ILPc/view?usp=sharing

 

Hey, could be worth to submit a bug report with your findings.

 

Hi Javier,

Sure, happy to help where I can

Our project has quite a lot of heavy textures. I'm curious as to what impact it could have. I'm guessing it shouldn't matter for script compilation and reloading, but could impact the 1.4 sec spent in AssetDatabase.V2.PostProcessAllAssets?

I'm guessing our project is also rather script heavy, I don't know what's a normal amount of script. I'm also not sure what proportions of scripts are from packages like Entities versus our own project.

I saw that unity 2019.3f1 was released over night so I switched to it. It seems to be a little faster (around 8 seconds). Or it could be due to a change my method of testing or environnement, I'm not sure. If you need I can send another profile.

The script you provided ran with an error:
NullReferenceException: Object reference not set to an instance of an object
AssetDistribution.GetAssetDistribution () (at Assets/Editor/AssetsDistribution.cs:18)

By stepping in with the debugger I found that AssetImporter.GetAtPath() returns null for one of the paths: "Packages/com.unity.entities/Unity.Entities.Editor/Mode/default.mode"

So I added a null-check

Code (CSharp):

1.  
2.             var curImporter = AssetImporter.GetAtPath(assetPath);
3.             if (curImporter == null) {
4.                 Debug.LogWarning($"No importer for asset at path {assetPath}");
5.                 continue;
6.             }
7.  

Here is what the script logged:

Code (CSharp):

1.  
2. UnityEditor.AssetImporter,971
3. UnityEditor.MonoImporter,2675
4. UnityEditor.PrefabImporter,85
5. UnityEditor.TextureImporter,921
6. UnityEditor.TrueTypeFontImporter,27
7. UnityEditor.TextScriptImporter,93
8. UnityEditor.ShaderImporter,102
9. UnityEditor.ModelImporter,9
10. Unity.VectorGraphics.Editor.SVGImporter,8
11. UnityEditorInternal.PackageManifestImporter,56
12. UnityEditorInternal.AssemblyDefinitionImporter,109
13. UnityEditor.StyleSheets.StyleSheetImporter,24
14. UnityEditor.PluginImporter,11
15. Unity.Build.BuildPipelineScriptedImporter,2
16. UnityEditor.UIElements.UIElementsViewImporter,11
17. UnityEditor.AudioImporter,10
18. UnityEditorInternal.AssemblyDefinitionReferenceImporter,1
19. Total assets: 5116
20. UnityEngine.Debug:Log(Object)
21. AssetDistribution:GetAssetDistribution() (at Assets/Editor/AssetsDistribution.cs:40)
22.  

I didn't make a bug report right away, because I wasn't sure if this would be treated like a bug, and reporting bug has proven quite time consuming for me in the past. Also our project is pretty big, would it still be useful for you if I made a bug report without attaching all of the project somehow (project might be in a broken state then) ?

 

(or are you mainly interested in my machine specs, info on the project like package used and profiler saves, in which case I can submit a bug without the project but with this info?)

 

Glad you could reproduce a similar refresh time with just unity packages. It seems there is sill a lot of work to do before reaching the 500ms goal mentioned at unite some time ago

Here is a link to a deep profile, reload happens at frame number 658.
https://drive.google.com/file/d/1-Fj7XBLPeMbsgyh_UZTU7cz8sMl4VpEE/view?usp=sharing

Thanks for the support! I really hope it will help, I'm happy to provide more details if needed

 

Indeed, I can't load the save either. I did another one and couldn't load it either, it seems to me like the profiler has a bug and doesn't save big profiles correctly. I get this error message when loading one of the deep profiles:


So I guess that's another bug to log ¯\_(ツ)_/¯
But in the meantime, since you managed to reproduce the high reload times with a project of your own, maybe you can pass it to the relevant teams, and they can do a deep profile by themselves?

 

I just want to add that I got this too and I cannot efficently edit assets because it wants to autosave and reimport everything that is related, which is the exact opposite what you would expect from nested prefabs, should just be links right? Also why does it have to block anyone's mainthread and usage at all? Also would be nice if it was a package so that we can fix stuff for ourselves.

Also the Asset pipeline v2 F***s things up for my special editor windows. It, for some god damn reason uses editor window stuff when it import it during a save which apparently causes one of my lists of script objects top be full of null during one stage which is quite weird. Which triggers some of my editor window sanity checks and set things to false because it is null, even though it isnt actually null.

//Rant

Edit:
Wanted to add my current project size

Code (CSharp):

1.  
2. UnityEditor.AssetImporter,2903
3. UnityEditor.MonoImporter,3397
4. UnityEditor.TextureImporter,5984
5. UnityEditorInternal.AssemblyDefinitionImporter,79
6. UnityEditor.ModelImporter,125
7. UnityEditor.PrefabImporter,667
8. UnityEditor.TrueTypeFontImporter,28
9. UnityEditor.AudioImporter,203
10. UnityEditor.PluginImporter,40
11. UnityEditor.TextScriptImporter,128
12. UnityEditor.ShaderImporter,229
13. UnityEditorInternal.PackageManifestImporter,59
14. UnityEditor.StyleSheets.StyleSheetImporter,117
15. UnityEditor.UIElements.UIElementsViewImporter,59
16. UnityEditor.ShaderGraph.ShaderGraphImporter,6
17. MLAgents.DemonstrationImporter,1
18. Barracuda.NNModelImporter,8
19. Total assets: 14033

Edit 2: also I cannot properly debug the save because unity crashes ect, but takes like 24 seconds.

 

If only Unity tried making just one full AA+ commercial game on their own, they would've catch 90% of annoying issues and regressions like this, without being surprised every time something like this is brought up by users.

 

A million times this. The more you structure things using nested prefabs, the more abysmal their usability gets due to all the progress bars anytime you change a variable with autosave (or you disable autosave and just have to grit your teeth every time you're about to save).

 

Okay I managed to save the dump file at home but it turns out to be 3.3 GIGABYTE, so I took some snippets that I hope will aid you Unity_Javier.

The context for this save is a fairly empty scene (where we will generate our procedural level) the scene is only about 740kb large so it is suprising that it takes 12-24seconds to save even though in this case I have not even changed anything at all but just pressed save.

I hope you will solve this, thanks.

 

Could you explain to me what UnityEditor.Rendering.DebugWindow is? and I will look into it. @VitalyV

Also concerning context, is that I am saving a small scene that will procedurally generate a larger scene world, but at the point of savig it is a fairly small scene, less than 1mb. We use the nested prefabs a lot, it is a 2D based project.

Unfortunately I cannot really share the project in itself.

 

Just like to point out that I've noticed these incredibly long script compile and enter play mode times with brand new projects. In 2019.3.0f4, I created a new URP project and without any modifications to the project I'm getting 20+ second play mode entry times. Here's what the profiler produces:



Just like to add how odd it is that it's using ~1.8GB GC allocation for a project that is ~1GB in size.

 

I created a new project in 2019.3.0f3, and they also seem to have the exact same issue - this didn’t always happen and has only just started recently. Is there perhaps some corrupted file perhaps in app data that each editor uses globally?

 

To continue the campaign of weirdness. Today I loaded up the same fresh URP project from my original post which had 20s+ play mode entry time (profiler screenshot in first post), and now get a completely expected ~3s play mode entry time:

 

since I had 4.6.9 reinstalled and tested editor regression I ran a quick and non scientific test of how much slower script reloading is. It is 3x slower compared to 4.6.9

 

I brought this up a while ago too:
https://forum.unity.com/threads/cas...ty-project-takes-significantly-longer.691996/

 

You can use OBS (it's free) for screen capture without watermarks. Also you can control your audio right there, IDK but maybe your reaction was the part of the comparison...

 

I've sent in a profile of my tiny project, with asmdefs, that takes several seconds both to compile script changes and to enter play mode. Case 1212215.

Edit: made another one for another thread. Case 1213483.

 

Things are better with 2019.3.0f5 but it is not completely gone yet.

 

The only thing preventing me from using the new scene reload settings is Unity. The new input system dies unless you do a classic full reload every time you enter Play mode.

 

@Unity_Javier @LeonhardP
I tried again asset pipeline v2 after some months hoping the slow down would have been fixed but they are still there.
I noticed something which could affect many here.
With v2 now whenever I change a script, all the scripts onValidate methods get called, even if these script aren't in the scene.
Why is that?
With V1 this didn't happen. I am using the latest LTS as of today.
Also is there a way to know on the onValidate method if we are being called normally or in some special case like builds(this is also quite new, old Unity versions wasn't doing this) or compilation?

 

For detecting whether it's during a build, you could use

BuildPipeline.isBuildingPlayer

:
https://docs.unity3d.com/ScriptReference/BuildPipeline-isBuildingPlayer.html

 

It doesn't seem to work inside OnValidate, cannot build the player with that method since it doesn't exist in that context.
Anyway I confirm that assets pipeline v2 is at least 10 times slower than v1, rolling back immediately again.
Even after a simple build error it did an additional refresh of probably the whole project for another 20 seconds, while with v1 I can work with the project immediately. I think it is nowhere near v1 at the moment, maybe with huge projects and cache server it makes sense, but not for medium projects where the additional overhead is way too much.

EDIT
Ok rolled back. Switching from v1 to v2 took me more than 1 hour.
Switching from v2 to v1, few seconds....

 

That's odd, I don't see why it shouldn't work. You did put the code inside an

#if UNITY_EDITOR  #endif

block, right?

 

I didn't think of it, I thought it would just ignore the code on build then.
I just tried and it works... does this mean the code is built two times for every build? One with the UNITY_EDITOR macro and one without? Pretty strange.

 

@Unity_Javier Are you still working on this issue with slow load times in general?

It seems from my testing, more than compile times, the issue is really one of file count.Once you get up to 10,000 files plus, your reload times are going to be >10s, and it really doesn't matter what you do...

I wonder if we can go back to first principles on this: We need a scripting layer, that has access to everything csharp-firstpass has, which can be re-compiled quickly. Without triggering a full-scan of project.

Is this really unachievable?

Is there not some alternate angle we can approach this from to just make it happen?
1. Maybe some new dll that sits on top of c-sharp-firstpass, meant specifically for "hot" script files? We could make /Scripts a special folder (optionally of course), that gets it's own top-level DLL and special watch rules.
2. Make /Plugins a 'safe space' for cold asset files. Give us some ability to manually reload /Plugins, so we can have some area of our project that can be ignored during 99% of our domain reloads. Then we can use everything outside of plugins as our "hot" asssets, and plugins can be all the cold stuff.

Any other thoughts?

The fundamental issue here is that projects have hot and cold files, and Unity does not acknowledge this reality. It lumps them all together, making reloads extremely wasteful. When 98% of your changes, are made to <5% of the files, we should not be re-loading 100% of the files on each line of code change.

We need in some way, to de-couple script rebuilds, and asset scanning.

 

I believe they addressed this with the new "Directory Monitoring" option, which was introduced in 2020.2 maybe?

 

Very interesting! I was just reading about this in the 2020.1 release notes.

Just upgraded and it does seem to have shaved off about 2-3 seconds! I'm at about 7 seconds now to reload, with about 10,000 files in the project and 3000 scripts (73 outside of /plugins). As long as it stays around here we're in a pretty good place!

Still would be really nice to have this scripts-prepass.dll that was like 1s compile time...

 

Awesome, thx for the info, and so glad you guys are still on this. By far the #1 issue in my books.

 

I'm suffering with this issue at the moment ... I set up assembly definition files to try to fix it and that had no effect other than to reduce compile times slightly. The delay all seems to be in domain reload. It also seems to coincide with us starting to use addressables in our project.

I profiled the editor and it shows this ...



Note the giant GC alloc for around 3GB. I suspect this might be it reloading all the textures we are using in our addressables database - this is just a guess at this stage. Anyone have any thoughts on this as waiting an entire minute for every compile and reload is hitting our productivity very hard.

 

It's not from Addressables and it also doesn't look like anything from Unity. It looks to me like this comes from a 3rd party plugin/package, because the method name looks obfuscated to me.

Do you use plugins from the Asset Store? The plugin contains a type named

DirStructChecker

with a method named

JPNNHJMOLJL

. This method is called by Unity, because it's decorated with the [InitializeOnLoadMethod] attribute.

EDIT: The offender in question seems to tbe CTS - Complete Terrain Shader:
https://forum.unity.com/threads/cts-complete-terrain-shader-deprecated.477615/page-54#post-5668513

 

OK - this is good info - I don't actually use CTS any more, so I think this is probably debris from the package being removed. Thankyou for the assistance.

 

@Unity_Javier

Hi,

with the latest versions of Unity (currently on 2020.2.7) we are dealing with sever performance issues. Changing single line of code results in 45s-90s refresh. It is stucked on ImportAndPostprocessOutOfDateAssets. I wasn't able to track down possible root of this issue. Why script reload triggers ImportAndPostprocessOutOfDateAssets? What does it do? How can we track down possible files/assets which are causing this issue?

Here is part of the editor log after changing script:
Mono: successfully reloaded assembly
- Completed reload, in 3.095 seconds
Platform modules already initialized, skipping
Reloading assemblies failed.
Refreshing native plugins compatible for Editor in 2.27 ms, found 6 plugins.
Preloading 0 native plugins for Editor in 0.00 ms.
Refresh completed in 46.114201 seconds.
RefreshInfo: RefreshV2(AllowForceSynchronousImport) scanfilter:
RefreshProfiler: Total: 46114.099ms
InvokeBeforeRefreshCallbacks: 0.468ms
ApplyChangesToAssetFolders: 0.064ms
Scan: 14.534ms
OnSourceAssetsModified: 0.773ms
InitializeImportedAssetsSnapshot: 17.187ms
GetAllGuidsForCategorization: 2.325ms
CategorizeAssets: 206.322ms
ImportAndPostprocessOutOfDateAssets: 45789.138ms (43767.687ms without children)
ImportManagerImport: 31.986ms (8.424ms without children)
ImportInProcess: 23.517ms
UpdateCategorizedAssets: 0.045ms
CompileScripts: 1931.153ms
PostProcessAllAssets: 13.150ms
ReloadImportedAssets: 1.995ms
EnsureUptoDateAssetsAreRegisteredWithGuidPM: 3.819ms
InitializingProgressBar: 0.003ms
PostProcessAllAssetNotificationsAddChangedAssets: 1.448ms
OnDemandSchedulerStart: 0.594ms
RestoreLoadedAssetsState: 12.721ms
UpdateImportedAssetsSnapshot: 24.582ms
ReloadSourceAssets: 40617.222ms

 

@Unity_Javier similar to @PerunCreative_JVasica we are hitting huge stalls when launching the Editor (currently 2019.4.23) when it does PostProcessAllAssets under ImportAndPostprocessOutOfDateAssets. It doesn't seem to happen on macOS but on Windows there's a 2-3 minute stall every launch of the project.

There's not a lot of info on the forums about this stuff. How can we find out what is causing all of this reimporting? Any tips would be appreciated.

eg.
ImportAndPostprocessOutOfDateAssets: 262236.289ms (3132.804ms without children)
....
PostProcessAllAssets: 249107.450ms

 

Thanks for the quick reply and tips, Javier. I'll profile a launch when we hit the wall again. It's happening a lot on Windows, but not on Mac at the moment. If I dig up anything useful I'll post or DM you - and sorry for the thread hijack but it seemed related and this thread shows up in search results a lot so the info will be useful

Update: thanks, Javier! The profiler helped track down a package doing expensive (dumb) stuff at launch and is showing some suspect postprocessing importing too. Not sure why Mac is ok, or why deleting the /Library and /Obj folders and reimporting assets seems to flush it out for a while, but we'll take it.