Any update regarding "Increased script assembly reload time"?

I’m not convinced the problem is object property serialization. I have a small number of objects in my hierarchy at startup. I am interested in understanding what Unity’s plan is to address the regression.

 

Oh it's not, and that's definitely not what I was implying. I was just listing some steps I took (things I had agency over) to mitigate the long reload times; steps that someone else in my position could try. Again, I got tangible positive results (lower reload times) from migrating the project to an SSD, and upgrading to VS Community edition 2019.

 

The project-specific optimization tips are appreciated.

I did want to say, I constantly poke into this issue/thread because it is explicitly about a documented reproducible Regression Bug.

https://issuetracker.unity3d.com/issues/increased-reload-time

I've taken so many steps to adjust my project to help with iteration time.

but no matter what we do, the same exact project will compile->reload faster on older versions of Unity.


I appreciate the steps the Unity team has taken already.

I'm posting here in between domain reloads, to add my voice to this issue. This is my number 1 thing I want from Unity--to return to previous version iteration times.

Imagine being able to code, save, hit Play, and see your changes 5 seconds later, every time. like in the old days.

 

On Monday I think I am going to try to create a reproduction project tailor made for the issue. I am thinking of writing C# code to generate 10,000 C# scripts. The scripts will exist in the project folder but are otherwise not used. This might show it is compilation or app-domain related.

I'll then benchmark in 2019 LTS, and then 2021.1.18f. Based on ongoing conversations, it seems like this may be a way to clearly demonstrate the issue. If you have ideas and/or suggestions please let me know.

There is a parallel active thread on this topic (https://forum.unity.com/threads/recompile-enter-play-mode-process.777950/page-2#post-7448198). I have 6K CS scripts, and another dev has 8K, and we are both seeing about 15-45s wait times.

Separately, the optimization tips are appreciated! My 8-core 4.8 GHz machine with 6 GB/s NVME SSD isn't fast enough for Unity.

 

I dont believe script compilation time is the primary issue, I think many unity devs who understand the assembly reload process understand that the compilation time would be faster today than before. The issue is the domain reload being longer, and also because of the many packages doing many different things that interact with the domain reload process. In the 2017 releases, where UPM didnt exist, the domain reload times were pretty quick however the compilation times werent. Despite that, the overall reload time was still quicker compared to 2021. What helps most from my testing on the overall reload times (not just compilation), were to not use any asmdefs at all, and use the least amount of packages as possible.

 

Let us know if you manage to reproduce it, and also if you get an answer from Unity about this regression

 

How does domain reload and compilation time change in this test if you have URP/HDRP project?

 

So I should try testing with N assemblies, each containing 1K files?

 

Still testing. Interestingly, this bug https://issuetracker.unity3d.com/is...count-exceed-the-limit-of-1167-scripts?page=2 manifests.

I can't complete import of 10K script files because it fails, regardless of how many scripts are placed in a directory. 10K in one directory fails. 1K per directory fails. 100 per directory fails.

 

It's actually a combination of different things. Compilation time might not be a problem in itself but compilation triggering for just an extra line of comments or a carriage return is something unity could and should avoid. Especially when knowing that domain reloads and stuff is already taking a lot of time.

Not using packages is well, not an option, each game needs what it needs.

 

You could test with the domain reload timings logged. I've found that the burst package has one of the biggest penalties, making urp/hdrp pretty much add 1 extra second because of their dependencies. Of course user code can also be responsible, so if unity serialized the whole game then obviously it would take longer. Though user code serialization should be ignored as that was always something that could make your domain reload times atrocious. You can inspect these times better if you perform a domain reload with deep profiling in editor enabled (though you might run out of ram when inspecting the exact frame). At this point its mostly which combination of packages has the lesser time (hint, 0, which is unfortunate as @TieSKey mentions). Also if the package manager window shows that you have 0 packages, you can still have packages in the project as can be seen in the manifest file, and in the detail domain reload timings log.

 

Yep, Burst have some nasty times. Which would be ok if it only triggered when burstable code is modified but unfortunately it's not the case. The project settings window usually takes 500ms to refresh upon domain reload for me (as reported like... 4 months ago in other thread) so closing every window/tab does help.

.... closing windows and restraining ourselves from using stuff, are we back to windows 95? T_T

 

“You can’t improve what you don’t measure.”

It's all conjecture until we have hard numbers. I'm at the stage of testing with scripts only (no serialization, no object hierarchy) and I haven't gotten to assembly app domain reloading yet. Unfortunately, I can test with only a maximum of 1167 scripts due to this issue (https://issuetracker.unity3d.com/is...count-exceed-the-limit-of-1167-scripts?page=2) in 2021.1.15f1. I am seeing 15+ second delays in the editor when one of these scripts is touched. As indicated above, there may be multiple contributing issues but clearly script count and compilation is a factor. Still gathering data and I'll summarize and report via ticket.

 

I am not creating any menu items. * Editing to reply that I understand what you are saying. Even though I am not adding menu items, the Unity editor generates one menu item per script automatically for the use case you describe. Thank you!

 

@LeonardP Wouldn't that negatively impact performance for larger projects, i.e. for individuals that have several thousand scripts?

 

Gist:
https://gist.github.com/stonstad/1b9281e3384b7644b34cf5c9ceddb42d

The enclosed script generates 10K CS files with the first 500 as monobehaviors to work around the issue described by LeonardP. The workflow for testing is 1) create test files, 2) enter playback, 3) touch 1, 10, or 100 files, 4) exit playback. 5) Wait for recompilation and app domain reload.



By design, the test has no serialized object properties or instantiated objects in the hierarchy.

Unfortunately, I could not reproduce the 15 to 45 second waits I am seeing in my other project. One thing I tried is adding 5 external assemblies, each containing 1,000 classes with generated scripts referencing classes in these assemblies. It had no impact on performance.

If my goal was to show the slow down is compilation and app domain reloading only, I am unsuccessful. What I found strange is that out of perhaps a 100 tests, two or three times it did pause up to 45 seconds on compilation of a single script and app domain reload. Similarly, the Visual Studio editor sometimes became stuck in an update loop. But in both scenarios it was rare and hard to reproduce.

I'm really disappointed that I couldn't isolate it to compilation and app-domain reloads.

 

Did the generated scripts have serialized fields? Perhaps the slowdown is related to how many public fields are in each script

 

OK, thanks to the Iteration Profiler (https://github.com/Unity-Technologies/com.unity.editoriterationprofiler), I identified the cause for my game's slow editor iteration performance. I added a carriage return to a single script, brought Unity to the foreground, and then let the profiler record Unity's iteration process. The V2 AssetDatabase is consuming 6.4s seconds in method ImportAndPostprocessOutOfDateAssets. 60% of my editor iteration time, resulting from changing a single script, is caused by the V2 asset database. This is why my previous testing couldn't reproduce the behavior -- it was not accounting for the size of the asset database in a large or (A)AA project.

 

@DoctorShinobi

In the interest of checking if a common thread is the v2 asset database, could you confirm if you are using it for your affected projects? Also, could you report the size of the asset database in your project?

// size of asset database
Debug.Log(AssetDatabase.GetAllAssetPaths().Length + " Assets");

Here's my timing:
AssetDatabase.V2.ImportAndPostprocessOutOfDateAssets. 22371 Assets, 6488ms

 

Quick question for any of you who are experiencing slow reload times - are you using Visual Studio Community 2017 (or maybe even an older version)?

Just want to reiterate that I am seeing different behavior during script compilation and domain reload after upgrading to VS Community 2019. I suspect that after upgrading to 2019, and migrating to asmdefs for all the code in my project, that 2019 is honoring the asmdefs, and not attempting to reload all the many Unity assemblies when I update my code. In fact, I just checked the date stamps on the assembly csproj files in the root of my project dir, and the csproj files that are specific to my namespace and asmdefs have a timestamp that reflects the last time I updated my code (today) whereas Assembly-CSharp.csproj, Assembly-CSharp.Player.csproj, etc. all have date stamps from a few days ago (not sure what triggered their recompilation - might have been a reimport of my project after downgrading from 2021.1.18 to 2021.1.13 if I remember correctly).

So for those of you who are suffering, have successfully namespaced their code and/or migrated to asmdefs, but haven't upgraded to 2019, you may want to try the upgrade and see how that affects your compilation behavior.

If you haven't namespaced and migrated to asmdefs, then it certainly won't hurt to get your code out of Assembly-CSharp*.csproj and see if that results in any gains in compilation and reload times.

 

Thread w/ discussion of V2 asset database ImportAndPostprocessOutOfDateAsset method:
https://forum.unity.com/threads/recompile-enter-play-mode-process.777950/page-2#post-7448198

Another related thread: https://forum.unity.com/threads/har...52.1707974302.1630328440-511853534.1602175685

 

Thanks @xoofx. Is one of those active bugs the v2 asset database refresh behavior for a single script change? This appears to be 60% of my iteration time in 2021.x.

 

Visual Studio 2019 16.9.0
Visual Studio Code Editor 1.2.3
Visual Studio Editor 2.0.11

 

ImportAndPostprocessOutOfDateAsset is slow because it calls/contains the domain reload (among other sources of slowness).

As for the domain reload itself I find Burst dll taking like 1 full second by itself.

 

Why is it sometimes fast and sometimes slow when I trivially add a carriage return to a script, as I describe in this thread: https://forum.unity.com/threads/recompile-enter-play-mode-process.777950/page-2#post-7448198 ?

 

Ha, for me it is always slow but my project is not that big so it is """"only"""" 15secs...

 

@xoofx Just to reiterate that this is a huge workflow hindrance which is affecting all the studios working on larger projects out there. Needless to say it should be a very high priority on you guys' agenda. If that is really not fixed in 2021.2 it will eventually become a deal-breaker.

 

I get better results on my 980pro (7000Mbs) than the 500Mbs SSD or the HDD, but the reloading times are huge anyway to get stuck every 1-2 minutes you are testing something in the editor...

2021.1.19f1 still has this issue. #crying

 

Honestly, if you open a small project with 2019 vs 2020. You will notice a huuge difference in loading times. That loading bar is keeping us waiting for a minute or so. I have no idea what have been introduced via 2020+ but that must be solved. Because it is the fundamental part of every day Unity use and it is oddly slow.

 

Having major issues here also - many other devs in the studio have the same problems, if it does not become solved soon it will push towards Unreal. In our cases it simply seems to lock up on reloading script assemblies.

 

I am still on 2019LTS since the reload times went all downhill since 2020. I am more worried that The LTS support will end in half a year. Do you concider to extend the LTS for 2019 if this isn’t fixed?

 

I was able to resolve the issue by following advice in this thread in regards to Visual Studio.

https://stackoverflow.com/questions/34823493/scanning-new-and-updated-mef-components/39454620

-------------------------
Particularly this response:
"I had to delete the contents of %localappdata%\Microsoft\VisualStudio\14.0\MEFCacheBackup as well as %localappdata%\Microsoft\VisualStudio\14.0\ComponentModelCache"
-------------------------
I was using VisualStudio 2019 v16.11.2 and the only difference was the version number folder. Unity version is v2020.3.11f1. Went from 5-10 minute script reload time to the standard 3-5 seconds or so. The issue hasn't happened again but I'm keeping an eye on it since I'm not sure if this will work for others.

 

blank projects with a single script still have this issue... it's way worse on MacOS M1 + 2021.1.20f1...

I love Unity but dunno why everything beyond 2020.1.17f1 is that slow...

 

Not working on my end. Still same reloading time with latest version from VS Installer

 

Does not matter if its a huge project or new one, it happens. Just create a new project and bingo. issue.

 

weird to read that when you can see so many people here with the same issue on different platforms... :/

 

Is 2020.1.17 faster than 2021.1 on M1?

 

Actually yes. In big projects the busy/delay time increases a lot in 2021. 2019 is the fastest (by far) tho.

 

That's reassuring news. I'm hoping the bar is high -- we need to return to subsecond iteration for script-only changes.

 

You guys should check out the alpha in regards to the iteration times, just sayin. Rumours say it might be a bug.

 

Better or worse, and by how much?

*update.

Domain reload may be disabled in the build you tested (https://forum.unity.com/threads/what-is-happening-with-this-alpha.1167083/). Not an option for me for playback but happy for those that can disable domain reloading. Hoping complete domain reload is skipped with script changes.