AssetDatabase v2 Refresh Significantly Slower Than v1

I'm doing some preliminary 2019.3 tests on a very large project in terms of number of individual assets. One of the features we were most excited about was v2 of the AssetDatabase, which was supposed to significantly increase Editor performance for large projects.

After creating two side-by-side versions of our project, one on 2019.2.2 and one on the latest 2019.3 beta release, not only did we not see any gains, but we actually took a ~40% hit in asset database refresh time (2-second average on 2019.2 and 2.8-second average on 2019.3). Please note that these tests were done on a machine with a significantly better SSD than most developers at our company possess, so in actuality these numbers will be several times higher for our average dev. This is a huge concern for us because refreshes are already extremely slow on our project and eat up a lot of productivity.

I'm hoping that someone can help me figure out what's going on here and whether or not this is something we can get fixed in a later release. I didn't see anything in the Known Issues list that seemed related to refreshes, which worries me a little bit more than I would be otherwise.

If specifics about our project, such as number of assets and directories (we have a lot of directories) would be helpful, please let me know and I can provide those numbers.

Any info/insight would be greatly appreciated!

 

Seems like you ran into this issue:
https://issuetracker.unity3d.com/is...tes-slower-when-compared-to-previous-versions

 

I'd like to say I'm also running into this, but much, much worse. Asset import times that were ~10ms in 2019.2 are now at 500ms, and every single individual model takes at least 13s. Doesn't seem to be related to asset type at all, and this is on a 2018 MacBook pro w/ solid state drive.

Whole project took 30 hours to re-import. :S

 

My project has been re-importing now for around 24 hours. It looks like it's importing all the .fbx files. But what I'm seeing is that every one is taking around 20 seconds. I guess I was hoping it was simply only updating the progress bar title occasionally, and was actually making more progress that what the progress bar seems to be indicating. If this is really only importing 1 item every 20 seconds, then I imagine it will take many days to complete the process. I guess I'll let it run, but I'm hoping this is a one-time process.

 

According to: https://issuetracker.unity3d.com/issues/focusing-the-editor-window-causes-noticeable-slowdowns
It was fixed in version 2020.1.0a19.

I am running a project in 2020.2.0a13 and it is persisting. This is an issue being seen by a lot of people rendering Unity unusable as reimporting scripts is taking a long long time.

 

Seeing this issue on 2020.2a15.1993 too a lot, change one character in a script and wait 20 sometimes 30 seconds for an asset database refresh. Almost unusable.

 

It is currently taking me upwards of 40 minutes to reimport or refresh anything, even just one script. I am struggling through as my game is nearly ready for release, but this is insane.

 

We just Updated to version 2019.4 and the AssetDatabase.refresh is taking huge chunks of time to finish when I simply add a new line to a script and click back in the editor. Took 32 seconds to finish loading the editor. They said they fixed it in 2019.3 https://issuetracker.unity3d.com/issues/focusing-the-editor-window-causes-noticeable-slowdowns But immediately in the next minor version people were reporting it again. And even in the 2020 version people are still experiencing the issue.

 

Restarting Unity solves this problem for me. However, after some time, the lag returns and have to restart again.

 

Update: I found that deleting prefabs means import takes seconds, I wonder if that fixes it for everyone else. Of course, we really need prefabs to work. I know in 2020 they had changes to prefabs but none of the 2020 version have decent prefab importing. 1.5 hours it started taking for my 20ish prefabs, Unity is becoming unusable and Unity are being silent.

 

I was having very long times, so I went from 2020 back to 2019. It seemed better, but then things slowed down again. 13 seconds to compile my script and 13 more just to start the game. Out of desparation, I rebooted Windows and the times both went from 13 down to 3.

So, maybe it isn't all Unity's fault?

 

exactly the same behavior for me - sometimes even more than 30 seconds. It completely slows down our progress in our current project...

 

Everybody who still experience significant delays with the latest Unity version (2019.4 LTS, 2020.x), please submit a bug-report as described in this document: https://unity3d.com/unity/qa/bug-reporting

If we don't submit those bug-reports, Unity Technologies probably won't or can't fix it. I like to think if they don't have a bug-report, the issue does not exist for them. Posting in this forum thread will not cause the issue to disappear, unfortunately.

It's important that you report these issues together with a reproduction project if you want them to get fixed. If you don't do it, it might be a long time until someone else reports them or until Unity Technologies find them.

After you submitted the bug-report, you receive a confirmation email with a bug-report Case number. Please post the Case number (number only, not the link) in this forum thread for Unity staff to pick up.

 

Submitted a bug for this a couple of weeks ago (case 1270910). It would be great if we at least got the issue on the issue tracker to be re-opened so people could vote on it. It’s kinda insane that it’s still closed despite there being tons of comments from people saying that the issue is still there. This delay on every single script change is really making Unity annoying to use.

@Peter77 BTW not to nitpick, but if you’re asking others to share their case numbers, why not share your own too?

 

Hi guys! Here is what I got after reporting this bug

"Hi,

Thank you for contacting Unity about your issue.

The reported problem is not a bug. AssetDatabase.Refresh window is intended to let the user know that Unity did not freeze and is currently working on something (Asset importing, script compilation, etc.). You can read more about Asset Database Refresh window here: https://docs.unity3d.com/Manual/AssetDatabaseRefreshing.html"

So, guys, relax! It's not a bug. It's a feature

My case number 1276368

 

(which is why reporting bugs to Unity is useless)

 

I'm having this issue big time now in Unity 2020.1.4f1...

 

I'm seeing this constantly now in 2020.1.6f1 and it's causing a huge amount of lost time. Just the simplest change will cause Unity to sit idle with AssetDatabase.Refresh showing for 6+ minutes. The only way around it is to restart Unity every 20 minutes or so, and this in itself causes a huge amount of lost productivity.

 

I received a similar response to my initial bug report where they said

which to be honest really pissed me off, because it doesn't even address the issue. But I sent them a long email linking to these forum threads and other issues, and as a result got the issue reproduced and opened (see https://issuetracker.unity3d.com/is...slow-asset-reimport-when-modifying-any-script - case number 1270910)

So I would say don't give up if the initial response is not what you hoped for. Just the fact that they re-checked the problem and ended up replying they reproduced it and opened a votable issue for it shows it's not useless to send bug reports

Maybe a question for Unity is if it would help if all of us experiencing the issue would use the iteration profiler (https://github.com/Unity-Technologies/com.unity.editoriterationprofiler) and send the exported profiler data as well? I'm not sure if this information would be useful for them.

 

I've been dealing with this issue for the last few days since I've updated from 2019 to 2020.1.9 ; 1 minute to save any script and same to run the game. I always save to Dropbox, so I closed Dropbox to see if there was a difference and it runs now just like before almost instantly. I don't know if that was the problem but I thought i post it here.

 

Currently getting this issue w/ a fresh-ish build on 2020.1.4f1.

 

It start from 2019.3.x as v2 pipline is forced to be used.
Unity wont fix this because this issue seems quite related to the hardware and even the same machine would have different performance.
It is really hard to create this problem even even even you reinstall your system.
This S*** really ruied unity and hope the team could take a little care on it.

 

Same here. 2 minute 40 seconds to update one line of code in a project just opened 3 minutes ago in v2020.1.6.
Intel I7, 16gb RAM.
Unity, right now, is a completely useless application.

 

I also find the new AssetDatabase.Refresh to be unacceptably slow.

Using the Experimental Enter Play Mode setting in Project Settings helps when doing scene playing and testing, but the time after script compile is really unacceptable and seems completely unavoidable in 2020.1.6f1. It's pretty sad when my extremely tiny projects take 30sec and my larger ones 2mins+.

I also use Rider and can start refreshing the asset database before I tab back over to Unity but that's not really a good enough solution.

 

I have exactly the same issue, not really sure what I can do about it. I'm using 2019.4.14 and every time I add a file/delete a file/compile it hangs for a good 30 seconds, this is for a tiny project that I just started. This didn't happen in 2019.3 with a huge project (20+ GB), even though I was using Asset Database V2 there as well. Maybe it's because I switched it from V1 to V2 instead of starting in V2 already.

I was able to capture the freeze using the Profiler after deleting a .txt file from my project, here's what it said:



Is it causing something to run in Visual Studio? That seems to be where the majority of the hold up is. It seems odd that this empty .txt file is causing all this work, it's not a script or anything so not sure why VS needs to do anything.

For clarity, I'm also running this new project in URP version 7 I believe, with minimal packages. I'm using the Input System and an SSAO package I got from the store. I'm including this info because I thought maybe the .dll files used by one of those packages could've been causing a slowdown, but I'm not totally sure. There are no errors so I'm not sure how they could be affecting it.

I also captured a freeze that happened from saving changes to a CS file from VS, before compilation. This one is a little different:



EndReloadAssembly seems to be the culprit here. For some reason, beforeProcessingInitializeOnLoad, afterProcessingInitializeOnLoad, and ReloadAssemblies each seem to take 35,000 ms:



Any ideas would be helpful here, I'm at a complete loss and really can't see myself building this project without a fix. In my last project changes like file deletion would take a few seconds at most, here it takes 30+, database would take a second or two before compiling, here it takes 90+ seconds. Sometimes it happens and sometimes it doesn't, I really have no clue why the behavior is inconsistent sometimes. I believe restarting Unity may be what causes the times to go back to normal.

 

rider wont solve,it is completely a hardware bug.And unity can not find and recreate it!

 

Been getting this as well, I'm in 2020.2. I have a powerful machine since I develop. Sometimes what would happen immediately will take at least 3 minutes.

I can wait for the issue to be fixed, but it's a bit annoying. If nothing can be done I'll just grin and bear it, then.

 

This seems to take between 10 to 15 minutes every time unity starts (the projects folder is 6.5gb). Given that I can probably expect the editor to crash 3 or 4 times a day on an average day (2020.1.17f1) it gives me a lot of time on my hands. Have tried updating to 2020.2.x but there appears to be anumber of breaking changes updating to that that we're not prepared for right now

 

Right, I figured it out shortly after I posted that VS was considering the .txt extension which is why it was going through that long chain. I also realized that the slowdown was only happening for me after I leave my project open for some time, and that restarting Unity seems to make it go back to normal, i.e. the wait time is as expected, only a second or two.

I actually ran a deep profile and have more details over on this thread, the deep profile specifically is on the post from December 7th. I also submitted a bug report and have been talking to the QA rep about reproduction steps; the case number is 1299648. I also thought ahead and saved out the deep profiler info on one of these slowdowns. I still have the file but the size is over 1 GB so if you have a secure way to transfer it and think it would help I can send it over.

To recap on my issue (can't say for certain what everyone else is facing) when I leave Unity open for a few hours is when it enters this "bugged" state and I end up with these massive slowdowns every time I compile or change a VS managed file. This is in Unity 2019.4.14f1. When I restart Unity the problem goes away so it's not as drastic as I first thought, but still clearly a bug that could get annoying and might be even worse on slower hardware.

 

I'm still not exactly certain what causes the issue in the first place but I believe I can say with some confidence now that updating Visual Studio 2017 seems to have solved the problem. Before I was using something around version 15.0 (I think build 26228.30), and when I did a manual update for VS 2017 it became version 15.9. I haven't once encountered the problem since that update which was about a month ago. Except for one occasion where Play mode was still slow to start even after a few tries, yet I'm pretty sure the .txt file deletion times were as expected. So that might've been a different issue or just my HDD being slow which tends to happen sometimes. Restarting Unity fixed it and I haven't experienced slow Play mode since then.

I guess there is just something about the older VS version which caused the Refresh to hang sometimes(?) although why it only happened after waiting with Unity open is a mystery still. As with versions of Unity I was also hesitant to update VS which might've broken my setup. Somehow I didn't consider that VS 2017 still got patches and I wouldn't be forced to go to 2019 or something like that. Updating from within VS went smoothly and even fixed up some other issues I was having with Unity like this problem. So I would recommend moving to a later patch if anyone experiences this slowdown. Hopefully that helps someone in the future! It was a pain to track down a fix but it ended up being pretty simple in the end.

Not sure if this will solve others' issues with refresh and slow loading times, but it's worth a try. I'm sure investigating the exact connection here might be helpful to improve Unity stability down the line, in the case something like this happens again. But with a solution for the end user available I would say it's not as high of a priority, though I'll let the Unity devs decide. Thanks again to everyone that weighed in and offered suggestions, it was a huge help and I'm extremely appreciative of the support!

 

I got exactly same behaviour by jumping to 2020.3.x LTS.

Starting the project takes 5 to 10min due to Refresh Asset Database on every startup.
Any news here?



Project startup before was under 1min.

 

I assume the Editor still makes logs during this process right? Can you check the Editor log file and see if there is anything odd happening during the refresh? Log file location can be found here.

The reason I'm curious is because while my update of VS has improved the situation, I still seem to have encountered a slow refresh while working in the project. When I checked the Editor logs I found that I was getting spammed with a ton of licensing errors, then after restarting Unity the errors would go away and the refresh speed would be back to normal. I wonder if you also have licensing error problems, then maybe we could track it down to the same underlying function call.

 

Hi. Thanks for support.
I openend the project with the profiler enabled.

Time to open the project:
Fast SSD: around 10min
Normal SSD: more than 20min


Unity Editor RAM use goes up to 48GB during opening and
falls back to 9.8GB when project is open

Then i added the
com.unity.performance.profile-analyzer package
to the project.

I stuck on

"That will tell the profiler to begin capturing startup, and when complete the data will be available on the profiler. We can go from there, especially if this is happening every time you open the editor!"

The Pull Data is not available after startup.
The Load button is available but where i can find the profiler capture from startup?

 

Ok. Thanks.
That did the job.
Here is the profiler content from a very fast SSD project startup.
Unity 2020.3.4f1

Is the Shader.CompileGPUProgram (Pos.3) the problem here? Is it done at every startup again?
This PC has 2x NVIDIA 3090 what would explain why this startup is much faster than on a normal



some fullscreen views






Same project.
PC with normal SSD and RTX 2080.
Unity 2020.3.3f1

 

Thanks for looking into.
A while a go i was surprised that a TextMesh Pro package upgrade to latest fired a most complete reimport of the assets. Only as side information.

The first capture came from a PC with fastest hardware components we have available.
128GB RAM, 2xNVIDA RTX 3090.....
Unity 2020.3.4LTS
I switched to this for testing because i was not able to work on the project anymore on our average pc´s.


Here is the capture from a normal high end working pc which shows a different behaviour.
But i have no profiler knowledge at all.
Same project.
But Unity 2020.3.3LTS
This is the average PC configuration we use and shows up my real world problem.
56GB RAM, Nivdia RTX 2080

1177918.0002ms = 19.63196667min startup
done at every startup




after update to
2020.3.4f1 from 2020.3.3f1
Textmesh 3.0.5 from 3.0.4
faster but still the long Shader.CompileGPUProgram,

 

Hi.

WorkingPC

Size of ´Library/ShaderCache/EditorEncounteredVariants´= 795KB

Size of `Library/Shadercache" =




Reopening after deletetion.

37853ms = 0,63088333min

Nice. Startup time is 0.6 min instead of 19min before.
Size of ´Library/ShaderCache/EditorEncounteredVariants´= 9KB now




Second test with project on the fast PC shows same speed now.




Do you know what is bloating up the
Library/ShaderCache/EditorEncounteredVariants'
over time?
DefaultBuild o9r rebuild of the Addressables or the general build at all?

I use one
HDRenderPipelineAsset
and
only one
Quality 'High` HDRP setting with this HDRenderPipelineAsset

for the main scene who is allowed loading 5 addressable scenes.(cars and tracks)
and other working scenes in the project.

Overall project size is more than160GB.

 

I did a build of my my main scene with addressables.
Library/ShaderCache/EditorEncounteredVariants´= 24KB then

Then i build a nother scene out of the project.
Library/ShaderCache/EditorEncounteredVariants´= 32KB then

i started with
Size of ´Library/ShaderCache/EditorEncounteredVariants´= 9KB after deletion

So i think that could be the problem. After every build the file grows?

Anywhere. Thanks for now. I could work again in my productive project.


Fun fact.
In another test project i migrated the MegaCity scene to 2020.3x and 2021.1.x releases.
There a inital build (further builds 10min) takes up to 6 hours and generates tons of shader variants.
I think for all subscenes and LOD´s again. But i know this is blood preview stuff and will be correted over time.

 

Thanks for the info and bringing me back to the track.
My project has reached a critical size i only can work with addressables and async loaded scenes with default addressables setups.
I think i need to work more into Asset Bundles. For now it s pretty new to me.

Would be nice if there is some more information or best practices to handle big projects where async loaded scenes like cars and tracks where each easily exceed the old 4GB limit.
After using addressables this worked for me. But i think there is more manual work needed now because project size is growing fast.

 

I did several new project starts and build content of adressables and builds on both projects with the same scene.

there is steady raise from
Library/ShaderCache/EditorEncounteredVariants´= +4t o +18 KB pro action

Because i do it often per day this would explain the inital size of 795KB i deleted.

In one or two weeks would be back there.
However. Now i know that deleting it solves my problem for now.