Loading a project went from <10 minutes to over 8 hours in a week

Hello,

Windows 10
Unity 2021.1.14f1

Up until last month my Unity game project was loading up under 10 minutes. With the sale I purchased some assets and tried them out. No issues even then. Now for the past week my project load times went from <10 minutes to a couple of hours and at time over 8 hours.

Once the project loads, I see no issues. Everything runs fine, no errors anywhere. But closing and opening the project again leads to same issues.

I tried deleting the entire Library folder. It helps the first time after that by keeping loading under 2-3 hours. But the next time goes back to over 6 hours again.

The Importing progress window seems to get stuck on some of these forever -
"Importing"
"Waiting for CacheServer"
"Importing (iteration 2)"

Any help or recommendations would be highly appreciated.

Thank you!

 

Can anyone give me any suggestions please?

 

https://forum.unity.com/threads/editor-progress-bar-stalls-request-for-bug-reports.1137817/

 

we don't have a solution, is from spring last year that I've had this issue.

I've think I've fix it but on the other hand I'm not actually working as before on unity as this issue really worn me down. lost months behind this and was not fun.

so maybe is fixed or maybe I've not hit yet with the problem

 

Oh, that's unfortunate to hear! So this is a known issue and reported someplace then?

How did you manage to work around this?

 

This is related to editor stalling AFTER the project is already loaded, right?

My issues is loading the project itself takes 2-8 hours.

 

no, is complex issue where one of the symptom is long loading times. but your are bit extreme. I've had 10-15 minutes of loading time.

 

Unfortunately, at this time, there's probably like 30 assets in my project. But until a couple weeks ago there was no issue. So not sure if any of the assets are the cause.

Is there a log somewhere I can check to see what is causing these reimports?

 

It keeps getting worse...

 

How many files are inside the entire project folder? Not only in the assets. The folder that have the assets and library and the rest. How many files there are?

And tell us if you have the project on normal HD. Not SSD, the old type with the rotating metal disk

 

@JeffDUnity3D it won't let me attach the editor log file saying attachment is too large. It's a 20MB file.

@dorusoftware 50GB project with 8000 folders and 194,000 files. Definitely massive, but to reiterate, all this was loading fine up until 2 weeks ago. So something got messed up in the project.

Also, Unity runs off an SSD, but the project files are on 7200rpm drive.

 

I did not find anything obvious in there, but I am not sure what even to look for. Seems like it takes a long time (re)importing every single asset in my project?

I have about 200 asset packages from the unity store added to my project. I did not know of a better way to do it than add something to my project, and then pick 1-2 things from the package to my scene. Each of those packages has a lot of folders & subfolders, so I can imagine the count blowing up.

Do we support pastebin or dropbox or something here, so I can share my log file?

 

So that's one problem I have - I have been adding these packages to my project, using a couple of assets from those packages, maybe not even using some. I have no idea which of these packages are actually used in any of my scenes at this point.

Also, I underestimated - there's 82 packages added to my project. I will PM you a dropbox link to my editor log. Thank you!

 

The issue you have is related with your hard drive fragmentation.

For each of the file you have imported in your project (like for example assets from store) unity creates a companion file that keeps information about that file. So the files in the project are at least double the imported assets files.

for some reason I don't understand unity will try accessing/recreating all these companion files at startup (or randomly while working on the project) Or maybe is a justified operation. I don't know, thing is, that it happens.

Anyway, these companion files are very small in size and the automatic file system of the operating system will write it where it finds empty spaces on your hard disk. And if these small companion files get fragmented they will split in smaller fragments and will be written all over the place on your hard disk.

because you have a lot of files and because the disk and the magnetic head needs to physically rotate and move to find each of those files it takes time. If the files are fragmented the access time gets even worse since multiple locations needs to be read just for one single small file.

I wish you good luck trying to figure out the log file but I don't think there is any clue there. unit will write in the log that it needs to reimport these assets or something, and it just takes a lot of time.

If there is no fix from unity team after looking at the log file, I suggest doing what I did, moving the entire project folder to an ssd. And off course remove assets that you are not using. If you use assets, at least delete all the various demo folders that many authors include, if you don't need them.

Actually be very careful in moving the project folder. Best to copy not cut. And it will take a lot of time because all those files need to be accessed so the HDD needs to spin and spin. You could delete the library folder just to cut on the copy time, the files will be recreated after on the SSD.

If you want to check the fragmentation of your disk you can use a program like defraggler. You can check what files are fragmented and where are located. You will see all those small files all over the place.

Here is the state of fragmentation on a project I have on an SSD. The project have 4200 artefacts files ( the small companion files)



In the top part we have a representation of where these artefacts files are located. The red square are fragmented files. With an average of 3 fragments for each file. So 15000 locations on disk.

Each of those files are located at random locations on disk so when we need to access them the disk will need to rotate and the magnetic head will need to move at all these locations.

This next screen shows one of the artefact file selected. is small and has 5 fragments. they are not near eachother. but in different disk locations.

now the SSD is fast and the number of files is relatively small so the startup time is not noticeble.



but with the huge number of files you have that is a problem.

when I've fixed my issue I've moved the various projects with 100k files each, to an empty SSD. I didn't check till now, but I notice that neither of the artefacts files in those project are fragmented. So they kept quite well till now. When moving them each got in their own place and remained in one piece.

I suspect that having huge amount of small files fragmented on an SSD may also create issues, not so extreme but the slowdown can be noticeable but will not be so worse as with a normal HDD.

So if you can, use and empty SSD.

What may had happen in your case is that for some reason unity triggered the reimport of ALL the files and recreated the library folder, and this have increased the fragmentation of all the files that previously where in one piece. This reimport happens if you add new assets. For some reason it triggers the reimport of ALL assets, not only those that are added new. I've seen this with various assets from the store, not with all of them.

So before this event your project worked fine, because the files where mostly in one piece, and contiguous on the disk, resulting in faster access.

You have imported an asset from store, this reimported all assets and created/updated all the artefacts(companion) files again, and all of them got fragmented and spread all over the disk.

Like grains of sands.

Deleting the library folder will do nothing because the process above just gets repeated. De-Fragmenting the disk is impossible, I've tried, it takes ages with all those small files.You could try deleting the library folder and de-fragmenting the disk and then let unity reimport the files again. You could get lucky and have the companion files remain in one piece. But what if the above happens again and everything gets messed up again?

 

Hi, thanks for the very informative post.

All my drives are set to defrag weekly, and they all look good to me -


I will try and make room on my SSD to copy the project folder and see if that makes a difference. I try to keep my SSD mainly for the OS and applications.

 

Copied the project to my SSD (which still has 50GB space left), and it's been over 10 hours and still loading.

Surprisingly, the log for this is very short. I am hoping this is more helpful to diagnose.

 

what software are you using for defrag the drives?

how long it took to copy the files over to the SSD? if was fast then my theory above may not apply. When I've copied the files it took same amount as unity loading the files.

may also be that unity is probably stuck in a loop of reading some files over and over so why it takes for ever to open.

Did the project actually opens? And if yes, you can work without any troubles?

If you can open the project even after hours, do this. Create a new empty scene then open that scene and save the project. So next time it opens it will open into this new empty scene. See if it makes any difference

 

1. Using windows 10 defrag (built in)
2. Didn't pay attention to it but It took about a couple of hours to copy
3. In the past couple of weeks I haven't ever completed loading the project (Once it's past 9-10 hours I just end task). It's only when I clean up the library and load the project is when it completes in a couple of hours and then I can work on it without issues.

update-loading the project from the SSD took the same time as loading from the regular drive

 

This turns out to be the culprit. If I empty out this folder, my project loads within minutes. Not sure what it is or how it got created, but when I add all the packages into my new project there is no folder such as this one.

 

mystery

but if it works now, that is great

 

Yes, but so glad it works now. Thank you all for the help and guidance!