Search Unity

  1. Welcome to the Unity Forums! Please take the time to read our Code of Conduct to familiarize yourself with the forum rules and how to post constructively.
  2. Dismiss Notice

Loading a project went from <10 minutes to over 8 hours in a week

Discussion in 'Editor & General Support' started by firozjokhi, Jan 9, 2022.

  1. firozjokhi

    firozjokhi

    Joined:
    Nov 11, 2020
    Posts:
    17
    Hello,

    Windows 10
    Unity 2021.1.14f1


    Up until last month my Unity game project was loading up under 10 minutes. With the sale I purchased some assets and tried them out. No issues even then. Now for the past week my project load times went from <10 minutes to a couple of hours and at time over 8 hours.

    Once the project loads, I see no issues. Everything runs fine, no errors anywhere. But closing and opening the project again leads to same issues.

    I tried deleting the entire Library folder. It helps the first time after that by keeping loading under 2-3 hours. But the next time goes back to over 6 hours again.

    The Importing progress window seems to get stuck on some of these forever -
    "Importing"
    "Waiting for CacheServer"
    "Importing (iteration 2)"


    Any help or recommendations would be highly appreciated.

    Thank you!
     
  2. firozjokhi

    firozjokhi

    Joined:
    Nov 11, 2020
    Posts:
    17
    Can anyone give me any suggestions please?
     
  3. Kurt-Dekker

    Kurt-Dekker

    Joined:
    Mar 16, 2013
    Posts:
    36,713
  4. altepTest

    altepTest

    Joined:
    Jul 5, 2012
    Posts:
    1,050
    we don't have a solution, is from spring last year that I've had this issue.

    I've think I've fix it but on the other hand I'm not actually working as before on unity as this issue really worn me down. lost months behind this and was not fun.

    so maybe is fixed or maybe I've not hit yet with the problem
     
  5. firozjokhi

    firozjokhi

    Joined:
    Nov 11, 2020
    Posts:
    17
    Oh, that's unfortunate to hear! So this is a known issue and reported someplace then?

    How did you manage to work around this?
     
  6. firozjokhi

    firozjokhi

    Joined:
    Nov 11, 2020
    Posts:
    17
  7. JeffDUnity3D

    JeffDUnity3D

    Unity Technologies

    Joined:
    May 2, 2017
    Posts:
    14,446
    Compare to a new/empty project. Add those assets one at a time to see which one may be causing the long load time.
     
  8. altepTest

    altepTest

    Joined:
    Jul 5, 2012
    Posts:
    1,050
    no, is complex issue where one of the symptom is long loading times. but your are bit extreme. I've had 10-15 minutes of loading time.
     
  9. firozjokhi

    firozjokhi

    Joined:
    Nov 11, 2020
    Posts:
    17
    Unfortunately, at this time, there's probably like 30 assets in my project. But until a couple weeks ago there was no issue. So not sure if any of the assets are the cause.

    Is there a log somewhere I can check to see what is causing these reimports?
     
  10. JeffDUnity3D

    JeffDUnity3D

    Unity Technologies

    Joined:
    May 2, 2017
    Posts:
    14,446
    You'll want to check the Editor logs https://docs.unity3d.com/Manual/LogFiles.html
     
    Last edited: Jan 13, 2022
  11. firozjokhi

    firozjokhi

    Joined:
    Nov 11, 2020
    Posts:
    17
    upload_2022-1-13_9-51-50.png
    It keeps getting worse...
     
  12. JeffDUnity3D

    JeffDUnity3D

    Unity Technologies

    Joined:
    May 2, 2017
    Posts:
    14,446
    Editor log please. My guess that it's likely one of the assets causing it, you'll want to try to find out as described. Use a binary search/bisection approach (divide and conqueror), should take about an hour I might guess. It would be good at least to rule it out.
     
  13. altepTest

    altepTest

    Joined:
    Jul 5, 2012
    Posts:
    1,050
    How many files are inside the entire project folder? Not only in the assets. The folder that have the assets and library and the rest. How many files there are?

    And tell us if you have the project on normal HD. Not SSD, the old type with the rotating metal disk
     
  14. firozjokhi

    firozjokhi

    Joined:
    Nov 11, 2020
    Posts:
    17
    @JeffDUnity3D it won't let me attach the editor log file saying attachment is too large. It's a 20MB file.

    @dorusoftware 50GB project with 8000 folders and 194,000 files. Definitely massive, but to reiterate, all this was loading fine up until 2 weeks ago. So something got messed up in the project.

    Also, Unity runs off an SSD, but the project files are on 7200rpm drive.
     
  15. JeffDUnity3D

    JeffDUnity3D

    Unity Technologies

    Joined:
    May 2, 2017
    Posts:
    14,446
    Got it! The large size of the Editor log might be just what you are looking for. Check and see if there is a single repeating message for example. Look for other errors in the file and report them here. Another thing to try is to exit Unity and delete the log file. Then open the project again, and then check the log. Do you expect your project to have 8,000 folders? That's a lot. Why so many?
     
  16. firozjokhi

    firozjokhi

    Joined:
    Nov 11, 2020
    Posts:
    17
    I did not find anything obvious in there, but I am not sure what even to look for. Seems like it takes a long time (re)importing every single asset in my project?

    I have about 200 asset packages from the unity store added to my project. I did not know of a better way to do it than add something to my project, and then pick 1-2 things from the package to my scene. Each of those packages has a lot of folders & subfolders, so I can imagine the count blowing up.

    Do we support pastebin or dropbox or something here, so I can share my log file?
     
  17. JeffDUnity3D

    JeffDUnity3D

    Unity Technologies

    Joined:
    May 2, 2017
    Posts:
    14,446
    Are you actually using all those assets? If not, then remove them. You can use Google drive, and we can exchange the email address in a private message on here, I'll take a look if I can. Basically you would look for the same (or very similar) line that repeats itself thousands of times, for example. Even if you don't know what it means. And the words "error" and "exception" for example. Often the errors are at the very beginning or end of the file also.
     
  18. firozjokhi

    firozjokhi

    Joined:
    Nov 11, 2020
    Posts:
    17
    So that's one problem I have - I have been adding these packages to my project, using a couple of assets from those packages, maybe not even using some. I have no idea which of these packages are actually used in any of my scenes at this point.

    Also, I underestimated - there's 82 packages added to my project. I will PM you a dropbox link to my editor log. Thank you!
     
  19. altepTest

    altepTest

    Joined:
    Jul 5, 2012
    Posts:
    1,050
    The issue you have is related with your hard drive fragmentation.

    For each of the file you have imported in your project (like for example assets from store) unity creates a companion file that keeps information about that file. So the files in the project are at least double the imported assets files.

    for some reason I don't understand unity will try accessing/recreating all these companion files at startup (or randomly while working on the project) Or maybe is a justified operation. I don't know, thing is, that it happens.

    Anyway, these companion files are very small in size and the automatic file system of the operating system will write it where it finds empty spaces on your hard disk. And if these small companion files get fragmented they will split in smaller fragments and will be written all over the place on your hard disk.

    because you have a lot of files and because the disk and the magnetic head needs to physically rotate and move to find each of those files it takes time. If the files are fragmented the access time gets even worse since multiple locations needs to be read just for one single small file.

    I wish you good luck trying to figure out the log file but I don't think there is any clue there. unit will write in the log that it needs to reimport these assets or something, and it just takes a lot of time.

    If there is no fix from unity team after looking at the log file, I suggest doing what I did, moving the entire project folder to an ssd. And off course remove assets that you are not using. If you use assets, at least delete all the various demo folders that many authors include, if you don't need them.

    Actually be very careful in moving the project folder. Best to copy not cut. And it will take a lot of time because all those files need to be accessed so the HDD needs to spin and spin. You could delete the library folder just to cut on the copy time, the files will be recreated after on the SSD.

    If you want to check the fragmentation of your disk you can use a program like defraggler. You can check what files are fragmented and where are located. You will see all those small files all over the place.

    Here is the state of fragmentation on a project I have on an SSD. The project have 4200 artefacts files ( the small companion files)

    Annotation 2022-01-14 021826.png

    In the top part we have a representation of where these artefacts files are located. The red square are fragmented files. With an average of 3 fragments for each file. So 15000 locations on disk.

    Each of those files are located at random locations on disk so when we need to access them the disk will need to rotate and the magnetic head will need to move at all these locations.

    This next screen shows one of the artefact file selected. is small and has 5 fragments. they are not near eachother. but in different disk locations.

    now the SSD is fast and the number of files is relatively small so the startup time is not noticeble.

    Annotation 2022-01-14 0218262.png

    but with the huge number of files you have that is a problem.

    when I've fixed my issue I've moved the various projects with 100k files each, to an empty SSD. I didn't check till now, but I notice that neither of the artefacts files in those project are fragmented. So they kept quite well till now. When moving them each got in their own place and remained in one piece.

    I suspect that having huge amount of small files fragmented on an SSD may also create issues, not so extreme but the slowdown can be noticeable but will not be so worse as with a normal HDD.

    So if you can, use and empty SSD.

    What may had happen in your case is that for some reason unity triggered the reimport of ALL the files and recreated the library folder, and this have increased the fragmentation of all the files that previously where in one piece. This reimport happens if you add new assets. For some reason it triggers the reimport of ALL assets, not only those that are added new. I've seen this with various assets from the store, not with all of them.

    So before this event your project worked fine, because the files where mostly in one piece, and contiguous on the disk, resulting in faster access.

    You have imported an asset from store, this reimported all assets and created/updated all the artefacts(companion) files again, and all of them got fragmented and spread all over the disk.

    Like grains of sands.

    Deleting the library folder will do nothing because the process above just gets repeated. De-Fragmenting the disk is impossible, I've tried, it takes ages with all those small files.You could try deleting the library folder and de-fragmenting the disk and then let unity reimport the files again. You could get lucky and have the companion files remain in one piece. But what if the above happens again and everything gets messed up again?
     
    Last edited: Jan 14, 2022
  20. firozjokhi

    firozjokhi

    Joined:
    Nov 11, 2020
    Posts:
    17
    Hi, thanks for the very informative post.

    All my drives are set to defrag weekly, and they all look good to me -
    upload_2022-1-13_19-58-37.png

    I will try and make room on my SSD to copy the project folder and see if that makes a difference. I try to keep my SSD mainly for the OS and applications.
     
  21. firozjokhi

    firozjokhi

    Joined:
    Nov 11, 2020
    Posts:
    17
    Copied the project to my SSD (which still has 50GB space left), and it's been over 10 hours and still loading.

    Surprisingly, the log for this is very short. I am hoping this is more helpful to diagnose.
     

    Attached Files:

    Last edited: Jan 14, 2022
  22. altepTest

    altepTest

    Joined:
    Jul 5, 2012
    Posts:
    1,050
    what software are you using for defrag the drives?

    how long it took to copy the files over to the SSD? if was fast then my theory above may not apply. When I've copied the files it took same amount as unity loading the files.

    may also be that unity is probably stuck in a loop of reading some files over and over so why it takes for ever to open.

    Did the project actually opens? And if yes, you can work without any troubles?

    If you can open the project even after hours, do this. Create a new empty scene then open that scene and save the project. So next time it opens it will open into this new empty scene. See if it makes any difference
     
  23. firozjokhi

    firozjokhi

    Joined:
    Nov 11, 2020
    Posts:
    17
    1. Using windows 10 defrag (built in)
    2. Didn't pay attention to it but It took about a couple of hours to copy
    3. In the past couple of weeks I haven't ever completed loading the project (Once it's past 9-10 hours I just end task). It's only when I clean up the library and load the project is when it completes in a couple of hours and then I can work on it without issues.

    update-loading the project from the SSD took the same time as loading from the regular drive
     
    Last edited: Jan 14, 2022
  24. firozjokhi

    firozjokhi

    Joined:
    Nov 11, 2020
    Posts:
    17
    This turns out to be the culprit. If I empty out this folder, my project loads within minutes. Not sure what it is or how it got created, but when I add all the packages into my new project there is no folder such as this one.

    upload_2022-1-15_13-29-9.png
     
  25. altepTest

    altepTest

    Joined:
    Jul 5, 2012
    Posts:
    1,050
    mystery :D

    but if it works now, that is great
     
  26. firozjokhi

    firozjokhi

    Joined:
    Nov 11, 2020
    Posts:
    17
    Yes, but so glad it works now. Thank you all for the help and guidance!