Official Asset Importing speedups in 2021.2 - call for feedback

Full project import would be delete library -> open? Or right click -> reimport?

 

These are such important improvements. You cannot imagine how happy that makes me. Thanks a lot Aras and team!

 

Will startup time optimize backport to 2020？
Start a project in HHD also quite slow.....

 

As someone with a threadripper who has been struggling with how slow the asset importer is on 2020, this is a great improvement. WP Aras & team. Will definitely be giving it a crack soon, and will put numbers here.

 

This is very exciting. I'm especially happy to see the editor becoming more multicore friendly. Good job!

 

How can I get the startup time? I'm trying to see if it's on the Editor.log, but there are way too many timings there, and I'm not sure which one is the one you want. Is this it?
[Project] Loading completed in 14.133 seconds

 

Tried the SimpleEditorLogParser, but it's giving me "Log is not in a supported format. Please implement new parsing for new format present in this log file." for Unity 2020.3.0 (current project version).

But from just reading the log, startup times are

2020.3.0f1:
Approximately 15.5 seconds

2021.2.0a20:
Approximately 14 seconds

BUT; I had to comment out some some code that happened at startup in 2020.3, since it caused errors on startup in 20201.2 (reflection code that interacted with things that have been removed). Specifically, I had two things that hooked up to DragAndDropService in order to handle some dragging and dropping. I don't know why it broke, but I don't have the time to update it now. I also had a custom UnityEditor.OverlayWindow that I booted at startup (apparently, as it was throwing exceptions on boot!). Those have been replaced now, so I had to comment that out as well.

I don't know how much cost running that code had, but it should probably take something like 1.5 milliseconds, not 1.5 seconds, so there's a good chance there's an improvement.

Reimport all time is next, I guess.

 

It seems like Assets->Reimport all is down from 2052 seconds to 994 seconds. Going to check deleted library as well

 

OMG this is amazing. Here my start-up numbers:

2021.1.11: 3m04s
2021.2.20: 0m36s

Assets folder: 102.448 files, 4544 folders, 43.3Gb

 

I couldn't use the profiler, because it would just try to import the profiler data for hours, so I ended up using the Editor Log for the timings. I also had to redo the imports a few times, because it would give me out of memory exception on 2020.3. The time of 2021.2 was in the middle of the editor log, so I don't know how accurate it is. I opened the project a few times until the startup time settled on a number (The first one right after an import was a little bit higher).

The project is mostly PNG files that would be loaded into sprites (~65K png files, ~ 9.5 GB). 2021.2 is set to 6 worker threads:

Startup:
2020.3.11f1: 13.135
2021.2.0a20: 10.869 20% faster

Import:
2020.3.11f1: 5,964 seconds (1:39 hours)
2021.2.0a20: 2618 seconds (43 minutes) 2.27x faster

This is on a 8700K 6 Cores, 32 GB of RAM and SSD. During the import the PC was completely on its own only doing this. I sometimes looked at the CPU/Disk usage. It was usually around 25% CPU usage and 140 MB/s disk usage.

 

Ok, I made a mistake. The numbers for the 2021.2 are for the default import with no worker count set, which is amazing. I'm gonna try again with 6 workers to see how it changes it.

 

With 6 workers it's now taking 100% CPU usage and >300MB/s disk usage. I'm still not super sure about the accuracy of these numbers as I still get out of memory errors at the end of the imports. I'll try to see if I can fix those.

1569 seconds (26 minutes) : 3.8x faster than 2020.3 1.6x faster than 2020.2 without worker threads

 

I guessed that it would be the case. I just wanted to go wild and see if I can get some more juice out of it. 3.8x still seems logical as the original was going around 25%.

The importing of packages is also multi-threaded now? I saw it goes over 70% when they're being processed.

Also, what's the difference between In-Process and Out of Process workers? Does it matter for performance or is it just as a test option?

 

That's an amazing news, because we had this issue for a long time. Our workaround always was to just open the project again until it reimports everything. The same thing happens when packing the sprite atlases.

 

I can't give exact figures, but my (rather large) project import went from "about a week" in 2020 to "about two days" in 2021.1 and "7 hours" in 2021.2a21; but in 2021.2b1 the import takes a little bit longer and never finishes due to an out of memory error. I don't use dx11, though.

 

My project is a little bit to big to send in and I always feel bad when the bug reporter tells me I don't send in enough information, but if you want to, I can search through the crash logs.

It doesn't happen with a specific asset, though. It always starts with

Code (CSharp):

1. d3d12: upload buffer was full! Waited for COPY queue for 5.151 ms.

several hundred times.
After that, I'll change to

Code (CSharp):

1. d3d12 : CreateCommittedResource 'BufferD3D12::CreateBufferResource() Buffer' (157200 x 1) format 0 failed (887a0005).
2.     Device Remove Reason (HRESULT): 0x887a0007
3.     Device Memory Stats:
4.         Local AvailableForReservation: 3749 MB, Budget: 7198 MB, CurrentUsage: 7170 MB, CurrentReservation: 64 MB
5.         Non-local AvailableForReservation: 4850 MB, Budget: 9444 MB, CurrentUsage: 9465 MB, CurrentReservation: 0 MB
6. d3d12: CreateResource for buffer failed.
7.  
8. d3d12: failed to create a buffer of size 157200
9. BufferD3D12 : Unexpected null accessor on Buffer

again several hundred times with different sizes (always ????? x 1, a one pixel wide texture?)

After those it'll close with:

Code (CSharp):

1. Could not allocate memory: System out of memory!
2. Trying to allocate: 4803120B with 16 alignment. MemoryLabel: Serialization
3. Allocation happened at: Line:69 in C:\buildslave\unity\build\Runtime\Utilities\dynamic_array.h
4. Memory overview
5.  
6. [ ALLOC_TEMP_TLS ] used: 44462896B | peak: 0B | reserved: 22347776B
7. [ ALLOC_MEMORYPROFILER ] used: 28778184B | peak: 28778184B | reserved: 33562624B
8. [ ALLOC_DEFAULT ] used: 1147040159B | peak: 0B | reserved: 1344184320B
9. [ ALLOC_TEMP_JOB_1_FRAME ] used: 0B | peak: 0B | reserved: 2097152B
10. [ ALLOC_TEMP_JOB_2_FRAMES ] used: 0B | peak: 0B | reserved: 2097152B
11. [ ALLOC_TEMP_JOB_4_FRAMES (JobTemp) ] used: 0B | peak: 0B | reserved: 134217728B
12. [ ALLOC_TEMP_JOB_ASYNC (Background) ] used: 0B | peak: 0B | reserved: 14680064B
13. [ ALLOC_GFX ] used: 12514140332B | peak: 0B | reserved: 12534845440B
14. [ ALLOC_CACHEOBJECTS ] used: 5146131464B | peak: 0B | reserved: 5191888896B
15. [ ALLOC_TYPETREE ] used: 1328609320B | peak: 0B | reserved: 1369440256B
16. [ ALLOC_PROFILER ] used: 757288B | peak: 850200B | reserved: 16777216B
17. [ ALLOC_PROFILER_EDITOR ] used: 0B | peak: 0B | reserved: 0B
18. Could not allocate memory: System out of memory!
19. Trying to allocate: 4803120B with 16 alignment. MemoryLabel: Serialization
20. Allocation happened at: Line:69 in C:\buildslave\unity\build\Runtime\Utilities\dynamic_array.h
21. Memory overview
22.  
23. [ ALLOC_TEMP_TLS ] used: 44462896B | peak: 0B | reserved: 22347776B
24. [ ALLOC_MEMORYPROFILER ] used: 28778184B | peak: 28778184B | reserved: 33562624B
25. [ ALLOC_DEFAULT ] used: 1147040159B | peak: 0B | reserved: 1344184320B
26. [ ALLOC_TEMP_JOB_1_FRAME ] used: 0B | peak: 0B | reserved: 2097152B
27. [ ALLOC_TEMP_JOB_2_FRAMES ] used: 0B | peak: 0B | reserved: 2097152B
28. [ ALLOC_TEMP_JOB_4_FRAMES (JobTemp) ] used: 0B | peak: 0B | reserved: 134217728B
29. [ ALLOC_TEMP_JOB_ASYNC (Background) ] used: 0B | peak: 0B | reserved: 14680064B
30. [ ALLOC_GFX ] used: 12514140332B | peak: 0B | reserved: 12534845440B
31. [ ALLOC_CACHEOBJECTS ] used: 5146131464B | peak: 0B | reserved: 5191888896B
32. [ ALLOC_TYPETREE ] used: 1328609320B | peak: 0B | reserved: 1369440256B
33. [ ALLOC_PROFILER ] used: 757288B | peak: 850200B | reserved: 16777216B
34. [ ALLOC_PROFILER_EDITOR ] used: 0B | peak: 0B | reserved: 0B
35. UnityEngine.StackTraceUtility:ExtractStackTrace ()
36. UnityEditor.ModelImporterPostProcessor:OnPostprocessAllAssets (string[],string[],string[],string[])
37. System.Reflection.RuntimeMethodInfo:Invoke (object,System.Reflection.BindingFlags,System.Reflection.Binder,object[],System.Globalization.CultureInfo)
38. System.Reflection.MethodBase:Invoke (object,object[])
39. UnityEditor.AssetPostprocessingInternal:InvokeMethod (System.Reflection.MethodInfo,object[])
40. UnityEditor.AssetPostprocessingInternal:PostprocessAllAssets (string[],string[],string[],string[],string[])
41.  
42. [C:\buildslave\unity\build\Runtime\Allocator\MemoryManager.cpp line 1624]
43.  

If I can help without uploading a 600gb project folder, I'm open to anything.

Additionally, there is a problem with the MaterialPostProcessor in hdrp (all materials get reimported after update and fill up memory), but I can manually change the cs file and force unity to save assets every 100 or so materials, but this crash happens before the material reimport.

 

I'm also wondering the same, is there a difference?

 

Related to the importer in the 2021.2 I made an animation recently and while i was trying to import the model in unity guess what, the model was not visible Look.


In unity 2019 it just works, look!.

So how am I supposed to test animations and features on the new version when it can't import an FBX correctly?!.

 

1351441 I hope the 20 minutes of waiting are worth. This is how the animation should look.

 

Well that did do something, but the model is still broken.

 

@richardkettlewell @Aras Let me know if you have a fix for this. I started doing good progress on something and it would be a shame if i have to stop because of 1 little thing.

 

Did something change in the last few versions? I could import my project in a21, but I can't open it in any beta. Even opening the project again and again until it finishes (just like before 2021) doesn't work anymore.

Edit: Beta 8 worked. Everything imported several times faster than before and I had to restart the editor only once after import, not inbetween.

 

BTW. Is there any way to see some import log in case when the project can't be imported? I mean the cases when it fails / freezes on the import process and you can't access the editor. It would be helpful to see what caused the problem.

 

We are seeing a regression from 2020.3 where each time we launch Unity it re-imports one of our large .fbx files. The file had a warning about the animation rig which has been resolved, however the problem persists.

It's a large .fbx with multiple animations for our character models.

 

I finally had time to build a repro case for this so the bug (1377285) is in.

It requires the included .fbx and HDRP must be present.

 

This might have changed, but doesn't Unity still import certain file types by opening an external tool, e.g. "blender" for *.blend files, or "etcpak" and friends for textures and images? Opening a tool is mostly single-threaded, and that's the dominating part for smaller files. That's probably why starting starting more workers is still very beneficial, from the numbers in this thread.
If that one-by-one import is still a thing - and for *.blend files it certainly looks like it - changing that should be a very high priority. Shamelessly quoting my own comment from https://blog.unity.com/technology/t...olid-foundation-for-speeding-up-asset-imports:
Sadly, at least part of the initial import is still done single-threaded, and - in the case of *.blend files - by firing up a process (!) for each imported file. For a project with roughly 300 *.blend-files weighing in at 200 MB, the editor's import time is about 2 minutes, with simple batching this time drops to less than 10 seconds.

 

WTF speed up?? when importing UNITY spawn some thousand processes and my PC with 64GB RAM only swaping!

 

Task manager screen when importimg about 200 files to UNITY. Processes can be more than on screen but taskmanager starting about 30 minutes. btw clean instal UNITY 2021.2.4f1 on win 10. No matter what importing. More details I cannot give you, cuz PC completely hang on

.

 

Mini rant and warning for everyone else that wants to benchmark android projects: DON'T.

I tried benchmarking this as was requested by this thread. Our project on Android takes a while (potentially hours, even days) to compress so I thought it would be a good benchmark.

I did the test a few times (I literally spent days), and I kept getting very inconsistent results, which puzzled me.

Eventually I figured it out...

The dropdown where you select an ETC compressor is broken in 2020LTS.


It selects whatever. (you can keep selecting the same dropdown value, and the actual compressor changes).

This is fixed in 2021, but AFAIK there are no plans to back port it to LTS (I guess working dropdowns are considered a feature these days?)

So beware, if you are testing android projects, it's almost impossible to do an apples to apples comparison, since you can't really guarantee what kind of compression you are using in 20202.

 

Posting the thread I posted earlier here as it seems relevant - when everything works we are seeing pretty good import times on a large project with 40GB+ assets.

But frequently importing gets stuck in a loop and it can sometimes take multiple restarts to get it through.

More details here.

Thanks.

 

- importing any .asses with .cs .fbx .png .jpg ....
- I cannot expant parent processes - I have not cursor on screen - PC completely hang
- I set nothing - default clean unity install

- in UNITY 2021.1 all tested assets are imported fine - about 0.5 - 2 mins each
- I uninstall version 2021.2 - working witth 2021.1 without issues - don`t want warm my PC

btw same issue when deleting bunch of files from project, this issue look like on each added/deleted file unity start reimport.. so we have:
- adding/deleting 4 files:
- 1st file added (copyed to disk) - 1 process
- 2nd file added - 1 old process + 2 new processes = 3
- 3rd file added - 3 old + 3 new = 6
- 4st file - 6 old + 4 new = 10

- I have fast ssd disk - so unity may not be able to import old file when new one is added and start reimport from start (old file not marked as imported yet). I don`t know unity import logic (how offen is file system checked for changes and so on), so it is only my assumption.

 

@Aras On the topic of ScriptedImporter for parallel importing.

I maintain an importer that can potentially take a long time importing. Might there be a way for me to optimize loading times by having public API access to this parallel importing system?

I understand that it only supports the model importer and texture importer, but I would be very happy to try and utilize these same workers configured from project settings to be able to import for custom ScriptedImporter types.

Could this be a feature available in the future? Or is there some current API I can hook into to try out custom parallel importing?