Where's the fast import?

I've heard a lot about a supposed new feature meant to make things import more quickly. Well, I'd sure like to know where it is! I dragged a single (admittedly quite large at 3.5 GB) FBX file into my project, and Unity has been spinning for over 2 hours now on the import process!

Using Unity 2019.3.10f1. I'm pretty sure the faster import feature is supposed to be in this version. Did I miss something? Does it need to be enabled in some option somewhere? I've been locked out of development for hours now because the importer still is not running quickly and still is not running in the background like long-running tasks should. (Seriously, we've had async/await for how many years now? Having long-running tasks block the UI is just absurd in 2020!) How do I fix this?

 

The feature is called Asset Import Pipeline V2. If your project was originally created in a version older than 2019.3.x, you have to tell it to switch to V2.

https://blogs.unity3d.com/2019/10/3...lid-foundation-for-speeding-up-asset-imports/

Import performance also is significantly affected by your hardware performance. If you're using a regular HDD or have a particularly slow or old computer, you're get significantly slower import performance.

Spoiler

I've seen quite a lot of bug fixes related to Asset Database V2, which goes along with Asset Import Pipeline V2, in the release notes for versions released after the one you're currently using. I'd consider upgrading to the latest 2019.4.x LTS if you're going to use this feature. But that's just me.

 

New computer, (bought it this year,) good hardware, SSD. The project is already on V2. I'm currently updating to the latest 2019.4 LTS. I sure hope that fixes it, because this is completely bizarre. It does not take 2+ hours to read 3.5 GB of data from an SSD and copy it into some new format in the Library folder. It shouldn't even take 2 minutes!

 

After upgrading the project, I tried the import again and left it running overnight.

10 hours later, it's still going!

Unity, your importer is badly broken. First off, the process of upgrading the project to the new version alone took over 4 hours, reimporting every single file even though they were already imported. There is no reason for that. A properly coded upgrade process would have taken maybe 5-10 minutes if that long, by applying filters to only look at files whose import format had changed between the version I was upgrading from and the version I'm upgrading to, rather than indiscriminately reimporting everything whether it needs it or not.

And now it's been spinning its wheels for over 10 hours on one single file, to the point where I don't know if it's hit a bug or just proceeding at a glacial pace. All I know is, there's no excuse for it. It should have been done over 10 hours ago.

I've imported packages this large from the Asset Store before and had it take a few minutes. But when it's one big FBX file, everything grinds to a halt? Sounds like your code is tripping over Shlemiel the painter’s algorithm somewhere.

 

What's the processor? Is the SSD SATA or NVMe? What percentage of your system memory is being consumed?

 

What's the reasoning behind not being on latest LTS?

 

Simply that the project wasn't there at the time. I updated to the latest LTS, as I noted above, and it didn't improve.

 

Windows reports the CPU as "AMD Ryzen 7 2700X Eight-Core Processor, 3700 Mhz, 8 Core(s), 16 Logical Processor(s)." I think the drive is NVMe but I'm not sure. System memory consumption gets up to around 80% a few minutes into the import process, then steadily decreases.

And none of this actually matters. As I mentioned above, I've imported Asset Store packages larger than this on this same hardware. Plenty of times in fact. It never took 10+ hours. It rarely even takes 10+ minutes! It's not the hardware; there is a problem with the importer specifically that's causing it to hang on processing a large FBX file.

This is a well-understood category of problems in computer science, known colloquially as "Shlemiel the painter's algorithm," (see the link I posted above for where the term comes from,) where simplistic data-processing code that did not take scale into account ends up needlessly taking exponentially longer on large-scale data processing. If I had access to the source code I'd track it down and fix it myself, and I'd fix project upgrading while I'm at it! Alas, Unity in their infinite wisdom has chosen not to make that option available to its community, to all of our detriment.

 

On the contrary you wouldn't be able to fix it with just access to the Unity source code as they are using the official Autodesk FBX SDK which is closed source too. Fortunately there is a way around both of these limitations. You can create a custom asset importer.

https://docs.unity3d.com/Manual/ScriptedImporters.html

 

From the linked manual page:

Also, that wouldn't help with fixing the broader reimport-everything problem with project upgrades.

Scripted importers are definitely cool to know about. They're just not useful in this particular case.

 

Sorry, I just realized that you're saying this is a 3.5 GB FBX. That is insanely large. Usually when people are talking about very large FBX files imported into Unity, they are still talking about 100-200 MB.

This comment from a Unity dev in another thread says the FBX importer used in Unity is still very old:
https://forum.unity.com/threads/upg...ier-uniqueness-violation.720605/#post-4823378

But >2GB FBX support wasn't even added to the official autodesk FBX SDK until just a few years ago.
https://help.autodesk.com/view/FBX/...GUID_2ADB9BCE_15EA_485F_87C2_32D43F2D219D_htm

So between those two bits of info, I'd put my money on Unity not actually supporting >2GB FBX files properly. Either because they aren't supported at all, or they aren't doing any QA with that large a file size. (I'd venture a guess they added support for the newer block header format, but are testing it with small file sizes for expediency, but that's just my QA experience talking, no actual incite into what Unity is doing)

 

What Joe said. Also stuff like "Generate Lightmap UVs" which if I'm not mistaken are on by default, can easily quadruple (or more) the import time.

 

I believe that most software that can read fbx most likely will die while trying to process this file.

You need to rethink your import process.

 

I have worked with very large (multi-gig) fbx files before and have never managed to load them in anything other than 3DS Max, despite how hard I try to avoid touching Autodesk stuff. Even then, it is painful and slow to work with.

I would suggest trying to split the fbx into several smaller parts, and reassemble it in Unity.

 

Unity determines file types solely off of the extension and not the actual contents of the file. You can very easily see this by taking a random file from your computer, giving it the .FBX extension, and trying to import it. Unity will respond with "ImportFBX Errors: Couldn't read file filename.fbx". Tested on Unity 2020.1.

Therefore you should be able to just create a custom importer expecting a different extension than the official importer, and add that extension to your assets before you import them into Unity.