Bug Import getting stuck after upgrading to 2021.2.2f1

Since upgrading to 2021.2.2f1 and the introduction of multi-process importing, we are seeing issue where imports get stuck in a cycle of waiting for workers that eventually times out. It seems to be worse in our Jenkins CI pipeline. We are currently building for PS4/PS5.

Once the import process get's stuck, its just hundreds of lines of this until I restart the job.

Code (CSharp):

1. 01:24:28 [OnDemand] Worker ##: WorkerState(Connecting), ImporterState(Initializing) worker timed out connecting with editor - while handling artifactKey Guid(00000000000000000000000000000000) Importer(-2,00000000000000000000000000000000) - path  - see Logs/AssetImportWorker###.log

I've tried disabling AV and firewall entirely - it seems to help but doesn't stop the issue entirely.

Wondering if anyone else is seeing this, and if they know any workarounds. I can't find anything relevant in the patches notes, or on Unity's bug tracker.

Here is an example of a worker that times out:

Code (CSharp):

1. nothing useful in the log files, just the worker process' launch diag.
2.  
3. Using pre-set license
4. Built from '2021.2/staging' branch; Version is '2021.2.2f1 (5e2b1e92c7f8) revision 6171422'; Using compiler version '192829333'; Build Type 'Release'
5. OS: 'Windows 10 Pro; OS build 19042.1348; Version 2009; 64bit' Language: 'ja' Physical Memory: 65414 MB
6. BatchMode: 1, IsHumanControllingUs: 0, StartBugReporterOnCrash: 0, Is64bit: 1, IsPro: 0
7.  
8. COMMAND LINE ARGUMENTS:
9. C:\Program Files\Unity\2021.2.2f1\Editor\Unity.exe
10. -adb2
11. -batchMode
12. -noUpm
13. -name
14. AssetImportWorker355
15. -projectPath
16. C:/jenkins/build-environment/ps5-01/master_build_ps5
17. -logFile
18. Logs/AssetImportWorker355.log
19. -srvPort
20. 64740
21. Successfully changed project path to: C:/jenkins/build-environment/ps5-01/master_build_ps5
22. C:/jenkins/build-environment/ps5-01/master_build_ps5
23. [UnityMemory] Configuration Parameters - Can be set up in boot.config
24.    "memorysetup-bucket-allocator-granularity=16"
25.    "memorysetup-bucket-allocator-bucket-count=8"
26.    "memorysetup-bucket-allocator-block-size=33554432"
27.    "memorysetup-bucket-allocator-block-count=8"
28.    "memorysetup-main-allocator-block-size=16777216"
29.    "memorysetup-thread-allocator-block-size=16777216"
30.    "memorysetup-gfx-main-allocator-block-size=16777216"
31.    "memorysetup-gfx-thread-allocator-block-size=16777216"
32.    "memorysetup-cache-allocator-block-size=4194304"
33.    "memorysetup-typetree-allocator-block-size=2097152"
34.    "memorysetup-profiler-bucket-allocator-granularity=16"
35.    "memorysetup-profiler-bucket-allocator-bucket-count=8"
36.    "memorysetup-profiler-bucket-allocator-block-size=33554432"
37.    "memorysetup-profiler-bucket-allocator-block-count=8"
38.    "memorysetup-profiler-allocator-block-size=16777216"
39.    "memorysetup-profiler-editor-allocator-block-size=1048576"
40.    "memorysetup-temp-allocator-size-main=16777216"
41.    "memorysetup-job-temp-allocator-block-size=2097152"
42.    "memorysetup-job-temp-allocator-block-size-background=1048576"
43.    "memorysetup-job-temp-allocator-reduction-small-platforms=262144"
44.    "memorysetup-temp-allocator-size-background-worker=32768"
45.    "memorysetup-temp-allocator-size-job-worker=262144"
46.    "memorysetup-temp-allocator-size-preload-manager=33554432"
47.    "memorysetup-temp-allocator-size-nav-mesh-worker=65536"
48.    "memorysetup-temp-allocator-size-audio-worker=65536"
49.    "memorysetup-temp-allocator-size-cloud-worker=32768"
50.    "memorysetup-temp-allocator-size-gi-baking-worker=262144"
51.    "memorysetup-temp-allocator-size-gfx=262144"
52. Refreshing native plugins compatible for Editor in 1188.84 ms, found 7 plugins.
53. Preloading 0 native plugins for Editor in 0.00 ms.
54. Initialize engine version: 2021.2.2f1 (5e2b1e92c7f8)
55. [Subsystems] Discovering subsystems at path C:/Program Files/Unity/2021.2.2f1/Editor/Data/Resources/UnitySubsystems
56. [Subsystems] Discovering subsystems at path C:/jenkins/build-environment/ps5-01/master_build_ps5/Assets
57. GfxDevice: creating device client; threaded=0; jobified=0
58. Direct3D:
59.    Version:  Direct3D 11.0 [level 11.1]
60.    Renderer: NVIDIA GeForce GTX 1660 Ti (ID=0x2182)
61.    Vendor:   NVIDIA
62.    VRAM:     5991 MB
63.    Driver:   30.0.14.7111
64. Initialize mono
65. Mono path[0] = 'C:/Program Files/Unity/2021.2.2f1/Editor/Data/Managed'
66. Mono path[1] = 'C:/Program Files/Unity/2021.2.2f1/Editor/Data/MonoBleedingEdge/lib/mono/unityjit-win32'
67. Mono config path = 'C:/Program Files/Unity/2021.2.2f1/Editor/Data/MonoBleedingEdge/etc'
68. Using monoOptions --debugger-agent=transport=dt_socket,embedding=1,server=y,suspend=n,address=127.0.0.1:56620
69. Begin MonoManager ReloadAssembly
70. Registering precompiled unity dll's ...
71. Register platform support module: C:/Program Files/Unity/2021.2.2f1/Editor/Data/PlaybackEngines/WindowsStandaloneSupport/UnityEditor.WindowsStandalone.Extensions.dll
72. Registered in 0.050056 seconds.
73. Native extension for WindowsStandalone target not found
74. Refreshing native plugins compatible for Editor in 646.83 ms, found 7 plugins.
75. Preloading 0 native plugins for Editor in 0.00 ms.
76. Mono: successfully reloaded assembly
77. - Completed reload, in  1.691 seconds
78. Domain Reload Profiling:
79.     ReloadAssembly (1692ms)
80.         BeginReloadAssembly (256ms)
81.             ExecutionOrderSort (0ms)
82.             DisableScriptedObjects (0ms)
83.             BackupInstance (0ms)
84.             ReleaseScriptingObjects (0ms)
85.             CreateAndSetChildDomain (1ms)
86.         EndReloadAssembly (1187ms)
87.             LoadAssemblies (227ms)
88.             RebuildTransferFunctionScriptingTraits (0ms)
89.             SetupTypeCache (114ms)
90.             ReleaseScriptCaches (0ms)
91.             RebuildScriptCaches (27ms)
92.             SetupLoadedEditorAssemblies (831ms)
93.                 LogAssemblyErrors (0ms)
94.                 InitializePlatformSupportModulesInManaged (35ms)
95.                 SetLoadedEditorAssemblies (0ms)
96.                 RefreshPlugins (647ms)
97.                 BeforeProcessingInitializeOnLoad (1ms)
98.                 ProcessInitializeOnLoadAttributes (107ms)
99.                 ProcessInitializeOnLoadMethodAttributes (41ms)
100.                 AfterProcessingInitializeOnLoad (0ms)
101.                 EditorAssembliesLoaded (0ms)
102.             ExecutionOrderSort2 (0ms)
103.             AwakeInstancesAfterBackupRestoration (0ms)
104. Platform modules already initialized, skipping
105. Registering precompiled user dll's ...
106. Registered in 0.378416 seconds.

Cheers.

 

I tried increasing the idle worker timeout to 120s, which seems to allow 1/20 workers to survive. Not sure if it's related at all, may be a fluke...

Will continue to investigate.

 

bump

 

bump

 

bump

 

bump

 

I am running into this as well on a Ubuntu VM. But not with every project. Have you have any luck?

 

Unfortunately not. Just running Jenkins on a server with anti-virus completely disabled.

 

I'm having this issue in 2021.2.17.f1

Sad to see there has been no response to the previous post.

 

Hi jdrewson,

That's what the example log is in the original post - and every single one is the same. Unless I'm missing something I don't think there is anything of note there.

Kind regards,
James

 

Ummm, it's exactly as described above. It just says it's timed out.

 

When it works it loads my project in under 2 minutes.

And the workers are timing out, but as the OP mentioned, it just starts a new one.

 

My unity version is 2021.3.0 macOS. Same problem.
I delete my layout cache(UserSettings/Layouts).seems better.
But stuck at some very big model importing. Delete the model. I finally get into my project.....

 

Good to hear, as I'm still having this issue and it's driving me crazy.

Digging through my AssetWorkers and Editor log, this always comes up.

Begin MonoManager ReloadAssembly

Symbol file LoadedFromMemory doesn't match image G:\Unity Projects\Swordsfall ARPG Main 8.0\Library\PackageCache\com.unity.visualscripting@1.7.8\Editor\VisualScripting.Core\Dependencies\DotNetZip\Unity.VisualScripting.IonicZip.dll

Symbol file LoadedFromMemory doesn't match image G:\Unity Projects\Swordsfall ARPG Main 8.0\Library\PackageCache\com.unity.visualscripting@1.7.8\Editor\VisualScripting.Core\Dependencies\YamlDotNet\Unity.VisualScripting.YamlDotNet.dll

 

Same with 2021.3.2f1, all AssetImporter log ends with "Begin MonoManager ReloadAssembly".

 

We have this problem when running the Unity editor in headless mode on an M1 Mac Studio. The log is full of:

01:40:37  Starting new worker id: 0 with log in

01:40:37  Starting new worker id: 1 with log in

01:41:12  [OnDemand] Worker 0: WorkerState(Connecting), ImporterState(Initializing) worker timed out connecting with editor - while handling artifactKey Guid(00000000000000000000000000000000) Importer(-2,00000000000000000000000000000000) - path  - see Logs/AssetImportWorker0.log

01:41:12  Starting new worker id: 2 with log in

01:41:12  [OnDemand] Worker 1: WorkerState(Connecting), ImporterState(Initializing) worker timed out connecting with editor - while handling artifactKey Guid(00000000000000000000000000000000) Importer(-2,00000000000000000000000000000000) - path  - see Logs/AssetImportWorker1.log

01:41:12  Starting new worker id: 3 with log in

01:41:41  [OnDemand] Worker 2: WorkerState(Connecting), ImporterState(Initializing) worker timed out connecting with editor - while handling artifactKey Guid(00000000000000000000000000000000) Importer(-2,00000000000000000000000000000000) - path  - see Logs/AssetImportWorker2.log

01:41:41  Starting new worker id: 4 with log in

01:41:41  [OnDemand] Worker 3: WorkerState(Connecting), ImporterState(Initializing) worker timed out connecting with editor - while handling artifactKey Guid(00000000000000000000000000000000) Importer(-2,00000000000000000000000000000000) - path  - see Logs/AssetImportWorker3.log

entries, and when inspecting the /Log output for the workers they all look like this:

Unity Editor version:    2022.1.9f1 (07e076b6d414)

Branch:                  2022.1/staging

Build type:              Release

Batch mode:              YES

macOS version:           Version 12.4 (Build 21F79)

Darwin version:          21.5.0

Architecture:            arm64

Running under Rosetta:   NO

Available memory:        32768 MB

Using pre-set license

Pro License: YES

2022-07-24 22:24:48.934 Unity[22428:558830] +[NSXPCSharedListener endpointForReply:withListenerName:]: an error occurred while attempting to obtain endpoint for listener 'ClientCallsAuxiliary': Connection interrupted

We are using the Unity Accelerator and the Unity instance is invoked as such:

                "${UNITY_PATH}" \

                -projectPath . \

                -buildTarget ${BUILD_TARGET} \

                -logFile - log.txt \

                -EnableCacheServer \

                -cacheServerEndpoint "${UNITY_ACCELERATOR_ADDRESS}" \

                -cacheServerEnableDownload \

                -cacheServerEnableUpload \

                -username xxx \

                -password xxx \

                -batchmode \

                -nographics \

                -quit \

                -silent-crashes

This happens always after the Library has been cleaned, and requires a few runs before it starts working again.
Please help us as this is pretty much a fatal error for our CI.

Thank you!

 

Update: The error goes away if I log in on the server, at which point I see the Unity splash screen flash. The -nographics option is clearly not working.

 

@TobiasThomsen Could you also look into the workers never starting if you run a command line instance (with -nographics) unless the user is logged in (on MacOs)? This is a regression and quite bad as it forces one to log in on the build machines. I posted more details above.

With Unity 2012.1.12 the editor gets stuck in this place forever if a user is not logged in:
00:17:42 Platform modules already initialized, skipping
00:17:42 Registering precompiled user dll's ...
00:17:42 Registered in 0.000869 seconds.
00:17:42 Starting new worker id: 0 with log in
00:17:42 Starting new worker id: 1 with log in
<-- never proceeds from here

Could you please fix the regression so that Unity works properly in command line -nographics mode on MacOs without a logged in user?

Thank you!

 

@TobiasThomsen For clarity, with Unity version 2012.1.12 the log gets stuck at Starting new worker id: 1, while with 2022.1.9 there were error messages about workers timing out (see my post from 24th July). But in neither case does it work. It just gets stuck there indefinitely until you kill it.

 

Hi. The last time I used Unity was around 2.5-3 years ago and at that point you could make command line builds no problem. I was actually quite surprised to see it not working in Unity 2022.1 as I am sure quite a lot of people run the editor in headless mode when they make builds. Could this be fixed quickly as surely it's a very common use case to have some kind of build machine without logged in users?

Thank you.

 

@TobiasThomsen Hi. Could I get some traction behind fixing the issue (with information located above in my previous posts) about Unity not working in headless mode on Macs with no logged in users? It's tangentially related to this thread as the workers time out until a user logs in. This kinda defeats the purpose of -batchmode -nographics and is almost a fatal issue.

Please help! This is a regression as this did work a few years ago when I was last using Unity, and as I said I'd assume having a build machine with Unity running in headless mode to be a very common use case.

 

Hi Folks, I have been plagued by this issue for months losing days of productivity at a time.

I'm using Unity 2022.1.7f1 and found a WORKAROUND.

I've seen similar posts on this topic and found the same clues as others:

01:01:20 [OnDemand] Worker ##: WorkerState(Connecting), ImporterState(Initializing) worker timed out connecting with editor - while handling artifactKey Guid(00000000000000000000000000000000) Importer(-2,00000000000000000000000000000000) - path - see Logs/AssetImportWorker###.log

The Editor.Log is full of these and will continue for days if left unattended.

The parallel import kicks off multiple AssetImportWorkers but they fail for some reason and then time out.
Seems to be an infinite loop and there really aren't any actionable clues in the worker log files.

The following lines seems a common theme:

Begin MonoManager ReloadAssembly
Symbol file LoadedFromMemory doesn't match image D:\My Project\Library\PackageCache\com.unity.visualscripting@1.7.8\Editor\VisualScripting.Core\Dependencies\DotNetZip\Unity.VisualScripting.IonicZip.dll
Symbol file LoadedFromMemory doesn't match image D:\My Project\Library\PackageCache\com.unity.visualscripting@1.7.8\Editor\VisualScripting.Core\Dependencies\YamlDotNet\Unity.VisualScripting.YamlDotNet.dll

These always seem to be there, even if you remove these PackageCaches and let it rebuild them. However, I think this could be a red herring.

WORKAROUND

This evening I managed to find a good workaround/fix to turn off the parallel asset import that is spawning the worker threads. However, how do you change project settings if you can't open the project?

Firstly I edit a random .CS file in project using notepad and then opened the project force Unity to fail and prompt to go into Safe Mode. A simple line of XXXX at the top of a random .CS file worked for me.

Once in Safe mode, I was able to go into the Edit > Project Settings and turn off the following two options:

Editor > Asset Pipeline > Remove Unused Artifacts on Restart
Editor > Asset Pipeline > Parallel Import

I then removed the XXXX in the random .CS file and exited safe mode.

This picked up the .cs and recompiled it. The project loaded without a problem a few mere seconds later.

I'm now able to open my 1TB project in under 90s.
Like I said, it's a workaround because I'd really like to use Parallel Import, but for the sake of the hundreds of lost hours I've wasted, I'm happy to wait until Parallel Import doesn't have this "feature".

By the way, deleting my whole library cache and having it rebuilt every time this issue occurs is not a good workaround when you have a massive project.

Good luck folks.

 

Amazing! I was looking for a solution to this right at the moment you posted this workaround. (I tried the workaround in the meantime)
The only other thing I found elsewhere was that Unity is not going to fix this issue: https://issuetracker.unity3d.com/is...nment-while-parallel-import-option-is-enabled
I guess they want people to use their own Build solution.

Thanks for the workaround. Worked for me!

 

Hi all, in case anyone still encountering this issue, I finally found a solution that worked for me.

In my case, it occurred while running headless Unity Editor (not sure if everyone here had the same case).

The solution:

Call:

xvfb-run -a --server-args="-screen 0 1024x768x24" $UNITY_PATH/Editor/Unity -batchmode -quit -nographics ...

Instead of:

$UNITY_PATH/Editor/Unity -batchmode -quit -nographics ...

Explanation:
In my case, I wanted to export by calling a C# method, from CLI. but the export process somehow triggered the

AssetImportWorker

. When looking on the logs of this worker I saw it tries to access the screen, and we already use

xvfb

for running tests. We tried to run the Editor under

xvfb

and it finally worked. I only can guess that it uses the screen's port in order to communicate with the Editor.

Good luck!
Hope it helps

 

I hate it that Unity doesn't fix the issue that at least M1 Macs can't do -batchmode -nographics unless a user is logged in (and a graphics context is available). It's a regression, this used to work. Now it just hangs infinitely in the worker threads. Come on please!

 

I have reported hundreds of bugs but they all get closed now for various reasons. It seems that no bugs get through anymore. So you are my only hope.

 

This is also happening when exporting assets as a package on Unity 2021.3.18f1. I noticed it in the Editor.log, although it did not freeze for me in particular. However, I have a co-worker who does experience the asset import worker timeout issue when trying to export assets this way. The co-worker in question does have a slower connection, so I think that may be contributing to why they are getting the error, for anyone else who is having this problem on exporting assets in Unity 2021. They are basically getting a TcpProtobufClient error with the error code 10060. This is the source I could find: 10060 Connection timed out error with proxy server or ISA Server on slow link - Windows Client | Microsoft Learn
Again, not a fix for the OP above, but in case anyone else is having a problem with the particular use case I mentioned, which is just similar.

 

For the problems involving the server saying Symbol at file doesnt match I created a new build, build config, and fleet and the errors disappeared

 

@TobiasThomsen
This issue still occurs, Unity version is 2022.1.24. When I start it with GUI, it starts correctly, but when I start it with CUI, it stops at this part.

Once the project is launched in GUI, it runs correctly when launched in CUI. If the project is started from a clean state and executed with CUI, it stops in the above state.

 

It looks like they fixed it in 2021.3.23. Have you tried it ? We were getting the same error so we swtopped using CI/CD all together. I am unable to test it right now.

• Editor: Fixed issue where -nographics command line argument was not being forwarded to the Asset Import Workers. (UUM-29722)

 

This seems to be happening on 2022.2.17f1

Going to try the various workarounds posted above...

 

I had the same issue, but what was curious to me was that running the command line build in my machine would work, but it would always break on my CI build machine. After some investigation, I realized the reason for that was that my CI machine makes a clean checkout which does not have a library folder, and if I deleted the library folder from my local machine the command line build would hang in the same point:

Starting new worker id: 0 with log in

Starting new worker id: 1 with log in

Worker ready: AssetImportWorker0 0

Worker ready: AssetImportWorker1 1

The way to fix it, as mentioned by @LegacySystem is to have the -nographics option removed from the command. For me it is enough to remove the -nographics option. But if you need the -nographics option because you don't have a GPU on your build machine, a workaround is to "keep/copy" the library folder in the build machine before building.

 

Unless I messed up something on my box, this issues appear to be "back" on 2021.3.24f, where it was not on .22f1

Anyone else on .24f1 seeing this issue?

 

@TobiasThomsen, any update on this? Our builds worked just fine before updating to 2022.2.17f1, now we need to remove the -nographics option to avoid the freeze, which makes the builds longer.

 

• Editor: Fixed issue where -nographics command line argument was not being forwarded to the Asset Import Workers. (UUM-29722)

This "fix" changed something, because before it I was able to use -nographics. Now after the "fix" it just halts.
I have reported the issue, but am not holding my breath.

 

fwiw, in 2021.3.24f1, if I turn off "Parallel Import", I can run headless again

 

Have you reported it? They will close it with a "There was not enough information to proceed" though so it's kinda a waste of time tbh.

 

I've just struck this too, after moving from 2021.3.2f1 to 2021.3.25f1 on a machine running CI for an open-source project. Is there a way to configure disabling of the parallel import prior to launching the Editor?

 

I've not spent time looking into how I can disable parallel import for CI builds, while leaving it on for local dev, w/o hacks to ProjectSettings/EditorSettings.asset (e.g. I'm not aware of a command line switch to turn if off and override EditorSettings).

I'll go ahead and report this.

 

Yes, this was introduced for us on .24f1, hadn't seen it on any version we've had on our build machines before then.

Edit: I do not know whether this is still the case without testing, but in the fairly recent past (i.e. within the lifecycle of our current product) running without -nographics was not an option at all, it resulted in Unity decompressing textures into memory during asset bundle builds, causing the build machines to run out of RAM.

Edit 2: We do have a prebuild step on our CI, we can disable -nographics for that initial import step. This is silly though, is this stuff being tested at all, or what?

 

We just ran into this problem when upgrading to the new 2022.3.0f1 LTS. The build would always hang after: "Worker ready: AssetImportWorker1 1". Fiddeling with parallel import settings did nothing. In the end we dropped the -nographics option from our ci script.

 

Same here with 2022.3.1f1 LTS. -nographics is not really an option as it disables logging, which is critical for CI builds.

How did it get to LTS with a bug like this?

 

What we've done was to add a pre-build step without -nographics, just opens the project then quits.

The fact it actually regressed in 2021LTS says a lot really.

 

Hello. Have the same problem in 2023.3.27f1. Disabling "Parallel Import" option helped me.