MonoManager ReloadAssembly freezes for hours, causes cancelation

Once the build started, it got to a certain point (line 53672), and then didn't do anything for 4 hours, eventually forcing the build server to cancel it. This is a seemingly random problem, since I've never had issues with this before.

Code (CSharp):

1. 53661: [Unity] Begin MonoManager ReloadAssembly
2. 53662: [Unity] Symbol file LoadedFromMemory doesn't match image /BUILD_PATH/joshcamas.ardenfall.dev-windows-64x/Ardenfall/Library/PackageCache/com.unity.ext.nunit@1.0.0/net35/unity-custom/nunit.framework.dll
3. 53663: [Unity] Native extension for AppleTV target not found
4. 53664: [Unity] Native extension for Lumin target not found
5. 53665: [Unity] Native extension for iOS target not found
6. 53666: [Unity] Native extension for Android target not found
7. 53667: [Unity] Native extension for LinuxStandalone target not found
8. 53668: [Unity] Native extension for OSXStandalone target not found
9. 53669: [Unity] Native extension for WebGL target not found
10. 53670: [Unity] Native extension for WindowsStandalone target not found
11. 53671: [Unity] Refreshing native plugins compatible for Editor in 2.80 ms, found 2 plugins.
12. 53672: [Unity] Preloading 0 native plugins for Editor in 0.00 ms.
13. 53673: Build was canceled
14. 53674: Build step 'Execute a set of scripts' marked build as failure
15. 53675: Finished: CANCELED
16.  
17. ********************************************************************************
18. This build was automatically canceled by the Cloud Build service. This usually
19. happens when log output stops for an extended period of time. If you suspect
20. this is due to a problem with the Cloud Build service, please contact support.
21. https://developer.cloud.unity3d.com/support/
22. ********************************************************************************

 

We might be experiencing the same issue here. About 80% of our cloud builds fail with this error.

Code (CSharp):

1. 2641: [Unity] Reloading assemblies after forced synchronous recompile.
2. 2642: [Unity] Begin MonoManager ReloadAssembly
3. 2643: Build was canceled
4. 2644: Killing process...
5. 2645: Using /BUILD_PATH/.rvm/gems/ruby-2.4.2
6. 2646: postbuildstatus finished successfully.
7. 2647: postbuildsteps finished successfully
8. 2648: Finished: CANCELED

One thing that seems to be interesting, is that builds always cancel at the same time, almost down to the second, 03:39:21


The builds that do work have a much shorter log and build time.
A difference between working and non-working builds seems to be that the working build only compiles once. The non-working build seems to be compiling multiple times.

Further up in the log of a broken build we also see this line, which seems to be pointing to the same nunit.framework.dll as joshcamas sees:

Code (CSharp):

1. 2257: [Unity] Symbol file LoadedFromMemory doesn't match image /BUILD_PATH/anduo-games.third-crisis.patron-windows-x64/Library/PackageCache/com.unity.ext.nunit@1.0.0/net35/unity-custom/nunit.framework.dll

However, the compile where this message is shown seems to finish. Another compile further down in the log is what seemingly hangs up the build and ultimately cancels it.

This started happening when we upgraded from Unity 2019.2.0f1 to 2020.1.8f1

Only windows builds seem to ever actually finish successfully. We also do Android and Webgl builds. These have not succeeded once since we upgraded unity.

Edit:
I just tested a local builds, it works just fine, so the issue seems to be confined to the build server.

 

For replicating this locally you would have to delete your Library folder - this issue looks like it's being encountered during the asset import step. Most likely how it would present locally is just a progress bar that doesn't progress.

You could potentially report this as an issue with the editor (Help -> Report an Issue) but without a local reproduction this is unlikely to get prioritized. If you open a support ticket we could look into this a little further and try to replicate it ourselves, though in either case this is likely to be an issue with the Editor so you would need to wait on a fix to be released and upgrade to that version.

If you are able to instead switch to 2019.4 then you may be able to avoid this issue - tech stream releases are less stable than LTS releases, especially when they are early in the release cycle (as with 2020.1)

 

Thanks so much for the fast reply. We downgraded to 2019.4 and it did indeed fix this issue. Cloud build is running fine now.

 

Just a note that I get a similar issue here in Editor when coming back from Rider with a code change - not sure of the actual cause.