Bug Batch Mode Hangs Due to an Issue with bee_backend

[I'm reporting this here because I most recently encountered it here. The first version of Unity wherein I've verified that this happens is in Unity 2022.1.9 (the only version of 2022.1 that I've tested). In testing the Splash Screen issue I posted, I tested our workflow with 2022.2.0b2 and was able to verify that this issue still occurs in the most recent version of Pre-release Unity (that I'm aware of).]

Context
I am the developer of Koreographer and our build system for the asset involves running Unity in Batch Mode and having it run through several build steps with a single static function call. It goes through multiple passes of:

1. Bring in scripts from a development repository.
2. Refresh the AssetDatabase with AssetDatabase.Refresh.
3. Run AssetDatabase.ExportPackage to create a sub-package to include with the asset.
4. Delete the files copied into the project in Step 1.

That #2 there probably made you cringe. It does me, given that I understand that script compilation is one thing that an AssetDatabase.Refresh call might trigger. The thing that "saves" us is the fact that our script doesn't return until after its run through all of its process - there is no opportunity for the domain reload.

Or at least that's how it did work from Unity 4.5.0 (our first supported version) through Unity 2021.3 (I think? Most of our testing has been with 2022. I understand that bee_backend is new as of Unity 2021 so it's possible that we've simply lucked out of [what feels like a race condition] thus far).

Batch Mode Hangs
With Unity 2022 we hit the following scenario pretty frequently:

1. Build is proceeding fine.
2. We get to the point where we start building exported packages.
3. Somewhere after two or three packages, the log starts to show bee_backend running a script compilation process for the Assembly-CSharp assembly.
4. The output shows that the bee_backend process fails due to certain script files not being found (this is because we deleted them).
5. The batch mode standard output simply stops - our build script never completes and we have to kill the process.

I came across something that looks similar from Unity 2021 on Linux here. I found that post by looking up what Bee was - it appears to be your build backend.

So what I suspect is happening is that our AssetDatabase refresh is importing scripts and eventually sending a message to the build backend that a new compilation pass is going to need to be triggered. That doesn't stop our script from processing* however. So our process happily goes forward and deletes the files that triggered the Bee build. Except that Bee fails to build and causes an error. That somehow stops our process and leaves us with a hung Batch Mode.

* To be clear, we don't want our script to stop processing - some of the scripts we are pulling in won't compile correctly because their dependencies are not brought into the project. They are simply brought in for the purpose of being bundled into a package - there is less-than-zero need for them to be compiled.​

I would think that Bee builds should be scheduled when AssetDatabase.Refresh is called but that it shouldn't actually be triggered. At the very least, our build script should not be interrupted and the result of the compilation should be wiped away if other manipulations occurred behind the scenes that resulted in a valid compilation by the time the script process has returned.

Is there something that can be done here? Is there a way for us to sidestep this already with the current APIs available?

 

Ahh, I should also include some of the lines of the Editor log that we see:

ExitCode: 4 Duration: 0s117ms

Rebuilding DAG because FileSignature timestamp changed: Library/Bee/200b0aE-inputdata.json

*** Tundra requires additional run (0.01 seconds), 0 items updated, 94 evaluated

ExitCode: 0 Duration: 0s367ms

Starting: /Applications/Unity/Hub/Editor/2022.2.0b2/Unity.app/Contents/bee_backend --dont-print-to-structured-log --ipc --defer-dag-verification --dagfile="Library/Bee/200b0aE.dag" --continue-on-failure --dagfilejson="Library/Bee/200b0aE.dag.json" --profile="Library/Bee/backend2.traceevents" ScriptAssemblies

WorkingDir: /path/to/project/SourcePackage

ExitCode: 3 Duration: 0s785ms

Finished compiling graph: 118 nodes, 595 flattened edges (582 ToBuild, 4 ToUse), maximum node priority 26

<Compile command and .rsp output snipped>

##### Custom Environment Variables

DOTNET_MULTILEVEL_LOOKUP=0

##### ExitCode

1

##### Output

error CS2001: Source file '/path/to/project/SourcePackage/Assets/path/to/Script1.cs' could not be found.

error CS2001: Source file '/path/to/project/SourcePackage/Assets/path/to/Script2.cs' could not be found.

*** Tundra build failed (0.77 seconds), 20 items updated, 115 evaluated

That's where it ends. The process is just sort of... stopped there at this point (which is bad for a Unity process started via the command line with -quit -batchmode). But... also the script building in the background shouldn't kill the active script processing (or possibly not halt the thread per-se, but somehow lock it out).

 

Just got another hang with 2022.2.0b2. This is where it stopped:

ExitCode: 2 Duration: 0s109ms

tundra: error: Failed IPC_Client_InitializeAndConnectToParent: connect failed to /var/folders/_2/zgryrd_x4vg0_6ds2v6v2c_c0000gn/T//5oc2rsuy.7nk/ipc_72280_cth

This line showed up a bit earlier in the output as well. Once again, our process did not complete.

 

I have not created a bug report. I was attempting to build a repro case but haven't yet successfully done so. The setup that does reproduce this only does so occasionally and it is incompatible with the standard Bug Report project packaging system (due to the nature of copying from various external projects into the current one and exporting packages).

Time permitting I will hack at the test project I've been building to see if I can get it to reproduce at some point.

 

Has this ever been resolved yet? We are on latest stable Unity 2021.3.16f1 and are setting up the build pipeline. It happens to us now where it just prints

Code (CSharp):

1. Rebuilding DAG because FileSignature timestamp changed

Rebuilding DAG because FileSignature timestamp changed

forever until the build process crashes

 

Seems likely that it hasn't, given your experience with the latest full release of the editor.

Can you file a bug? I was never able to get a project together where it would reliably reproduce...

 

It does seem to be an ongoing issue with our production repo - this increases our build times from a few minutes to an hour IF it does not crash...
Will try to get an Unity engineer source access again..

 

Hi. Are you guys on top of this? There's logs here that you can use to investigate the issue. This has to get fixed asap.

 

Can confirm that we have this problem too with 2021.3.16f1. Jenkins builds randomly freeze and print the quoted error.

 

We had to fix it by overwriting the bee_backend script from an older Unity version. There is a different forum post that explains this process but I don't have it at hand right now unfortunately.

 

Huh. Would love to read it over if you do find time to locate that post. I did a quick search myself and the closest thing I could find was another thread where a Linux-specific fix involved removing a specific flag (--stdin-canary) from the startup command (which doesn't appear to be used on macOS anyway).

 

May you bring it here later? We're not finding it at all.

Btw, the problem of ours is the same that have been said here. Hung at Bee steps and the whatnot from batch mode of CI/CD agents.

 

Was this ever fixed? And if so, what version?

 

Our project build assetbundles in batchmode by jenkins.
It was randomly freezed when called AssetDatabase.Refresh()

Unity 2021.3.21f1