Search Unity

  1. Welcome to the Unity Forums! Please take the time to read our Code of Conduct to familiarize yourself with the forum rules and how to post constructively.

Bug Batch Mode Hangs Due to an Issue with bee_backend

Discussion in '2022.2 Beta' started by SonicBloomEric, Jul 22, 2022.

  1. SonicBloomEric

    SonicBloomEric

    Joined:
    Sep 11, 2014
    Posts:
    1,072
    [I'm reporting this here because I most recently encountered it here. The first version of Unity wherein I've verified that this happens is in Unity 2022.1.9 (the only version of 2022.1 that I've tested). In testing the Splash Screen issue I posted, I tested our workflow with 2022.2.0b2 and was able to verify that this issue still occurs in the most recent version of Pre-release Unity (that I'm aware of).]

    Context
    I am the developer of Koreographer and our build system for the asset involves running Unity in Batch Mode and having it run through several build steps with a single static function call. It goes through multiple passes of:
    1. Bring in scripts from a development repository.
    2. Refresh the AssetDatabase with AssetDatabase.Refresh.
    3. Run AssetDatabase.ExportPackage to create a sub-package to include with the asset.
    4. Delete the files copied into the project in Step 1.
    That #2 there probably made you cringe. It does me, given that I understand that script compilation is one thing that an AssetDatabase.Refresh call might trigger. The thing that "saves" us is the fact that our script doesn't return until after its run through all of its process - there is no opportunity for the domain reload.

    Or at least that's how it did work from Unity 4.5.0 (our first supported version) through Unity 2021.3 (I think? Most of our testing has been with 2022. I understand that bee_backend is new as of Unity 2021 so it's possible that we've simply lucked out of [what feels like a race condition] thus far).

    Batch Mode Hangs
    With Unity 2022 we hit the following scenario pretty frequently:
    1. Build is proceeding fine.
    2. We get to the point where we start building exported packages.
    3. Somewhere after two or three packages, the log starts to show bee_backend running a script compilation process for the Assembly-CSharp assembly.
    4. The output shows that the bee_backend process fails due to certain script files not being found (this is because we deleted them).
    5. The batch mode standard output simply stops - our build script never completes and we have to kill the process.
    I came across something that looks similar from Unity 2021 on Linux here. I found that post by looking up what Bee was - it appears to be your build backend.

    So what I suspect is happening is that our AssetDatabase refresh is importing scripts and eventually sending a message to the build backend that a new compilation pass is going to need to be triggered. That doesn't stop our script from processing* however. So our process happily goes forward and deletes the files that triggered the Bee build. Except that Bee fails to build and causes an error. That somehow stops our process and leaves us with a hung Batch Mode.

    * To be clear, we don't want our script to stop processing - some of the scripts we are pulling in won't compile correctly because their dependencies are not brought into the project. They are simply brought in for the purpose of being bundled into a package - there is less-than-zero need for them to be compiled.

    I would think that Bee builds should be scheduled when AssetDatabase.Refresh is called but that it shouldn't actually be triggered. At the very least, our build script should not be interrupted and the result of the compilation should be wiped away if other manipulations occurred behind the scenes that resulted in a valid compilation by the time the script process has returned.

    Is there something that can be done here? Is there a way for us to sidestep this already with the current APIs available?
     
  2. SonicBloomEric

    SonicBloomEric

    Joined:
    Sep 11, 2014
    Posts:
    1,072
    Ahh, I should also include some of the lines of the Editor log that we see:


    ExitCode: 4 Duration: 0s117ms
    Rebuilding DAG because FileSignature timestamp changed: Library/Bee/200b0aE-inputdata.json
    *** Tundra requires additional run (0.01 seconds), 0 items updated, 94 evaluated
    ExitCode: 0 Duration: 0s367ms
    Starting: /Applications/Unity/Hub/Editor/2022.2.0b2/Unity.app/Contents/bee_backend --dont-print-to-structured-log --ipc --defer-dag-verification --dagfile="Library/Bee/200b0aE.dag" --continue-on-failure --dagfilejson="Library/Bee/200b0aE.dag.json" --profile="Library/Bee/backend2.traceevents" ScriptAssemblies
    WorkingDir: /path/to/project/SourcePackage
    ExitCode: 3 Duration: 0s785ms
    Finished compiling graph: 118 nodes, 595 flattened edges (582 ToBuild, 4 ToUse), maximum node priority 26

    <Compile command and .rsp output snipped>

    ##### Custom Environment Variables
    DOTNET_MULTILEVEL_LOOKUP=0
    ##### ExitCode
    1
    ##### Output
    error CS2001: Source file '/path/to/project/SourcePackage/Assets/path/to/Script1.cs' could not be found.
    error CS2001: Source file '/path/to/project/SourcePackage/Assets/path/to/Script2.cs' could not be found.
    *** Tundra build failed (0.77 seconds), 20 items updated, 115 evaluated


    That's where it ends. The process is just sort of... stopped there at this point (which is bad for a Unity process started via the command line with -quit -batchmode). But... also the script building in the background shouldn't kill the active script processing (or possibly not halt the thread per-se, but somehow lock it out).
     
    Last edited: Jul 22, 2022
  3. SonicBloomEric

    SonicBloomEric

    Joined:
    Sep 11, 2014
    Posts:
    1,072
    Just got another hang with 2022.2.0b2. This is where it stopped:


    ExitCode: 2 Duration: 0s109ms
    tundra: error: Failed IPC_Client_InitializeAndConnectToParent: connect failed to /var/folders/_2/zgryrd_x4vg0_6ds2v6v2c_c0000gn/T//5oc2rsuy.7nk/ipc_72280_cth


    This line showed up a bit earlier in the output as well. Once again, our process did not complete.
     
    Last edited: Jul 22, 2022
  4. LeonhardP

    LeonhardP

    Unity Technologies

    Joined:
    Jul 4, 2016
    Posts:
    3,064
    Hi @SonicBloomEric,

    Thanks for writing this up, we shared it with the team. Did you submit this via a bug report as well? If not, could you please do so?
     
  5. SonicBloomEric

    SonicBloomEric

    Joined:
    Sep 11, 2014
    Posts:
    1,072
    I have not created a bug report. I was attempting to build a repro case but haven't yet successfully done so. The setup that does reproduce this only does so occasionally and it is incompatible with the standard Bug Report project packaging system (due to the nature of copying from various external projects into the current one and exporting packages).

    Time permitting I will hack at the test project I've been building to see if I can get it to reproduce at some point.
     
  6. CineTek

    CineTek

    Joined:
    Jul 12, 2013
    Posts:
    98
    Has this ever been resolved yet? We are on latest stable Unity 2021.3.16f1 and are setting up the build pipeline. It happens to us now where it just prints
    Code (CSharp):
    1. Rebuilding DAG because FileSignature timestamp changed
    Rebuilding DAG because FileSignature timestamp changed

    forever until the build process crashes
     
  7. SonicBloomEric

    SonicBloomEric

    Joined:
    Sep 11, 2014
    Posts:
    1,072
    Seems likely that it hasn't, given your experience with the latest full release of the editor.

    Can you file a bug? I was never able to get a project together where it would reliably reproduce...
     
  8. CineTek

    CineTek

    Joined:
    Jul 12, 2013
    Posts:
    98
    It does seem to be an ongoing issue with our production repo - this increases our build times from a few minutes to an hour IF it does not crash...
    Will try to get an Unity engineer source access again..
     
  9. ThisIsNotMyName123

    ThisIsNotMyName123

    Joined:
    May 10, 2022
    Posts:
    41
    Hi. Are you guys on top of this? There's logs here that you can use to investigate the issue. This has to get fixed asap.
     
  10. antti_partagames

    antti_partagames

    Joined:
    Nov 5, 2015
    Posts:
    17
    Can confirm that we have this problem too with 2021.3.16f1. Jenkins builds randomly freeze and print the quoted error.
     
  11. CineTek

    CineTek

    Joined:
    Jul 12, 2013
    Posts:
    98
    We had to fix it by overwriting the bee_backend script from an older Unity version. There is a different forum post that explains this process but I don't have it at hand right now unfortunately.
     
  12. SonicBloomEric

    SonicBloomEric

    Joined:
    Sep 11, 2014
    Posts:
    1,072
    Huh. Would love to read it over if you do find time to locate that post. I did a quick search myself and the closest thing I could find was another thread where a Linux-specific fix involved removing a specific flag (--stdin-canary) from the startup command (which doesn't appear to be used on macOS anyway).
     
  13. RGV

    RGV

    Joined:
    Jan 25, 2016
    Posts:
    40
    May you bring it here later? We're not finding it at all.

    Btw, the problem of ours is the same that have been said here. Hung at Bee steps and the whatnot from batch mode of CI/CD agents.
     
    Last edited: Mar 1, 2023