Linux editor stuck on loading because of bee_backend w/ workaround

So, after creating a new hdrp project with unity 2021.1.19f1 on my recently installed linux box I found that the editor got stuck at some random places in the launch process.

After a bit of investigating I found out that the culprit was bee_backend and more particularly the --stdin-canary that was preventing the process to close on completion.

The last line in the editor log is the following:

Starting: /......../unity/editors/2021.1.19f1/Editor/Data/bee_backend --dagfile="Library/Bee/2400b0aE.dag" --profile="Library/Bee/profiler.json" --continue-on-failure --stdin-canary ScriptAssemblies
WorkingDir: /......../unity/my-lovely-project​

Forcefully killing the process was causing build errors, despite the completion of the tasks, so instead I found a neat, temporary solution until the problem get resolved: add a shell script in the middle to remove the --stdin-canary and forward every other args to the real bee_backend executable.

In case someone is having the exact same issue, here are the steps I did to make it work:
- go to your unity editor folder, then in the Data folder
- rename bee_backend to bee_backend_real
- create a bee_backend file, make it executable
- write the following in the newly created bee_backend file

Code (shell):

1. #! /bin/bash
2.  
3. args=("$@")
4. for ((i=0; i<"${#args[@]}"; ++i))
5. do
6.     case ${args[i]} in
7.         --stdin-canary)
8.             unset args[i];
9.             break;;
10.     esac
11. done
12. ${0}_real "${args[@]}"

(note, this is a shell script and will be executed on your machine. Please don't put random code you don't understand on your machine)

 

I use game-ci + github actions with self-hosted runners for build my project and have the look like issue with Unity 2021.2.x.

Github Actions fails runs with the message:
> The hosted runner: Hosted Agent lost communication with the server. Anything in your workflow that terminates the runner process, starves it for CPU/Memory, or blocks its network access can cause this error.

The last lines writed by Uinity is

....

2021-12-16T21:20:36.3536601Z Starting: /opt/unity/Editor/Data/bee_backend 

--profile="Library/Bee/backend_profiler3.traceevents" 

--stdin-canary --dagfile="Library/Bee/Player43e62af5.dag"

 --continue-on-failure --dagfilejson="Library/Bee/Player43e62af5.dag.json" Player

2021-12-16T21:20:36.3539849Z WorkingDir: /github/workspace/TheGame



..next build step

2021-12-16T21:26:49.1462513Z Post job cleanup.

More than 6 minutes without any log records could look like stuck the build process

I've used a virtual machine with Ubunty linux for debugging the issue and find that the bee command local runtime is more than 13 minutes.

Looks like a github runner watchdog is too strong

Anyway 13 minutes without activity in logs looks strange. I guess an heartbeat line in a build log every minute can solve this )

 

It happens when I launch the editor on an empty/new/template project. The editor is stuck in the splash-screen. 100% of the time.
I am using opensuse tumbleweed. (and I have the same issue on two different installs of opensuse tumbleweed).
It has happened to every version of the editor since the post. And each time I upgrade the editor version, I simply apply the same process. I did not have an issue with the script. (but admittedly, my project is super small).

From what I remember of the issue, the editor closes the pipe and bee_backed doesn't close? (but it's been a long time since I investigated the thing)

 

I've got a successful build.The story is not about watchdog, is about bee_backed output bufferization. I found clang++ with OOM in /var/log/syslog. Github-hosted runners have 7GB RAM - https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners. Self-hosted runner with 16GB RAM must have in my case

Please flush to a build log an buffered output from commands (like bee_bakend) periodically. It gives developers more information about build issues on remote machines.

 

You saved my life, I can't thank you enough. Please someone give this man a medal.

I can confirm this bug is happening in 2021.3.4f1 LTS and in 2022.1.4f1. Please, Unity guys, we like you so much, solve that for us! We buy you a beer!

 

I encountered the problem of that bee freezing back when i used openSUSE distro (yes, i know its not officially supported). On Ubuntu 20.04 it worked fine, and on Alt Linux it worked fine too for me.

 

Same issue as @Sergey_OneUp_Games also using Game.ci

Can confirm that this is occurring when performing a build using Github Actions on a Linux runner when building for Android. Does not happen on my local machine when perform build via editor on a Mac machine. This is on a "blank" project with nothing in it besides the default. No extra/special packages. Editor version: 2021.3.3f1

Local build is ~3mins. Cloud build: ~15mins

Log:

Code (CSharp):

1. 2022-07-03T03:03:56.2376560Z *** Tundra requires additional run (32.05 seconds), 161 items updated, 667 evaluated
2. 2022-07-03T03:03:56.2377661Z *** Additional run caused by: contents change of /github/workspace/Library/Bee/artifacts/Android/il2cppOutput/cpp
3. 2022-07-03T03:03:56.2379673Z Starting: /opt/unity/Editor/Data/Tools/netcorerun/netcorerun "/opt/unity/Editor/Data/PlaybackEngines/AndroidPlayer/AndroidPlayerBuildProgram.exe" "/opt/unity/Editor/Data/PlaybackEngines/AndroidPlayer/Bee:/opt/unity/Editor/Data/Tools/BuildPipeline" "Library/Bee/Player0857283b.dag.json" "Library/Bee/Player0857283b-inputdata.json" "Library/Bee/buildprogram2.traceevents"
4. 2022-07-03T03:03:56.2381141Z WorkingDir: /github/workspace
5. 2022-07-03T03:03:56.9272142Z ExitCode: 0 Duration: 0s712ms
6. 2022-07-03T03:03:56.9286244Z Starting: /opt/unity/Editor/Data/bee_backend --profile="Library/Bee/backend_profiler3.traceevents" --stdin-canary --dagfile="Library/Bee/Player0857283b.dag" --continue-on-failure --dagfilejson="Library/Bee/Player0857283b.dag.json" Player
7. 2022-07-03T03:03:56.9291787Z WorkingDir: /github/workspace
8. 2022-07-03T03:11:20.8181532Z ExitCode: 0 Duration: 7m:23s
9. 2022-07-03T03:11:20.8197839Z Finished compiling graph: 1352 nodes, 3586 flattened edges (3580 ToBuild, 8 ToUse), maximum node priority 814

Not great, considering every minute of stalled time in a cloud build is time wasted. @jonas-echterhoff

 

i am using this automated build system https://game.ci/
and bee_backend is taking too much time, which is not ok on ci

 

In headless builds, cancel button behavior should not result in a hanging build.

This has not been an issue for us in 2020 LTS, but since 2021 all our headless builds fail after a 60 minute timeout after this command.

Is there any chance this could be fixed in the next LTS update?

 

I am seeing the same thing!

I just upgraded a project from 2020.1.15f1 to the latest 2021.3.8f1 LTS and I am hitting this problem. I am using game.ci with a linux ubuntu runner and it stalls on bee_backend

Code (CSharp):

1. *** Tundra requires additional run (42.88 seconds), 446 items updated, 2564 evaluated
2. *** Additional run caused by: contents change of /github/workspace/Library/Bee/artifacts/Android/il2cppOutput/cpp
3. Starting: /opt/unity/Editor/Data/Tools/netcorerun/netcorerun "/opt/unity/Editor/Data/PlaybackEngines/AndroidPlayer/AndroidPlayerBuildProgram.exe" "/opt/unity/Editor/Data/PlaybackEngines/AndroidPlayer/Bee:/opt/unity/Editor/Data/Tools/BuildPipeline" "Library/Bee/Player0857283b.dag.json" "Library/Bee/Player0857283b-inputdata.json" "Library/Bee/buildprogram2.traceevents"
4. WorkingDir: /github/workspace
5. ExitCode: 0 Duration: 1s169ms
6. Starting: /opt/unity/Editor/Data/bee_backend --profile="Library/Bee/backend_profiler3.traceevents" --stdin-canary --dagfile="Library/Bee/Player0857283b.dag" --continue-on-failure --dagfilejson="Library/Bee/Player0857283b.dag.json" Player
7. WorkingDir: /github/workspace
8. Error: The operation was canceled.

It works fine when building the player on my local windows 10 machine.

 

Try add more RAM to your runner and try again

 

+1 on this bug. It makes GitHub builds on their server unusable. :-/

 

We also seem to (again) have this on Apple Silicon Macs.

 

also having this issue game.ci selfhosted Github Actions Linux runners when building for Windows & WebGl

 

We have this problem as well, on Windows!

The log shows:

WorkingDir: G:/repos/photonjam
ExitCode: 0 Duration: 2m:34s
[BUSY 6s] Csc Library/Bee/artifacts/1900b0aEDbg.dag/photonjam.dll (+2 others)
[BUSY 16s] Csc Library/Bee/artifacts/1900b0aEDbg.dag/photonjam.dll (+2 others)
[BUSY 26s] Csc Library/Bee/artifacts/1900b0aEDbg.dag/photonjam.dll (+2 others)
[BUSY 36s] Csc Library/Bee/artifacts/1900b0aEDbg.dag/photonjam.dll (+2 others)
[BUSY 46s] Csc Library/Bee/artifacts/1900b0aEDbg.dag/photonjam.dll (+2 others)
[BUSY 56s] Csc Library/Bee/artifacts/1900b0aEDbg.dag/photonjam.dll (+2 others)
[BUSY 66s] Csc Library/Bee/artifacts/1900b0aEDbg.dag/photonjam.dll (+2 others)
[BUSY 76s] Csc Library/Bee/artifacts/1900b0aEDbg.dag/photonjam.dll (+2 others)
[BUSY 86s] Csc Library/Bee/artifacts/1900b0aEDbg.dag/photonjam.dll (+2 others)
[BUSY 96s] Csc Library/Bee/artifacts/1900b0aEDbg.dag/photonjam.dll (+2 others)
[BUSY 106s] Csc Library/Bee/artifacts/1900b0aEDbg.dag/photonjam.dll (+2 others)
[BUSY 116s] Csc Library/Bee/artifacts/1900b0aEDbg.dag/photonjam.dll (+2 others)
[BUSY 126s] Csc Library/Bee/artifacts/1900b0aEDbg.dag/photonjam.dll (+2 others)
[BUSY 136s] Csc Library/Bee/artifacts/1900b0aEDbg.dag/photonjam.dll (+2 others)
[BUSY 146s] Csc Library/Bee/artifacts/1900b0aEDbg.dag/photonjam.dll (+2 others)
[465/474 154s] Csc Library/Bee/artifacts/1900b0aEDbg.dag/photonjam.dll (+2 others)
[467/474 0s] ILPostProcess Library/Bee/artifacts/1900b0aEDbg.dag/post-processed/photonjam.editor.dll (+pdb)​

Our photonjam assembly contains just 13 files and every change resulted in 2 minutes or more of compile time... thats ridiculus
Regenerating the lib folder helps for a few cycles but the bug still comes bag
I do not understand why --stdin-canary is even needed... Canceling the compilation of scripts when tabbing back into Unity is not even a thing (no button on the dialog). If thats the only reason for the parameter just get rid of it when automatically reloading scripts! When doing a build the two minutes are irrelevant so there its not a big problem...

Well as i am on Windows right now, a shell script was not an option so i resorted to a lil custom c# program... Blazingly fast compilation now

Code (CSharp):

1. using System;
2. using System.Collections.Generic;
3. using System.Diagnostics;
4.  
5. namespace bee_backend
6. {
7.     class Program
8.     {
9.         private static void Main(string[] args)
10.         {
11.             var newArgs = new List<string>(args);
12.             //which one is it?... no idea... it works...
13.             newArgs.Remove("--stdin-canary");
14.             newArgs.Remove("stdin-canary");
15.             var arguments = string.Join(" ", newArgs.ToArray());
16.             Console.WriteLine("Running bee with arguments: " + arguments);
17.             var p = new Process();
18.             p.StartInfo.UseShellExecute = false;
19.             p.StartInfo.RedirectStandardOutput = true;
20.             p.StartInfo.FileName = "bee_backend_real.exe";
21.             p.StartInfo.Arguments = arguments;
22.             p.Start();
23.         }
24.     }
25. }

 

Bumping this because it's a big problem. I'm fine with "slow" builds, but a recent version of our app using gameCI and github hosted build runners takes 75 minutes. One of the bee_backend compilation steps takes 39 minutes with no meaningful output while it's running.

Code (csharp):

1. 2022-10-12T17:39:15.1283083Z Starting: /opt/unity/Editor/Data/bee_backend --profile="Library/Bee/backend_profiler3.traceevents" --stdin-canary --dagfile="Library/Bee/Player0857283b.dag" --continue-on-failure --dagfilejson="Library/Bee/Player0857283b.dag.json" Player
2. 2022-10-12T17:39:15.1284665Z WorkingDir: /github/workspace
3. 2022-10-12T18:18:20.3971266Z ExitCode: 0 Duration: 39m:05s
4. 2022-10-12T18:18:20.5102106Z Finished compiling graph: 2405 nodes, 6741 flattened edges (6733 ToBuild, 10 ToUse), maximum node priority 1637

 

Ci builds failed on 2021.3.11f1 due to this Bee System, Any workaround to fix this on game-ci?

 

On windows I am using the bee_backend from editor 2021.3.7f1 at the moment and it is blazingly fast, might be an adaptable workaround on linux as well...

 

i got compiling assembly always failed on unity2021.3.8f1 ~ 2021.3.15f1 till now....it's looks the bee cause it.

 

On 2021.3.16f1 with game-ci our WebGL project build time is about 1h 15min of which about 15 minutes is the project build and the rest is waiting for the bee. Our VM has 16GB ram, does it really need more? I don't get this when I build on my linux laptop locally with 32GB ram

 

Any workaround to fix the issue on game ci???

 

We see this across all build VMs now, using Unity 2021.3.16f1.....
On MacOS, the custom batch script creates error code 256 in editor so that does not seem to be an option(?)

 

Having the same issue on Unity 2021.3.21

 

I was facing the same issue. Freeing up disk space on Ubuntu runner worked for me.
Followed this medium tutorial. Added the below step in my yml file.

Code (JavaScript):

1. - name: Free Disk Space for Android Build
2.         run: |
3.           df -h
4.           sudo swapoff -a
5.           sudo rm -f /swapfile
6.           sudo rm -rf /usr/share/dotnet
7.           sudo rm -rf /opt/ghc
8.           sudo rm -rf "/usr/local/share/boost"
9.           sudo rm -rf "$AGENT_TOOLSDIRECTORY"
10.           df -h

You can also try deleting unused docker images

Code (JavaScript):

1. - name: Clean up Docker resources
2.         run: |
3.           docker system prune -a --volumes --force

 

Hi I'm having issues with a build at the moment running on a docker-windows executor with a game-ci based image. It is not with linux but I get a some logs related to bee that I don't get on my computer.

Unity 2021.3.28f1

Code (CSharp):

1. Refreshing native plugins compatible for Editor in 6.64 ms, found 14 plugins.
2. Starting: C:\Program Files\Unity\Hub\Editor\2021.3.28f1\Editor\Data\bee_backend.exe --profile="Library/Bee/backend_profiler0.traceevents" --stdin-canary --dagfile="Library/Bee/1900b0aE.dag" --continue-on-failure ScriptAssemblies
3. ExitCode: 4 Duration: 2s499ms Require frontend run.
4. Library/Bee/1900b0aE.dag couldn't be loaded

Later my builds are failing because and editor script from fmod fails to find a native dll

Someone is having a similar issue in linux

https://discussions.unity.com/t/fal...assembly-type-unknown-type-member-null/269749

And here is my post in fmod forums:

https://qa.fmod.com/t/docker-windows-dllnotfoundexception-fmodstudiol/20491

 

Just started happening for me on Arch Linux using 2021.3.22 and 2021.3.29, was working fine yesterday. The problem happens even when creating a fresh project.

Tried rolling back any recent system updates, but it didn't help.

2022 series Unity seems to be working fine. The shell script workaround helped.

 

I also tried the tampered exe that @Odd_Hero posted but even though the "couldn't be load" went away then the project fails to load DLLs and fails to execute the external method so it didn't fix the issue for me.

 

Hi, since yesterday I've also got this problem on Arch Linux with Unity 2021.3.15 and 2021.3.23, either when opening existing projects or creating new ones.

I'll try the workaround and I'll let you now.

[Edit]

I can confirm that the workaround solved the problem, thank you!
I hope it will be properly fixed in a future release.

Bye

 

Same issue here: on Arch, recently updated. Opening a previous Unity project or a new one results in a stuck "Initial Asset Database Refresh" dialog box.

Thanks for the post @neamtim, your solution works!

ETA: I'm using Unity 2022.1.23f1.

 

This bug appeared on Manjaro (arch based) a week ago, for me all 2021.3 versions are affected, but 2022 can open projects freely. I've used same machine and same 2021.3 version for 3 month, but since previous week it gets broken after a while

Workaround in post did help open a project, but I have "Internal build system error. Backend exited with code 255." error which prevents playing the game.

For me, it happens on old projects and on freshly created ones

 

I also had a 'Backend exited with code 255' error.

The fix for me was to edit the custom bee_backend script and remove the tab characters from the start of each line. They are added unexpectedly when copying code from the forums. (Made worse by the fact I used Dolphin's feature to paste text as a new file without ever opening the script.)

 

Wow, it really was the issue! Thanks a lot for the help!

 

I use Manjaro 23.0.0. Unity worked good for last half of the year. (Well, not good. It is Unity, it can not work good)
And yesterday I got this issue. None of the projects open. Even if I create new empty project, the same thing happens.
Tried a bunch of 2021 versions. The same thing. It opens with 2022, but it is not what I need.
I am glad, I found this page. It solved the issue for now.

 

it seems this bug started happening on arch linux distros.
this solution fixed my problem.

 

Thank you @neamtim!
For me it fixed the issue of any project being stuck on "initialize asset database" step (arch linux).

So fun we still have to use hacks from 2 years ago to fix the bugs now

 

Same here, Arch Linux seems to have a configuration that causes this issue.
The fix works, thanks!

I'm guessing it's due to a custom shell? I use fish as my login shell which might mess with the pipeline IPC shenanigans.

 

In Arch just recently update UnityHub from AUR and can't import or update projects only create new ones.

You save my life, thanks man.

 

Happened to me on Archlinux, unity editor 2021.3.18f1 after a system update. This workaround works. Thanks a lot! Saved my life!

 

Thank you so much, this post saved my life.

 

Thank you so much, this post saved my life too!!!

 

Never encountered this issue before upgrading Ubuntu to 23.10 and trying to launch any project under any 2021.3.x version. This script saved my work, literally.

Is there any bug report we can upvote about this?

 

I can't believe this error is still here.
It fixed my project too. Thank you very much.

 

I got the stuck on import hang after upgrading to ubuntu 23.10. There is also an issue tracked here: https://issuetracker-mig.prd.it.uni...t-open-or-create-projects-on-ubuntu-23-dot-10

 

this script saved my job and my life, thanks!

 

I just came here to thank you for this genius solution!
Outstanding!

 

Thanks. This is still an issue. Still fixed by the above.

 

Wow! Thank you so much for the workaround! This fixed my Arch Unity install!

 

Thank you so much, this workaround works for me on my Manjaro installation with Unity 2021.3.16f1!
This really should be fixed, or at least acknowledged in the Unity Hub somehow. I couldn't open a team project because of this bug and it was quite frustrating (especially as I had no problems with versions 2022.x before).

 

Great work neamtim! Still had this issue on Fedora with Unity version 2021.3.25. Your approach of removing the "--stdin-canary" flag fixed it for me. Made a python version of your fix:

- go to your unity editor Data folder (e.g. [...]/unity/2021.3.25f1/Editor/Data)
- rename bee_backend to bee_backend_real
- create a bee_backend file, make it executable (chmod +x bee_backend)
- write the following in the newly created bee_backend file

Code (python):

1. #!/usr/bin/env python3
2. import sys
3. import subprocess                      
4. args = [arg for arg in sys.argv[1:] if arg != '--stdin-canary']
5. subprocess.run([sys.argv[0] + '_original'] + args)