Search Unity

  1. Welcome to the Unity Forums! Please take the time to read our Code of Conduct to familiarize yourself with the forum rules and how to post constructively.
  2. Dismiss Notice

2019.4.18 - CLI build fail (Editor works) - Emscripten [Error 6] The handle is invalid

Discussion in 'WebGL' started by MikeSz, Jan 22, 2021.

  1. MikeSz

    MikeSz

    Joined:
    Jul 7, 2014
    Posts:
    26
    Hello,

    I hope I did not miss it in search

    We have a problem with building our upcoming games (2 of them) using Command Line Interface. When using the very same build command from editor everything works fine

    We tried it on 2 different PCs with 2019.4.16 and 2019.4.18 and they receive the same error. This one is from TeamCity build chain, but it looks the same (sans paths) on my local machine. Second game results in exactly identical issue and again builds just fine from editor itself. Android Builds also build just fine from CLI, it's just WebGL that is causing headache. New/empty WebGL projects build fine. So it's not an issue with machines or Unity installation itself

    I am removing parts of the pathnames for clarity

    [UNITY] -> Unity Install Location \ WebGLSupport\BuildTools

    --- code follows (super long lines, was breaking code tags) ---

    Filename: "[UNITY]\Emscripten_Win\python\2.7.5.3_64bit\python.exe"
    Arguments: -E "[UNITY]\Emscripten\emcc"
    @"D:\AgentUnity1\work\37c533f1543ab8a2\Assets\..\Temp\emcc_arguments.resp"
    index: -1
    Failed running "[UNITY]\Emscripten_Win\python\2.7.5.3_64bit\python.exe" -E
    "C[UNITY]\Emscripten\emcc"
    @"D:\AgentUnity1\work\37c533f1543ab8a2\Assets\..\Temp\emcc_arguments.resp"

    stdout:
    stderr:Traceback (most recent call last): File "[UNITY]\Emscripten\emcc.py", line 3063, in <module> sys.exit(run()) File "[UNITY]\Emscripten\emcc.py", line 1632, in run extra_files_to_link += system_libs.calculate([f for _, f in sorted(temp_files)] + extra_files_to_link, in_temp, stdout_=None, stderr_=None, forced=forced_stdlibs) File "[UNITY]\Emscripten\tools\system_libs.py", line 520, in calculate symbolses = shared.Building.parallel_llvm_nm(list(map(os.path.abspath, temp_files)))
    File "[UNITY]\Emscripten\tools\shared.py", line 1844, in parallel_llvm_nm object_contents = pool.map(g_llvm_nm_uncached, files)
    File "[UNITY]\Emscripten_Win\python\2.7.5.3_64bit\lib\multiprocessing\pool.py", line 250, in map return self.map_async(func, iterable, chunksize).get()
    File "[UNITY]\Emscripten_Win\python\2.7.5.3_64bit\lib\multiprocessing\pool.py", line 554, in get raise self._valueWindowsError: [Error 6] The handle is invalid

    ---

    Overall: [Error 6] The handle is invalid

    The command used was rather simple, just project path, execute method and the standard "-quit -nographics -batchmode -buildTarget WebGL"

    We spent quite some time on Google looking for some suggestions, but things we found (like removing jslibs from project) did not help

    Full log is >8 MB and contains things that we cant show due to NDA, but here's a redacted part starting with Emscripten linking process:
    https://imdev.pl/share/6c84886a-8ae3-491d-b8ad-ea23c8f26710/buildlog_webgl_dev.txt

    We will be very helpful for any help here
     
  2. jukka_j

    jukka_j

    Unity Technologies

    Joined:
    May 4, 2018
    Posts:
    940
    One thing to try to diagnose is to set global environment variables EMCC_CORES=1 and/or EMCC_DEBUG=1 , then rerun. First one will disable Emscripten multithreading, second one will add more debug printing to Emscripten.
     
  3. MikeSz

    MikeSz

    Joined:
    Jul 7, 2014
    Posts:
    26
    @jukka_j
    I added those but same error unfortunately :( And no extra debug info. Unless Emscripten keeps it logs somewhere else that is

    2019.4.18f1 - cutting full folder names

    ---

    Filename: "BuildTools\Emscripten_Win\python\2.7.5.3_64bit\python.exe"
    Arguments: -E "BuildTools\Emscripten\emcc" @"D:\Work\PROJECT\Assets\..\Temp\emcc_arguments.resp"
    index: -1
    Failed running "\BuildTools\Emscripten_Win\python\2.7.5.3_64bit\python.exe" -E "\BuildTools\Emscripten\emcc" @"D:\Work\PROJECT\Assets\..\Temp\emcc_arguments.resp"

    Emscripten_Win\python\2.7.5.3_64bit\lib\multiprocessing\pool.py", line 554, in get raise self._valueWindowsError: [Error 6] The handle is invalid

    ---

    Im not sure why its calling multiprocessing part of library (or maybe it doesnt matter?) but the line 554 relates to error raising from:

    #
    # Class whose instances are returned by `Pool.apply_async()`
    #

    class ApplyResult(object):

    PS: it works just fine in 2020.2 but we have significant in-game issues and bugs there, so cant upgrade easily
     
    Last edited: Feb 10, 2021
  4. jukka_j

    jukka_j

    Unity Technologies

    Joined:
    May 4, 2018
    Posts:
    940
    Sorry to hear. If you are able to reduce the failure down to a small test case, which fails in 2019.4.18f1 but works in 2020.2, you can try reporting that as a bug. It sounds like it is an issue that has been fixed, but the fix has not made its way back to 2019.4.18f1. Maybe a bug report can help QA bisect down where the particular fix happened.
     
  5. MikeSz

    MikeSz

    Joined:
    Jul 7, 2014
    Posts:
    26
    Thanks @jukka_j

    Unfortunately not much I can do. I spent the whole day yesterday just binary-search-building through commits history and it led me absolutely nowhere

    I did get one commit to build and it was from right before our 2019.3->2019.4 transition. No further commits built properly, so I went back to the one that worked and... It failed with Handle Error 6. Restarted my PC, re-downloaded that commit, cleaned up library and everything. Will not build ever again

    We are at a complete 100% loss, been looking at it since early December and tried basically every single thing...
     
  6. jukka_j

    jukka_j

    Unity Technologies

    Joined:
    May 4, 2018
    Posts:
    940
    Searching the web, there are similar odd python multiprocessing issues reported, e.g. here: https://stackoverflow.com/questions...he-handle-is-invalid-python-3-multiprocessing . The Emscripten project ( https://github.com/emscripten-core/emscripten ) uses Python multiprocessing for parallel task spawning.

    However I am surprised that the issue would occur even when EMCC_CORES=1 is set. Emcripten should disable multiprocessing with EMCC_CORES=1 environment variable is set. Can you try double-checking that option? (e.g. set it as a global env. var for all users, and restart the computer for good measure)

    It sounds like there is some nondeterminism in play that is causing a flaw in python-Emscripten interaction.
     
  7. MikeSz

    MikeSz

    Joined:
    Jul 7, 2014
    Posts:
    26
    Thanks @jukka_j

    I tried multiple options and collected just the error logs in the linked zip file

    Before that:
    - reinstalled Unity, 2019.4.20f1
    - deleted project + downloaded and imported from scratch

    There are:
    - baseline - with no system environment variables
    - just EM_POPEN_WORKAROUND=1 - which was suggested in some places

    - EM_POPEN_WORKAROUND=1 + EMCC_CORES=1
    - EM_POPEN_WORKAROUND=1 + EMCC_CORES=1 + EMCC_DEBUG=1
    - EMCC_CORES=1
    - Adding DEBUG caused a weird "PYTHON not defined" error so I added python.exe location from Unity Install to environment variables but this changed nothing so I skipped this log

    I restarted Windows before each and every test

    Handle Error 6... always

    https://imdev.pl/share/a51d56ff-8bb9-4d34-a3fc-8a20710056c2/ErrorLogs.zip
     
  8. jukka_j

    jukka_j

    Unity Technologies

    Joined:
    May 4, 2018
    Posts:
    940
    Other threads on the web also mention about python multiprocessing having this bug:
    - https://stackoverflow.com/questions...he-handle-is-invalid-python-3-multiprocessing
    - https://bugs.python.org/issue37380
    - https://stackoverflow.com/questions...vice-oserror-winerror-6-the-handle-is-invalid (these workarounds are what the EM_POPEN_WORKAROUND=1 are doing, I authored that flag in 2013 to Emscripten, at https://github.com/emscripten-core/emscripten/pull/1631 )

    In 2013 I seem to have observed that having 32-bit python and 64-bit LLVM installed can cause a "
    WindowsError: [Error 6] The handle is invalid" error: https://github.com/emscripten-core/emscripten/pull/1958 . I wonder if you have 32-bit python installed manually? Maybe removing that from PATH or uninstalling it to try could help.

    Ultimately, last weekend I decided to migrate Emscripten away from using Python multiprocessing at all. See these two fixes:
    - https://github.com/emscripten-core/emscripten/pull/13465
    - https://github.com/emscripten-core/emscripten/pull/13493

    That should resolve all these multiprocessing troubles for good.

    Unfortunately it will take a short while until those make their way to Unity, we are targeting to get those for Unity 2021.2 release. In the meanwhile, I would recommend to work around using a different PC installation if nothing else helps. Apologies for such a nuclear suggestion, but I don't have anything better to offer here, given that the issue is so elusive.
     
  9. MikeSz

    MikeSz

    Joined:
    Jul 7, 2014
    Posts:
    26
    Damn...

    Well, since I spent further 2 days working on it... Might as well share

    I tried 3 things:
    - EMCC_CORES
    - EMCC_DEBUG
    - direct change in emscripten.py
    Like I said in last post - those did not help

    But after this I started looking closely at the log and the very last thing that was linked was Live2D animation plugin, with their webgl .bc library. So I removed it (breaking the project in the process) and... it compiled fine

    Now the fun part - we have another project in development that is using Live2d and builds with no issues. It's Live2d plugin library is a bit newer so I copied it and...

    It built. Almost opened the champagne but decided to give it one more try... FAIL

    And that's when we gave up to enjoy the weekend

    On Monday morning I discovered that editor will not make builds either, mostly due to changes to Environment Variables and python scripts. So I removed ENV Variables, restored Python files and... NADA, FAIL

    Completely reinstalling 2019.4.20 did fix it fortunately but I'm now too scared and too scarred to touch it :(

    PS My early tests in 2020.2 showed the game was building fine there but... There was enough broken bits that we didnt follow with it (and 2020.1 breaks the way WebGL index and template files are handled, so we would have a lot of extra work with it as well. Unstable build + those changes + release window for the game = nope, not gonna risk it now)
     
  10. MikeSz

    MikeSz

    Joined:
    Jul 7, 2014
    Posts:
    26
    Just a quick update @jukka_j - nuclear option worked and Mac was able to build it fine. Thus we assume that so should Linux, so we will move the build process to a VM on our server. Feels safer than updating a Live game to 2020.2
     
  11. jukka_j

    jukka_j

    Unity Technologies

    Joined:
    May 4, 2018
    Posts:
    940
    Between 2019.4 and 2020.2 there have been no changes to the Emscripten toolchain that is used, so it is expected that whatever failures would happen in one, would also occur in the other. It sounds like there is some amount of randomness in play that the issue does not occur all the time, leading to occassional successful or failed builds.

    Linux is quite different from Mac in general. On Linux it may boil down to what distro and distro version is being used, and one may need to apt/dpkg install some packages outside Unity to get Emscripten builds to work. The Handle is Invalid error should definitely not occur on Mac or Linux, that is for sure.
     
  12. MikeSz

    MikeSz

    Joined:
    Jul 7, 2014
    Posts:
    26
    @jukka_j ok, we decided to bite the bullet and move to Unity 2020.3.3 only to... get bitten by this abomination of an error

    Is there any chance this fix make it to main Unity anytime sooner?
     
  13. MikeSz

    MikeSz

    Joined:
    Jul 7, 2014
    Posts:
    26
    @jukka_j any chance you could look here? do you maybe have a working but "not tested" Emscripten that we could replace manually just to unlock the build process? This is insanely frustrating issue :(
     
  14. jukka_j

    jukka_j

    Unity Technologies

    Joined:
    May 4, 2018
    Posts:
    940
    I think I read conversations in our Slack about this issue just recently. If I understood the issue correctly, it could be something to do with the way that the Python interpreter takes in stdin handles, and Unity build system feeds stdin input over to the Python subprocess it spawns.

    I don't unfortunately have a fix or a workaround to propose right now, but prodded an internal conversation about this to find out more.
     
  15. jukka_j

    jukka_j

    Unity Technologies

    Joined:
    May 4, 2018
    Posts:
    940
    Btw, we have not received a single bug report about this, so would be good if you could raise a bug report so our QA can try reproing it as well.
     
  16. MikeSz

    MikeSz

    Joined:
    Jul 7, 2014
    Posts:
    26
    Thanks for remembering about this

    I really wish I could, but the projects are... NDAd, large and contain paid plugins. I will see what happens if I just try to build an empty test project, but since there are no errors I assume it will work without problems

    Like I said before - based on the full output log I traced that the very last thing that goes to the linker is Live2d plugin. So I will also try to include in the test project and see what happens
     
  17. MikeSz

    MikeSz

    Joined:
    Jul 7, 2014
    Posts:
    26
    @jukka_j hah. Found it

    The cause is Live2d as I suspected. I tried building just an empty Unity project with their Unity plugin and nothing else - and it broke in the very same way as above. I contacted them but who knows how much time it will take them to reply, so maybe you have some idea?

    It looks like during the CLI build (but not during editor build) Live2d opens... another process that... does nothing it seems, just prints the plugin version


    But it IS a new process and [ERROR 6] is directly related to handling multiple processes. So I'd blame it

    Unfortunately I don't see anything within their plugin that spawns that process - so it's probably part of the plugin's library

    Is there any way of blocking spawning additional processes during the build process? Maybe that would be enough... dirty hack to save us buying and setting up additional non-Windows build machine...

    Here's the test project - again, just empty Unity + Live2d sample
    https://imdev.pl/share/d36110cb-a13c-43e0-b51e-05cc8e05f374/Live2dCLITest.zip

    And it can be built using the command (adapt the paths of course)
    d:\Work\Unity\2020.3.3f1\Editor\Unity.exe -quit -nographics -batchmode -buildTarget WebGL -projectPath d:\Work\Live2dCLITest -logFile D:\BuildWebGL.txt -executeMethod BuildSystem.Build_WebGL

    Thanks for help
     
  18. jukka_j

    jukka_j

    Unity Technologies

    Joined:
    May 4, 2018
    Posts:
    940
    Sorry, cannot quite think of a way to prevent C# code from spawning a subprocess like that. It would be great if the plugin authors are able to react, and if they think the usage should be fine, have them report a bug against Unity - they can probably give more information about their required use case on the plugin.
     
  19. MikeSz

    MikeSz

    Joined:
    Jul 7, 2014
    Posts:
    26
    @jukka_j damn, thanks anyway. Yes, I wrote them but it's a Japanese company and I don't expect to see much help from them :(
     
  20. MikeSz

    MikeSz

    Joined:
    Jul 7, 2014
    Posts:
    26
    @jukka_j it took some time but finally got a reply from there. Might be of interest to you so Im posting it here:

    So as I understand we're basically out of options other than Linux/Mac until... Unity 2021.2, right?
     
  21. jukka_j

    jukka_j

    Unity Technologies

    Joined:
    May 4, 2018
    Posts:
    940
    Thank you, that is most appreciated!

    We have been suspecting that the root issue here is in Python 2.7 branch. Unity 2021.2 Beta channel when it rolls forward should have Python 2 replaced with Python 3 along with utilizing Emscripten 2.0.19, which should make this issue vanish completely. Although the fact that https://bugs.python.org/issue3905 has not actually been closed for Python 3.x line makes me worry.

    We contributed a drastic preventive measure into Emscripten 2 tree at https://github.com/emscripten-core/emscripten/pull/13493 that removed the other source of the Handle Is Invalid error we know about, related to Python multiprocessing. After that change, shipped in Emscripten ~2.0.10 or so, Emscripten no longer uses Python multiprocessing to drive multithreaded compilation.

    Unfortunately I am at a loss to suggest a way to resolve this, the original root Python bug was closed as a Wontfix as Python 2 is at end of life. What makes this issue particularly unbearable is that we are not able to reproduce it - the issue happens only on some systems and never on others.
     
  22. MikeSz

    MikeSz

    Joined:
    Jul 7, 2014
    Posts:
    26
    Thank you. This is weird indeed. However based on our limited tests this issue happens 100% for this specific plugin. We tried it on a few different machines and different Unity versions - always with same result

    But other than that... We are clueless. And since the plugin doesn't have full source available (there's a .bc library there) we can't really dig into it

    Btw - tried building it on 2021.2 alpha (not that the game worked properly, but heck, why not try?) and... the build in CLI crashed in an even more interesting way. Editor build worked fine of course..

    Created GICache directory at C:/Users/MikeSz/AppData/LocalLow/Unity/Caches/GiCache. Took: 0.034s, timestamps: [260.236 - 260.270]
    gi::BakeBackendSwitch: switching bake backend from 3 to 1.
    [00:00:00] Precompute started.
    Received signal SIGSEGV
    Stack trace:
    RtlLookupFunctionEntry returned NULL function. Aborting stack walk.
    <Missing stacktrace information>

    We're cursed
     
  23. jukka_j

    jukka_j

    Unity Technologies

    Joined:
    May 4, 2018
    Posts:
    940
    If you do have a project where it always occurs for you and the project is something that can be zipped over, we should certainly try reporting that project as a bug - glancing through the conversation history looks like we did not try that yet.

    Oh dear, looks like something in the GI cache.. is this crash deterministic? Maybe that same project repro will be reporting two bugs... :oops:
     
  24. MikeSz

    MikeSz

    Joined:
    Jul 7, 2014
    Posts:
    26
    Indeed, I did not report it as a bug as I wasnt sure if this is a problem on Unity or plugin side. But the project was attached above, pasting URL to it once more below. Should I report it somehow?

    https://imdev.pl/share/d36110cb-a13c-43e0-b51e-05cc8e05f374/Live2dCLITest.zip

    That error was in our live project, cant share it :( But it's a fully 2D project, there's no GI, no lighting at all. Why would it fail on GI... Is there any way to kill GI completely so I can verify it builds properly? EDIT: Just tried again. created new Lighting settings file, unchecked GI and everything related to lighting. And got the same CLI crash as before

    Oh, and Live2D Plugin DID build in 2021.2.0a17 without a hiccup

    So overall - pre-2021.2 our games will not build due to an issue with Live2D+Emscripten, 2021.2 will not build due to GI error (and we don't use GI). But regardless, I don't think alpha version of Unity is ready for primetime and certain widely used plugins (e.g. Doozy UI) do not support it at all and go into infinite loop being unable to create some required assets
     
    Last edited: May 26, 2021
  25. jukka_j

    jukka_j

    Unity Technologies

    Joined:
    May 4, 2018
    Posts:
    940
    Please do, using the built-in Unity bug reporter UI. That should send the project directly over to the QA to try to reproduce.