Search Unity

  1. Are you interested in providing feedback directly to Unity teams? Sign up to become a member of Unity Pulse, our new product feedback and research community.
    Dismiss Notice

Include native C++ code into WebGL build - how?

Discussion in 'WebGL' started by larku, Mar 5, 2015.

  1. larku

    larku

    Joined:
    Mar 14, 2013
    Posts:
    1,421
    I've got a project that I currently build native libraries for Android, iOS, Linux and Windows Standalone - I am very keen to get a WebGL build working.

    My native library works well on all the mentioned platforms - does this provide any indication if it will also work for WebGL?

    I currently use posix sockets, pthreads, etc - for the Windows Standalone dll I'm using mingw32/64 on linux, for Android I use the ndk on Linux, for iOS I use clang++

    From the brief bits I've seen online it appears that I can drop my source code somewhere and Unity will build it as part of the build process.

    Questions:
    1. Where do I put the source to achieve this?
    2. How would I reference the native code from c#? (does it use the same [DllImport ("__Internal")] notation as an iOS AOT build?)
    3. What about compiler flags, include folder, etc.
    4. Are there any restrictions on what C++ code can be included? Are all the standard libraries available? For example: STL?
    5. What C++ compiler is used by Unity for this process?
    6. Any pointers do some documentation, how-to's, resources of any kind?

    Any assistance will be greatly appreciated.
     
  2. ShabihDesperado

    ShabihDesperado

    Joined:
    Oct 27, 2013
    Posts:
    41
  3. larku

    larku

    Joined:
    Mar 14, 2013
    Posts:
    1,421
    Thanks @ShabihDesperad,

    I've read that post and unfortunately it only states:

    It then leverages the standard platform C++ compilers to produce native binaries.

    Which for iOS/OSX is Xcode, but for Windows I'm not sure - maybe mingw ? But I've since discovered that Unity WebGL uses emscripten to convert LLVM byte code to JS - so mingw is most likely not the beast.. I expect there must be a LLVM compiler for Windows or perhaps bundled with emscripten in some way, I'm just researching emscripten now to get a feel for what it offers, its limitiations and how to go about building my code with LLVM and then emscripten'ing it.

    I'll keep this thread updated with my findings - I expect the Unity devs already know all the answers here so it'd be great if they chime in :)
     
  4. Dantus

    Dantus

    Joined:
    Oct 21, 2009
    Posts:
    5,667
  5. larku

    larku

    Joined:
    Mar 14, 2013
    Posts:
    1,421
    :( that's what I've just found.. Though, probably not a huge effort to refactor it to use web workers (I think that's the term they used) - not sure how to do this in my C++ code to make emscripten happy... Anyone know if they have an abstraction that I should implement to replace my threads with? My threads are mostly just passing messages between thread safe queues anyway so this seems to fit nicely with what JS needs.
     
  6. jonas-echterhoff

    jonas-echterhoff

    Unity Technologies

    Joined:
    Aug 18, 2005
    Posts:
    1,647
    As others wrote: This sounds like it won't be a matter of simply recompiling your code to webgl - as threads are simply not supported by the platform at this point. Neither are posix sockets, though emscripten will try to wrap posix socket code to use web sockets instead, which may or may not be useful depending on your needs.

    I'll still answer your questions as far as i can for reference:

    1. Anywhere in your assets folder, really. You can specify to compile it for WebGL in the inspector on the cpp files.
    2. Yes, [DllImport ("__Internal")] is correct.
    3. You don't really have control over these. Compiler flags are default, include folder should be only the folder your code is in (not quite sure about that). This works well for writing simple wrappers in a handful of .cpp files. For larger codebases, I'd suggest a different approach: Use the emscripten compiler to compile and link your source files to a single .bc file. That way you can specify the command line used yourself. Just drop that .bc file into Unity, and it will be used for the webgl build.
    4. Standard C++ stuff should all work.
    5. emscripten
    6. http://docs.unity3d.com/Manual/webgl-interactingwithbrowserscripting.html - which is not that much, yet, admittedly.
     
  7. larku

    larku

    Joined:
    Mar 14, 2013
    Posts:
    1,421
    Hey @jonas thanks for your detailed reply - much appreciated!

    I've got a couple of question around this - these are not Unity questions so I don't expect you to spend your time solving my problems, but it's worth asking in case you've done this and have answers at hand :)

    posix threads
    Yes it seems the threading will need to be reworked to use 'web workers' instead, this doesn't sound like an overly arduous task but I'll need to track down how to do this. Do you have any idea what I would need to do to the C++ code to enable this? Is there some sort of c++ web worker implementation I could implement in my c++ source that will then seamlessly translate to js using emscripten?

    posix sockets
    Like wise - is there a set method for ensuring my socket usage will convert to web sockets properly? The actual socket code is in RakNet (not Unity's RakNet stuff, I'm using a custom implementation). I'm only using UDP for the RakPeer transport - I'm not sure if this simplifies or complicates matters (I know almost nothing about web sockets - I have a feeling that is going to change soon :) )..

    I've done some searching already and while there are many discussions about both posix sockets and threads all I've read state that you need to use their 'web' counterparts - but no details on how one would do this. I assume that Unity has already had to undertake similar action on their runtime/engine? Any chance of passing on some additional wisdom on this?

    Thanks again for you input already, and like I said, I'm asking questions that are beyond what I'd expect from Unity staff so feel free to reply accordingly :)
     
  8. jonas-echterhoff

    jonas-echterhoff

    Unity Technologies

    Joined:
    Aug 18, 2005
    Posts:
    1,647
    To use web workers from C++ code in emscripten, see here:
    http://kripken.github.io/emscripten-site/docs/api_reference/emscripten.h.html#worker-api

    Note that web workers differ a lot from real threads, and thread code usually does not easily map to web workers. Basically you cannot share memory between workers or with the main thread, and you will only receive data from workers between main thread callbacks (ie, between calls to the unity main loop). Depending on what you want to do, a solution inside of Unity (using coroutines for instance) may be simpler.

    I find it unlikely that you'd get RakNet to work with WebSockets without heavy modifications, and you will not be able to communicate with other peers not using WebSockets.

    A different approach might be to use WebSockets directly from Unity code. I made a little sample wrapper to use JS WebSockets from C# code in Unity here: http://files.unity3d.com/jonas/SimpleWebSockets.unitypackage
     
  9. larku

    larku

    Joined:
    Mar 14, 2013
    Posts:
    1,421
    Wow, speedy reply!

    The web workers would actually map well for my use of threads since I don't really share any memory between them as it is now - they already just pass events/data via some blocking queues which I believe will fit the pattern of web workers. Though having said that, RakNet does use a couple of threads itself - so this could be the downfall, though that may be moot given your opinion that RakNet could be a no-go due to its sockets..

    This is somewhat disappointing as I've invested a lot (time, effort and hair) implementing our game's networking solution on a custom RakNet implementation. It's a binary protocol aimed at hosting large volumes of users on commodity VPS instances (we'll see how that turns out :) ). We're not talking an MMO here, just a two player sport game that passes action via our central 'master' server instances keeps some state, player matching, etc..

    We've targeted mobile initially with the intention of extending that to web and WebGL got me all excited as it seemed like the silver bullet - seems that was some premature excitement!

    The game is almost ready to release - I'm just finalising the scalability of the server now ready for release. But this discussion has now made me consider that I may have made a poor choice for the networking solution.

    Perhaps a possible workaround here is to create a websocket to RakNet bridge on our server side.. Hmmm. (sorry thinking out aloud here)..

    WebSocketClient ---> WebSocketServerBridge+RakNet ---> RakNet server instance

    Or something similar.... Any thoughts?
     
  10. jonas-echterhoff

    jonas-echterhoff

    Unity Technologies

    Joined:
    Aug 18, 2005
    Posts:
    1,647
    That may make sense. For Unity 5.1 we plan to ship our own new networking solution in Unity, UNET, which does something similar (implement a WebSocket bridge on the server to allow WebGL-based clients to connect).
     
  11. larku

    larku

    Joined:
    Mar 14, 2013
    Posts:
    1,421
    Well, if it's good enough for the Unity engineers then it sounds good enough for me! :)

    Thanks again for your great input here!
     
  12. thekend

    thekend

    Joined:
    Jun 16, 2017
    Posts:
    2

    According to these answers I tried to use a C++ library in a WebGL build. The native build works without a problem (uses a native plugin as a dll) but unfortunately I've run into a strange problem with the WebGL build (using the bc file build from the same library). The browser console shows the following log:

    According to the log there's some problem with some global variable initialization and missing a function called llvm_invariant_start_p0i8.

    I've tested that a simple library with some simple functions manipulating global variables is working with WebGL build.

    I use emscripten version 1.37.9 (the latest precompiled sdk version activated through emsdk). Maybe the issue is that Unity uses a slightly different version of emscripten. If so, how can I use the one found in Unity (using the related emscripten, llvm, cmake toolchain etc.)?

    Thank you in advance.
     
  13. Marco-Trivellato

    Marco-Trivellato

    Unity Technologies

    Joined:
    Jul 9, 2013
    Posts:
    1,654
    I recommend using the same version of Emscripten used in Unity, which is a bit older. To know the version, check <install folder>\Editor\Data\PlaybackEngines\WebGLSupport\BuildTools\Emscripten\emscripten-version.txt
     
  14. thekend

    thekend

    Joined:
    Jun 16, 2017
    Posts:
    2
    Hi! Thank you very much for your response. I built the bc with the related version of emscripten (1.37.3), and it works. Awesome.

    Until I called GUI.Label somewhere in an OnGUI callback. Then the build crashes on some TextMeshGenerator destructor stuff (the label was displayed on the first frame).

    How this is related with my c++ plugin I don't know. Something very strange can happen in the background.
     
  15. larku

    larku

    Joined:
    Mar 14, 2013
    Posts:
    1,421
    Hey Jonas,

    Necroing an old thread here - I'm finally getting to this task (long time!, anyway) I'm trying to get the cpp code to build but Unity never attempts to build it, instead I get unresolved externals at link time.

    Using Unity-2017.2.0p3.

    I've tried having my code in the following folders

    .../Assets/native/<source tree here>
    .../Assets/Resources/native/<source tree here>
    .../Assets/Plugins/native/<source tree here>


    My source tree is like:

    native/folder1/FileX.cpp
    native/folder1/FileY.cpp
    native/folder2/FileZ.cpp
    native/folder2/FileA.cpp
    native/File.cpp
    native/File2.cpp
    native/File2.cpp


    (.h header files are in there too)

    But it never attempts to build these cpp files.

    The compilation error I get is:

    C:\unity\unified\project-p\unity\Temp\StagingArea\Il2Cpp\il2cppOutput/Bulk_Assembly-CSharp_3.cpp:63598: undefined reference to `projectX_setGameType

    etc

    These have well formed dllimport statements in the C# code. The same code works fine with unity when I compile it to native libraries and import it that way..

    So where I once had different dllimport names for each platform, I'm now using __Internal and wanting Unity to build and link the code at compile time.

    I've also tried setting the "Select platforms for plugin" on the cpp files to "Any Platform" and also "Android" (targeting Android for this test before I try WebGL)..

    Any ideas?
     
  16. Marco-Trivellato

    Marco-Trivellato

    Unity Technologies

    Joined:
    Jul 9, 2013
    Posts:
    1,654
    - make sure you are exporting C symbols (as opposed to cpp).
    - make sure WebGL is included in the plugin inspector on the cpp files.
    - maybe try to introduce a compile error in a cpp file, to check whether the file is actually compiled.

    By the way, this is only supported on WebGL. It won't work on Android. As per manual: Unity supports native plug-ins for Android written in C/C++ and packaged in a shared library (.so).

    AFAIK, WebGL is the only Unity platforms that supports native plugins as c/cpp files.
     
  17. larku

    larku

    Joined:
    Mar 14, 2013
    Posts:
    1,421
    Oh, only for WebGL, that explains a lot..

    Yep WebGL build and it successfully builds my code (win!) but then get stuck during the compilation process with the status shown "Compile asm.js module".

    CPU idle, everything idle, waited many hours, never finishes. The node.exe process seems to be where it's stuck. If I kill this process Unity gains control again and prints the following error to the console:

    Code (CSharp):
    1. warning: unresolved symbol: glGetInternalformativwarning: unresolved symbol: glUnmapBufferwarning: unresolved symbol: glMapBufferRangewarning: unresolved symbol: glFlushMappedBufferRangeTraceback (most recent call last):
    2.   File "C:\Program Files\Unity-2017.2.1f1\Editor\Data\PlaybackEngines\WebGLSupport\BuildTools\Emscripten\emcc", line 13, in <module>
    3.     emcc.run()
    4.   File "C:\Program Files\Unity-2017.2.1f1\Editor\Data\PlaybackEngines\WebGLSupport\BuildTools\Emscripten\emcc.py", line 1670, in run
    5.     final = shared.Building.emscripten(final, append_ext=False, extra_args=extra_args)
    6.   File "C:\Program Files\Unity-2017.2.1f1\Editor\Data\PlaybackEngines\WebGLSupport\BuildTools\Emscripten\tools\shared.py", line 1745, in emscripten
    7.     call_emscripten(cmdline)
    8.   File "C:\Program Files\Unity-2017.2.1f1\Editor\Data\PlaybackEngines\WebGLSupport\BuildTools\Emscripten\emscripten.py", line 1842, in _main
    9.     temp_files.run_and_clean(lambda: main(
    10.   File "C:\Program Files\Unity-2017.2.1f1\Editor\Data\PlaybackEngines\WebGLSupport\BuildTools\Emscripten\tools\tempfiles.py", line 78, in run_and_clean
    11.     return func()
    12.   File "C:\Program Files\Unity-2017.2.1f1\Editor\Data\PlaybackEngines\WebGLSupport\BuildTools\Emscripten\emscripten.py", line 1847, in <lambda>
    13.     DEBUG=DEBUG,
    14.   File "C:\Program Files\Unity-2017.2.1f1\Editor\Data\PlaybackEngines\WebGLSupport\BuildTools\Emscripten\emscripten.py", line 1748, in main
    15.     temp_files=temp_files, DEBUG=DEBUG)
    16.   File "C:\Program Files\Unity-2017.2.1f1\Editor\Data\PlaybackEngines\WebGLSupport\BuildTools\Emscripten\emscripten.py", line 93, in emscript
    17.     glue, forwarded_data = compiler_glue(metadata, settings, libraries, compiler_engine, temp_files, DEBUG)
    18.   File "C:\Program Files\Unity-2017.2.1f1\Editor\Data\PlaybackEngines\WebGLSupport\BuildTools\Emscripten\emscripten.py", line 296, in compiler_glue
    19.     cwd=path_from_root('src'), error_limit=300)
    20.   File "C:\Program Files\Unity-2017.2.1f1\Editor\Data\PlaybackEngines\WebGLSupport\BuildTools\Emscripten\tools\jsrun.py", line 122, in run_js
    21.     raise Exception('Expected the command ' + str(command) + ' to finish with return code ' + str(assert_returncode) + ', but it returned with code ' + str(proc.returncode) + ' instead! Output: ' + str(ret)[:error_limit])
    22. Exception: Expected the command ['C:/Program Files/Unity-2017.2.1f1/Editor/Data\\Tools\\nodejs\\node.exe', '--stack_size=512', '--max-old-space-size=1024', 'C:\\Program Files\\Unity-2017.2.1f1\\Editor\\Data\\PlaybackEngines\\WebGLSupport\\BuildTools\\Emscripten\\src\\compiler.js', 'C:\\unity\\unified\\unified-new\\ipc-unity\\unity\\Temp\\EmscriptenTemp\\tmpa1hevu.txt', 'C:\\Program Files\\Unity-2017.2.1f1\\Editor\\Data\\PlaybackEngines\\WebGLSupport\\BuildTools\\lib\\Audio.js', 'C:\\Program Files\\Unity-2017.2.1f1\\Editor\\Data\\PlaybackEngines\\WebGLSupport\\BuildTools\\lib\\Cursor.js', 'C:\\Program Files\\Unity-2017.2.1f1\\Editor\\Data\\PlaybackEngines\\WebGLSupport\\BuildTools\\lib\\Eval.js', 'C:\\Program Files\\Unity-2017.2.1f1\\Editor\\Data\\PlaybackEngines\\WebGLSupport\\BuildTools\\lib\\FileSystem.js', 'C:\\Program Files\\Unity-2017.2.1f1\\Editor\\Data\\PlaybackEngines\\WebGLSupport\\BuildTools\\lib\\Logging.js', 'C:\\Program Files\\Unity-2017.2.1f1\\Editor\\Data\\PlaybackEngines\\WebGLSupport\\BuildTools\\lib\\Profiler.js', 'C:\\Program Files\\Unity-2017.2.1f1\\Editor\\Data\\PlaybackEngines\\WebGLSupport\\BuildTools\\lib\\SystemInfo.js', 'C:\\Program Files\\Unity-2017.2.1f1\\Editor\\Data\\PlaybackEngines\\WebGLSupport\\BuildTools\\lib\\UnetWebSocket.js', 'C:\\Program Files\\Unity-2017.2.1f1\\Editor\\Data\\PlaybackEngines\\WebGLSupport\\BuildTools\\lib\\Video.js', 'C:\\Program Files\\Unity-2017.2.1f1\\Editor\\Data\\PlaybackEngines\\WebGLSupport\\BuildTools\\lib\\WebCam.js', 'C:\\Program Files\\Unity-2017.2.1f1\\Editor\\Data\\PlaybackEngines\\WebGLSupport\\BuildTools\\lib\\WebGL.js', 'C:\\Program Files\\Unity-2017.2.1f1\\Editor\\Data\\PlaybackEngines\\WebGLSupport\\BuildTools\\lib\\WebRequest.js', 'C:\\unity\\unified\\unified-new\\ipc-unity\\unity\\Assets\\Plugins\\WebGL\\GameAnalytics.jslib', 'C:\\unity\\unified\\unified-new\\ipc-unity\\unity\\Assets\\Plugins\\WebGL\\GameAnalyticsUnity.jslib', 'C:\\Program Files\\Unity-2017.2.1f1\\Editor\\Data\\PlaybackEngines\\WebGLSupport\\BuildTools\\Emscripten\\src\\library_pthread_stub.js'] to finish with return code 0, but it returned with code 1 instead! Output: // The Module object: Our interface to the outside world. We import
    23. // and export values on it, and do the work to get that through
    24. // closure compiler if necessary. There are various ways Module can be used:
    25. // 1. Not defined. We create it here
    26. // 2. A function parameter, function(Module) { ..gener
    27. UnityEngine.GUIUtility:ProcessEvent(Int32, IntPtr)

    Any ideas on how I can proceed to determine what's failing?

    I attempted to run that node.exe line manually to see if I could get more error output except I was unable to successfully run it. I expect my environment requires some attention to build from the CLI..

    Tried Unity 2017.2.0p3/p4, 2017.2.1f1 and 5.6.2p2

    Ideas?
     
    Last edited: Dec 19, 2017
  18. Marco-Trivellato

    Marco-Trivellato

    Unity Technologies

    Joined:
    Jul 9, 2013
    Posts:
    1,654
    I am not sure why that takes so long. If you submit a bug report, we will take a look at your project.
     
  19. larku

    larku

    Joined:
    Mar 14, 2013
    Posts:
    1,421
    Unfortunately I'm not able to submit our project.

    Any chance you could give some hints on how I can debug this somewhat on my end? First step for me would be knowing how I could invoke that node.exe (compiler.js) command from the CLI with and environment configured as the Unity build env would be (are there some sh/bat files I can source or similar?) This way I could observe why it fails?

    Note I've also successfully built this same cpp code using (on a different linux box - main project is using Unity on Windows 7):

    emcc -O3 -g0 -DUNITY_WEBGL=1 -s PRECISE_F32=2 -s NO_EXIT_RUNTIME=1 -s USE_WEBGL2=1 -s DISABLE_EXCEPTION_CATCHING=0 -s TOTAL_MEMORY=268435456 --memory-init-file 1 --emit-symbol-map --separate-asm --output_eol linux -o "custom-lib-webgl.bc" ${LOCAL_SRC_FILES}

    this runs fine and generates a seemingly valid custom-lib-webgl.bc, which if added to the project instead of the cpp source will yield the same result (as far as I can tell).

    I'm pretty sure the compilation node.exe step fails instantly. If I watch the process monitor once up to the IL2CPP stage, I see:

    il2cpp.exe eating CPU, then
    clang++.exe eating CPU, then
    opt.exe, then
    llc.exe,

    then node.exe uses CPU for about 2 seconds (using the highly accurate task manager to guess this).
    Then node.exe goes idle, whole machine is now idle, no activity.

    Killing node.exe at this point unfreezes Unity and emits the log listed above. Waiting for hours shows no sign of change, no CPU use, etc..

    I seems that the node.exe step completes very quickly but does not exit (maybe).

    Edit: removed some of post here - was incorrect.
     
    Last edited: Dec 19, 2017
  20. larku

    larku

    Joined:
    Mar 14, 2013
    Posts:
    1,421
    So I managed to get the node.exe line to run. I copied the .txt config JSON file from the temp area during compilation and put it into a temp location so I could invoke:

    Code (csharp):
    1. "C:/Program Files/Unity-2017.2.1f1/Editor/Data\\Tools\\nodejs\\node.exe" --stack_size=1024 --max-old-space-size=2048 "C:\\Program Files\\Unity-2017.2.1f1\\Editor\\Data\\PlaybackEngines\\WebGLSupport\\BuildTools\\Emscripten\\src\\compiler.js" "C:\\unity\\unified\\tmpq5zhgu.txt" "C:\\Program Files\\Unity-2017.2.1f1\\Editor\\Data\\PlaybackEngines\\WebGLSupport\\BuildTools\\lib\\Audio.js" "C:\\Program Files\\Unity-2017.2.1f1\\Editor\\Data\\PlaybackEngines\\WebGLSupport\\BuildTools\\lib\\Cursor.js" "C:\\Program Files\\Unity-2017.2.1f1\\Editor\\Data\\PlaybackEngines\\WebGLSupport\\BuildTools\\lib\\Eval.js" "C:\\Program Files\\Unity-2017.2.1f1\\Editor\\Data\\PlaybackEngines\\WebGLSupport\\BuildTools\\lib\\FileSystem.js" "C:\\Program Files\\Unity-2017.2.1f1\\Editor\\Data\\PlaybackEngines\\WebGLSupport\\BuildTools\\lib\\Logging.js" "C:\\Program Files\\Unity-2017.2.1f1\\Editor\\Data\\PlaybackEngines\\WebGLSupport\\BuildTools\\lib\\Profiler.js" "C:\\Program Files\\Unity-2017.2.1f1\\Editor\\Data\\PlaybackEngines\\WebGLSupport\\BuildTools\\lib\\SystemInfo.js" "C:\\Program Files\\Unity-2017.2.1f1\\Editor\\Data\\PlaybackEngines\\WebGLSupport\\BuildTools\\lib\\UnetWebSocket.js" "C:\\Program Files\\Unity-2017.2.1f1\\Editor\\Data\\PlaybackEngines\\WebGLSupport\\BuildTools\\lib\\Video.js" "C:\\Program Files\\Unity-2017.2.1f1\\Editor\\Data\\PlaybackEngines\\WebGLSupport\\BuildTools\\lib\\WebCam.js" "C:\\Program Files\\Unity-2017.2.1f1\\Editor\\Data\\PlaybackEngines\\WebGLSupport\\BuildTools\\lib\\WebGL.js" "C:\\Program Files\\Unity-2017.2.1f1\\Editor\\Data\\PlaybackEngines\\WebGLSupport\\BuildTools\\lib\\WebRequest.js" "C:\\unity\\unified\\unified-new\\ipc-unity\\unity\\Assets\\Plugins\\WebGL\\GameAnalytics.jslib" "C:\\unity\\unified\\unified-new\\ipc-unity\\unity\\Assets\\Plugins\\WebGL\\GameAnalyticsUnity.jslib" "C:\\Program Files\\Unity-2017.2.1f1\\Editor\\Data\\PlaybackEngines\\WebGLSupport\\BuildTools\\Emscripten\\src\\library_pthread_stub.js"
    Which emitted a heap of js code (and took about 5 seconds to execute) that was perhaps the UnityLoader js code (??).

    Is that what you'd expect? So perhaps it's also working when executed during the build process, but the process doesn't exit, then I kill it and Unity gets a bad exit code.
     
    Last edited: Dec 19, 2017
  21. larku

    larku

    Joined:
    Mar 14, 2013
    Posts:
    1,421
    Actually, when I execute that command, the node.exe process does not exit. It outputs all the .JS code (which if redirected to a file and inspected in an editor, looks well formed and without error) and requires a ctrl-c to kill it. It does not appear to be accepting input from stdin - at least nothing is echo'd to the terminal if I type - only ctrl-c to kill the process seems to have any effect (unless there's some escape key I can hit to get more info?)...

    This is the same behaviour I'm seeing during the build.

    Any idea what's going on here? What could stop that process from exiting (or is that expected behaviour?)?
     
  22. larku

    larku

    Joined:
    Mar 14, 2013
    Posts:
    1,421
    I think I've fix this issue.

    In Unity's compile.js, it appears node.exe is not exiting after successfully running. Perhaps there is still some processing happening on the event loop.

    Anyway, the original main block in compile.js looks like so:

    Code (JavaScript):
    1.  
    2. //===============================
    3. // Main
    4. //===============================
    5. B = new Benchmarker();
    6.  
    7. try
    8. {
    9.     var dummyData = {
    10.         functionStubs: []
    11.     }
    12.     JSify(dummyData);
    13.  
    14.     //dumpInterProf();
    15.     //printErr('paths (fast, slow): ' + [fastPaths, slowPaths]);
    16.     B.print('glue');
    17.  
    18.     if (DEBUG_MEMORY)
    19.     {
    20.         print('zzz. last gc: ' + gc());
    21.         MemoryDebugger.dump();
    22.         print('zzz. hanging now!');
    23.         while (1)
    24.         {};
    25.     }
    26. }
    27. catch (err)
    28. {
    29.     if (err.toString().indexOf('Aborting compilation due to previous errors') != -1)
    30.     {
    31.         // Compiler failed on user error, print out the error message.
    32.         printErr(err + ' | ' + err.stack);
    33.     }
    34.     else
    35.     {
    36.         // Compiler failed on internal compiler error!
    37.         printErr('Internal compiler error in src/compiler.js! Please raise a bug report at https://github.com/kripken/emscripten/issues/ with a log of the build and the input files used to run. Exception message: "' + err + '" | ' + err.stack);
    38.     }
    39.  
    40.     if (ENVIRONMENT_IS_NODE)
    41.     {
    42.         // Work around a node.js bug where stdout buffer is not flushed at process exit:
    43.         // Instead of process.exit() directly, wait for stdout flush event.
    44.         // See https://github.com/joyent/node/issues/1669 and https://github.com/kripken/emscripten/issues/2582
    45.         // Workaround is based on https://github.com/RReverser/acorn/commit/50ab143cecc9ed71a2d66f78b4aec3bb2e9844f6
    46.         process['stdout']['once']('drain', function()
    47.         {
    48.             process['exit'](1);
    49.         });
    50.         console.log(' '); // Make sure to print something to force the drain event to occur, in case the stdout buffer was empty.
    51.         // Work around another node bug where sometimes 'drain' is never fired - make another effort
    52.         // to emit the exit status, after a significant delay (if node hasn't fired drain by then, give up)
    53.         setTimeout(function()
    54.         {
    55.             process['exit'](1);
    56.         }, 500);
    57.     }
    58.     else throw err;
    59. }
    60.  
    I've moved the process exit flush workaround (ENVIRONMENT_IS_NODE if block) to be outside the catch block and to exit with the appropriate value. Like so:

    Code (JavaScript):
    1.  
    2. //===============================
    3. // Main
    4. //===============================
    5.  
    6. B = new Benchmarker();
    7.  
    8. var error_code = 1;
    9.  
    10. try
    11. {
    12.   var dummyData = {functionStubs: []}
    13.   JSify(dummyData);
    14.   error_code = 0;
    15.   //dumpInterProf();
    16.   //printErr('paths (fast, slow): ' + [fastPaths, slowPaths]);
    17.   B.print('glue');
    18.  
    19.   if (DEBUG_MEMORY) {
    20.     print('zzz. last gc: ' + gc());
    21.     MemoryDebugger.dump();
    22.     print('zzz. hanging now!');
    23.     while(1){};
    24.   }
    25. }
    26. catch(err)
    27. {
    28.   error_code = 1;
    29.   if (err.toString().indexOf('Aborting compilation due to previous errors') != -1)
    30.   {
    31.     // Compiler failed on user error, print out the error message.
    32.     printErr(err + ' | ' + err.stack);
    33.   } else
    34.   {
    35.     // Compiler failed on internal compiler error!
    36.     printErr('Internal compiler error in src/compiler.js! Please raise a bug report at https://github.com/kripken/emscripten/issues/ with a log of the build and the input files used to run. Exception message: "' + err + '" | ' + err.stack);
    37.   }
    38. }
    39.  
    40.  
    41. if (ENVIRONMENT_IS_NODE)
    42. {
    43.     // Work around a node.js bug where stdout buffer is not flushed at process exit:
    44.     // Instead of process.exit() directly, wait for stdout flush event.
    45.     // See https://github.com/joyent/node/issues/1669 and https://github.com/kripken/emscripten/issues/2582
    46.     // Workaround is based on https://github.com/RReverser/acorn/commit/50ab143cecc9ed71a2d66f78b4aec3bb2e9844f6
    47.     process['stdout']['once']('drain', function () {
    48.       process['exit'](error_code);
    49.     });
    50.     console.log(' '); // Make sure to print something to force the drain event to occur, in case the stdout buffer was empty.
    51.     // Work around another node bug where sometimes 'drain' is never fired - make another effort
    52.     // to emit the exit status, after a significant delay (if node hasn't fired drain by then, give up)
    53.     setTimeout(function() {
    54.       process['exit'](error_code);
    55.     }, 500);
    56. }
    57.  
    This now proceeds past the Compile asm.js module step. But now fails with:

    Code (csharp):
    1.  
    2. C:\Program Files\Unity-2017.2.1f1\Editor\Data\PlaybackEngines\WebGLSupport\BuildTools\Emscripten\tools\eliminator\node_modules\uglify-js\lib\parse-js.js:272
    3.         throw new JS_Parse_Error(message, line, col, pos);
    4.         ^
    5. TypeError: (intermediate value) is not a function
    6.     at new JS_Parse_Error (C:\Program Files\Unity-2017.2.1f1\Editor\Data\PlaybackEngines\WebGLSupport\BuildTools\Emscripten\tools\eliminator\node_modules\uglify-js\lib\parse-js.js:261:21)
    7.     at js_error (C:\Program Files\Unity-2017.2.1f1\Editor\Data\PlaybackEngines\WebGLSupport\BuildTools\Emscripten\tools\eliminator\node_modules\uglify-js\lib\parse-js.js:272:15)
    8.     at croak (C:\Program Files\Unity-2017.2.1f1\Editor\Data\PlaybackEngines\WebGLSupport\BuildTools\Emscripten\tools\eliminator\node_modules\uglify-js\lib\parse-js.js:742:17)
    9.     at token_error (C:\Program Files\Unity-2017.2.1f1\Editor\Data\PlaybackEngines\WebGLSupport\BuildTools\Emscripten\tools\eliminator\node_modules\uglify-js\lib\parse-js.js:749:17)
    10.     at unexpected (C:\Program Files\Unity-2017.2.1f1\Editor\Data\PlaybackEngines\WebGLSupport\BuildTools\Emscripten\tools\eliminator\node_modules\uglify-js\lib\parse-js.js:755:17)
    11.     at Object.semicolon (C:\Program Files\Unity-2017.2.1f1\Editor\Data\PlaybackEngines\WebGLSupport\BuildTools\Emscripten\tools\eliminator\node_modules\uglify-js\lib\parse-js.js:775:51)
    12.     at prog1 (C:\Program Files\Unity-2017.2.1f1\Editor\Data\PlaybackEngines\WebGLSupport\BuildTools\Emscripten\tools\eliminator\node_modules\uglify-js\lib\parse-js.js:1314:29)
    13.     at simple_statement (C:\Program Files\Unity-2017.2.1f1\Editor\Data\PlaybackEngines\WebGLSupport\BuildTools\Emscripten\tools\eliminator\node_modules\uglify-js\lib\parse-js.js:906:35)
    14.     at C:\Program Files\Unity-2017.2.1f1\Editor\Data\PlaybackEngines\WebGLSupport\BuildTools\Emscripten\tools\eliminator\node_modules\uglify-js\lib\parse-js.js:816:35
    15.     at C:\Program Files\Unity-2017.2.1f1\Editor\Data\PlaybackEngines\WebGLSupport\BuildTools\Emscripten\tools\eliminator\node_modules\uglify-js\lib\parse-js.js:1297:32
    16. wrote symbol map file to C:\unity\unified\unified-new\ipc-unity\unity\Temp\StagingArea\Data\linkresult_asm\build.js.symbols
    17. Traceback (most recent call last):
    18.   File "C:\Program Files\Unity-2017.2.1f1\Editor\Data\PlaybackEngines\WebGLSupport\BuildTools\Emscripten\emcc", line 13, in <module>
    19.     emcc.run()
    20.   File "C:\Program Files\Unity-2017.2.1f1\Editor\Data\PlaybackEngines\WebGLSupport\BuildTools\Emscripten\emcc.py", line 1972, in run
    21.     JSOptimizer.flush()
    22.   File "C:\Program Files\Unity-2017.2.1f1\Editor\Data\PlaybackEngines\WebGLSupport\BuildTools\Emscripten\emcc.py", line 1868, in flush
    23.     run_passes(chunks[0], title, just_split=False, just_concat=False)
    24.   File "C:\Program Files\Unity-2017.2.1f1\Editor\Data\PlaybackEngines\WebGLSupport\BuildTools\Emscripten\emcc.py", line 1841, in run_passes
    25.     final = shared.Building.js_optimizer(final, passes, debug_level >= 4, JSOptimizer.extra_info, just_split=just_split, just_concat=just_concat)
    26.   File "C:\Program Files\Unity-2017.2.1f1\Editor\Data\PlaybackEngines\WebGLSupport\BuildTools\Emscripten\tools\shared.py", line 1828, in js_optimizer
    27.     ret = js_optimizer.run(filename, passes, NODE_JS, debug, extra_info, just_split, just_concat)
    28.   File "C:\Program Files\Unity-2017.2.1f1\Editor\Data\PlaybackEngines\WebGLSupport\BuildTools\Emscripten\tools\js_optimizer.py", line 559, in run
    29.     return temp_files.run_and_clean(lambda: run_on_js(filename, passes, js_engine, source_map, extra_info, just_split, just_concat))
    30.   File "C:\Program Files\Unity-2017.2.1f1\Editor\Data\PlaybackEngines\WebGLSupport\BuildTools\Emscripten\tools\tempfiles.py", line 78, in run_and_clean
    31.     return func()
    32.   File "C:\Program Files\Unity-2017.2.1f1\Editor\Data\PlaybackEngines\WebGLSupport\BuildTools\Emscripten\tools\js_optimizer.py", line 559, in <lambda>
    33.     return temp_files.run_and_clean(lambda: run_on_js(filename, passes, js_engine, source_map, extra_info, just_split, just_concat))
    34.   File "C:\Program Files\Unity-2017.2.1f1\Editor\Data\PlaybackEngines\WebGLSupport\BuildTools\Emscripten\tools\js_optimizer.py", line 501, in run_on_js
    35.     assert proc.returncode == 0
    36. AssertionError
    37. UnityEngine.GUIUtility:ProcessEvent(Int32, IntPtr)
    38.  
     
  23. larku

    larku

    Joined:
    Mar 14, 2013
    Posts:
    1,421
    My latest findings..

    After getting the compile.js script to exit, the next failure is in seperate_asm.py

    Code (csharp):
    1.  
    2.   File "C:\Program Files\Unity-2017.2.1f1\Editor\Data\PlaybackEngines\WebGLSupport\BuildTools\Emscripten\tools\separate_asm.py", line 26, in <module>
    3.     assert m, 'cannot figure out the closured name of Module statically'
    4. AssertionError: cannot figure out the closured name of Module statically
    5. wrote symbol map file to C:\unity\unified\unified-new\ic-unity\unity\Temp\StagingArea\Data\linkresult_asm\build.js.symbols
    6. next value is: (C:\unity\unified\unified-new\ic-unity\unity\Temp\EmscriptenTemp\tmp2xgvnj.cl.js : C:\unity\unified\unified-new\ic-unity\unity\Temp\EmscriptenTemp\tmp2xgvnj.cl.js)
    7. Traceback (most recent call last):
    8.   File "C:\Program Files\Unity-2017.2.1f1\Editor\Data\PlaybackEngines\WebGLSupport\BuildTools\Emscripten\emcc", line 13, in <module>
    9.     emcc.run()
    10.   File "C:\Program Files\Unity-2017.2.1f1\Editor\Data\PlaybackEngines\WebGLSupport\BuildTools\Emscripten\emcc.py", line 2079, in run
    11.     subprocess.check_call([shared.PYTHON, shared.path_from_root('tools', 'separate_asm.py'), js_target, asm_target, js_target])
    12.   File "C:\Program Files\Unity-2017.2.1f1\Editor\Data\PlaybackEngines\WebGLSupport\BuildTools\Emscripten_Win\python\2.7.5.3_64bit\lib\subprocess.py", line 542, in check_call
    13.     raise CalledProcessError(retcode, cmd)
    14. subprocess.CalledProcessError: Command '['C:\\Program Files\\Unity-2017.2.1f1\\Editor\\Data\\PlaybackEngines\\WebGLSupport\\BuildTools\\Emscripten_Win\\python\\2.7.5.3_64bit\\python.exe', 'C:\\Program Files\\Unity-2017.2.1f1\\Editor\\Data\\PlaybackEngines\\WebGLSupport\\BuildTools\\Emscripten\\tools\\separate_asm.py', 'C:\\unity\\unified\\unified-new\\ic-unity\\unity\\Temp\\StagingArea\\Data\\linkresult_asm\\build.js', 'C:\\unity\\unified\\unified-new\\ic-unity\\unity\\Temp\\StagingArea\\Data\\linkresult_asm\\build.asm.js', 'C:\\unity\\unified\\unified-new\\ic-unity\\unity\\Temp\\StagingArea\\Data\\linkresult_asm\\build.js']' returned non-zero exit status 1
    15. UnityEngine.GUIUtility:ProcessEvent(Int32, IntPtr)
    16.  
    So this script is looking for either 'var Module' or 'ENVIRONMENT' in the file C:\unity\unified\unified-new\ic-unity\unity\Temp\StagingArea\Data\linkresult_asm\build.js

    Inspecting the content of this file suggests that it's already been minified or similar and neither such string exist.

    Any ideas?
     
  24. bhupiister

    bhupiister

    Joined:
    Dec 13, 2019
    Posts:
    33
    @Marco-Trivellato If I want to build a webassembly separately with new version of emscripten and include it in unity webgl build, will it work? I am asking because the version unity uses internally is too old and not easily available from emscripten and moreover it has some issues related to python.

    And if I build webassembly, where should I include .webasm file? Plugin folder? and use DllImport("_Internal") on it ?
     
unityunity