Search Unity

  1. Unity 6 Preview is now available. To find out what's new, have a look at our Unity 6 Preview blog post.
    Dismiss Notice
  2. Unity is excited to announce that we will be collaborating with TheXPlace for a summer game jam from June 13 - June 19. Learn more.
    Dismiss Notice

Low-level plug-in interface - UnityRegisterRenderingPlugin and XR Interfaces

Discussion in 'Web' started by De-Panther, Oct 6, 2020.

  1. De-Panther

    De-Panther

    Joined:
    Dec 27, 2009
    Posts:
    590
    We are working on a WebXR Export package, got the Unity XR SDK, and while implementing it, we noticed that we need a way for Unity to call "UnityPluginLoad" and "UnityPluginUnload" on the native side.

    We found a blog post https://blogs.unity3d.com/2017/01/19/low-level-plugins-in-unity-webgl/
    which talks about "UnityRegisterRenderingPlugin".
    We tried it, and it only implements the "IUnityGraphics", but we need a way to register XR Plugin ("IUnityXRInputInterface" and "IUnityXRDisplayInterface").

    As I understand "UnityRegisterRenderingPlugin" is the only hook.

    Are there other ways to initiate a call to "UnityPluginLoad" with the needed interfaces?

    Is it a bug? feature request? Who should we contact to add a hook for the XR interfaces?

    Thanks

    https://github.com/De-Panther/unity-webxr-export
     
  2. jukka_j

    jukka_j

    Unity Technologies

    Joined:
    May 4, 2018
    Posts:
    953
    Even though the function is name UnityRegisterRenderingPlugin, it is actually used to register any type of plugin in WebGL builds. That is, something like

    Code (CSharp):
    1. #include "IUnityInterface.h"
    2. #include "XR/IUnityXRTrace.h"
    3. #include "XR/UnitySubsystemTypes.h"
    4.  
    5. #include "ProviderContext.h"
    6.  
    7. static ProviderContext* s_Context{};
    8.  
    9. UnitySubsystemErrorCode Load_Display(ProviderContext&);
    10. UnitySubsystemErrorCode Load_Input(ProviderContext&);
    11.  
    12. static bool ReportError(const char* name, UnitySubsystemErrorCode err)
    13. {
    14.     if (err != kUnitySubsystemErrorCodeSuccess)
    15.     {
    16.         XR_TRACE_ERROR(s_Context->trace, "Error loading subsystem: %s (%d)\n", name, err);
    17.         return true;
    18.     }
    19.     return false;
    20. }
    21.  
    22. extern "C" void UNITY_INTERFACE_EXPORT UNITY_INTERFACE_API
    23. UnityPluginLoad(IUnityInterfaces* unityInterfaces)
    24. {
    25.     auto* ctx = s_Context = new ProviderContext;
    26.  
    27.     ctx->interfaces = unityInterfaces;
    28.     ctx->trace = unityInterfaces->Get<IUnityXRTrace>();
    29.  
    30.     if (ReportError("Display", Load_Display(*ctx)))
    31.         return;
    32.  
    33.     if (ReportError("Input", Load_Input(*ctx)))
    34.         return;
    35. }
    36.  
    37. extern "C" void UNITY_INTERFACE_EXPORT UNITY_INTERFACE_API
    38. UnityPluginUnload()
    39. {
    40.     delete s_Context;
    41. }
    Behind the Unity XR SDK Sign Up link you can find the above sample, which adapted to the web would look like

    Code (CSharp):
    1. #include "IUnityInterface.h"
    2. #include "XR/IUnityXRTrace.h"
    3. #include "XR/UnitySubsystemTypes.h"
    4.  
    5. #include "ProviderContext.h"
    6.  
    7. static ProviderContext* s_Context{};
    8.  
    9. UnitySubsystemErrorCode Load_Display(ProviderContext&);
    10. UnitySubsystemErrorCode Load_Input(ProviderContext&);
    11.  
    12. static bool ReportError(const char* name, UnitySubsystemErrorCode err)
    13. {
    14.     if (err != kUnitySubsystemErrorCodeSuccess)
    15.     {
    16.         XR_TRACE_ERROR(s_Context->trace, "Error loading subsystem: %s (%d)\n", name, err);
    17.         return true;
    18.     }
    19.     return false;
    20. }
    21.  
    22. static void // Changed to static to avoid any linkage collision with function name
    23. UnityPluginLoad(IUnityInterfaces* unityInterfaces)
    24. {
    25.     auto* ctx = s_Context = new ProviderContext;
    26.  
    27.     ctx->interfaces = unityInterfaces;
    28.     ctx->trace = unityInterfaces->Get<IUnityXRTrace>();
    29.  
    30.     if (ReportError("Display", Load_Display(*ctx)))
    31.         return;
    32.  
    33.     if (ReportError("Input", Load_Input(*ctx)))
    34.         return;
    35. }
    36.  
    37. static void // Changed to static to avoid any linkage collision with function name
    38. UnityPluginUnload()
    39. {
    40.     delete s_Context;
    41. }
    42.  
    43. #ifdef __EMSCRIPTEN__
    44.  
    45. extern "C" void UnityRegisterRenderingPlugin(PluginLoadFunc loadPlugin, PluginUnloadFunc unloadPlugin);
    46.  
    47. extern "C" void UNITY_INTERFACE_EXPORT UNITY_INTERFACE_API RegisterMyWebXRPlugin()
    48. {
    49.     UnityRegisterRenderingPlugin(UnityPluginLoad, UnityPluginUnload);
    50. }
    51. #endif
    52.  
    53.  
    and then in C# side:

    Code (CSharp):
    1. class MyRenderPlugin
    2. {
    3.     #if UNITY_WEBGL && !UNITY_EDITOR
    4.     [DllImport ("__Internal")]
    5.     private static extern void RegisterMyWebXRPlugin();
    6.     #endif
    7.  
    8.     void Start()
    9.     {
    10.         RegisterMyWebXRPlugin();
    11.     }
    12. }
    Basically the function UnityRegisterRenderingPlugin() is a replacement to the lack of ability to do dlsym() on the web, since everything is statically linked to.
     
  3. De-Panther

    De-Panther

    Joined:
    Dec 27, 2009
    Posts:
    590
    Thanks,

    we tried that in Unity 2019.4.7f1 and it failed.

    if we add to main
    #include "XR/IUnityXRDisplay.h"
    #include "XR/IUnityXRInput.h"

    and after
    ctx->trace = unityInterfaces->Get<IUnityXRTrace>();

    we add
    Code (CSharp):
    1.     s_Graphics = unityInterfaces->Get<IUnityGraphics>();
    2.     printf("Get<IUnityGraphics>\n");
    3.     if (s_Graphics == NULL)
    4.     {
    5.         printf("s_Graphics == NULL\n");
    6.     }
    7.     else
    8.     {
    9.         printf("WE HAVE s_Graphics !!!!\n");
    10.     }
    11.    
    12.     s_XrDisplay = unityInterfaces->Get<IUnityXRDisplayInterface>();
    13.     printf("Get<IUnityXRDisplayInterface>\n");
    14.     if (s_XrDisplay == NULL)
    15.     {
    16.         printf("s_XrDisplay == NULL\n");
    17.     }
    18.     else
    19.     {
    20.         printf("WE HAVE s_XrDisplay !!!!\n");
    21.     }
    22.    
    23.     s_XrInput = unityInterfaces->Get<IUnityXRInputInterface>();
    24.     printf("Get<IUnityXRInputInterface>\n");
    25.     if (s_XrInput == NULL)
    26.     {
    27.         printf("s_XrInput == NULL\n");
    28.     }
    29.     else
    30.     {
    31.         printf("WE HAVE s_XrInput !!!!\n");
    32.     }
    will print:
    WE HAVE s_Graphics !!!!
    s_XrDisplay == NULL
    s_XrInput == NULL
     
  4. jukka_j

    jukka_j

    Unity Technologies

    Joined:
    May 4, 2018
    Posts:
    953
    That does look odd. My understanding is that the code should be able to obtain the interfaces (although I have not done any work with the XR API to say for sure)

    Does the same code work to obtain XR APIs when you build for native Windows? If so, that does suggest a WebGL bug. If not, that might be a general XR issue.
     
  5. De-Panther

    De-Panther

    Joined:
    Dec 27, 2009
    Posts:
    590
    Yes.
    Works on windows and fail on WebGL when using the "UnityRegisterRenderingPlugin".

    Should I report a bug and post here the bug number?
     
  6. De-Panther

    De-Panther

    Joined:
    Dec 27, 2009
    Posts:
    590
    Reported - Case 1283105
     
  7. jukka_j

    jukka_j

    Unity Technologies

    Joined:
    May 4, 2018
    Posts:
    953
    Oh hmm, now thinking back I realize the whole XR API might (currently) not be available on the web. But maybe it should be for the exact purposes of implementing XR plugins like this.

    Thanks for reporting!
     
    xrtk-build-bot and De-Panther like this.
  8. De-Panther

    De-Panther

    Joined:
    Dec 27, 2009
    Posts:
    590
    Thanks :)

    If there's any progress behind the scenes, can you please update about that?

    It's an open source project that we work on in our free time, but it's a few weeks now that we can't progress with the implementations.
     
  9. De-Panther

    De-Panther

    Joined:
    Dec 27, 2009
    Posts:
    590
    Maybe will also help to mention, that there's a file "UnitySubsystemsManifest.json" that ask for "libraryName".
    So I set this to "__Internal", instead of a library file name... like when using DllImport on WebGL.
     
  10. jukka_j

    jukka_j

    Unity Technologies

    Joined:
    May 4, 2018
    Posts:
    953
    I think the updates should come as a CC to your email inbox. I'll try to remember to post here for anything that does not.

    I believe the UnitySubsystemsManifest.json will be just ignored in WebGL builds (although not 100% sure, have not used that before)
     
    De-Panther likes this.
  11. De-Panther

    De-Panther

    Joined:
    Dec 27, 2009
    Posts:
    590
  12. brendanduncan_u3d

    brendanduncan_u3d

    Unity Technologies

    Joined:
    Jul 30, 2019
    Posts:
    461
    The problem is that the XR module subsystems are not available to the WebGL platform. Subsystem requirements are usually declared with UnitySubsystemsManifest.json, but this is not available on WebGL, due to the file system. There will be a lot of work required to get these things to work with the platform limitations WebGL has. Having a WebXR driver for the XR subsystem would be great, I would like to use it, but it will likely be a longer term fix.
     
    fherbst and De-Panther like this.
  13. De-Panther

    De-Panther

    Joined:
    Dec 27, 2009
    Posts:
    590
    Does that mean that it won't be fixed, or just that it'll take more time to fix it?
    Thanks for reviewing the issue.
     
  14. thep3000

    thep3000

    Unity Technologies

    Joined:
    Aug 9, 2013
    Posts:
    400
    FYI this issue has been fixed in 2020.3.6f1+, 2021.1.4f1+ and 2021.2.0a10+.
     
    De-Panther likes this.
  15. De-Panther

    De-Panther

    Joined:
    Dec 27, 2009
    Posts:
    590
    Thanks!
    Also just noticed that I got a reply in the bug tracker with a sample package.
    The sample package didn't work, and the C/C++ code from my bug project seems to have errors now when it compiles with Unity.
    But managed to test the basic use case of getting the XRDisplay and XRInput, and it works!


    I guess that I'll need to compile a wasm bitcode file/lib and use it in the project, instead of raw C/C++ code.
     
  16. De-Panther

    De-Panther

    Joined:
    Dec 27, 2009
    Posts:
    590
    Figured out I can use "-std=c++11" in the emscriptenArgs.
    The sample code works :)
    Thanks!
    upload_2021-7-3_10-58-24.png
     
  17. De-Panther

    De-Panther

    Joined:
    Dec 27, 2009
    Posts:
    590
    2 years later, and I got to work on this again.

    For now I focus on LTS 2020.3.11f1 and LTS 2022.3.10f1.

    What works:
    - Headset/device tracking - works great!
    - Display rendering - Setting the views and render texture.

    What doesn't work on both versions:
    - Single-Pass and Multiview - I'm not expecting Multiview to work, but I was thinking that single pass might work.
    - The RenderTexture of the display is updated only on the first frame, or if there's another enabled camera on scene with Target Eye set to None (Main Display). In all other cases, the RenderTexture stays the same.

    What fails on 2022.3.10f1:
    - WebGL Warnings on XR mode - "INVALID_VALUE: bufferSubData: buffer overflow", "INVALID_OPERATION: drawBuffers: BACK or NONE", "INVALID_ENUM: invalidateFramebuffer: invalid attachment".
    - Once on XR mode, and enabling one more camera for spectator mode (Target Eye set to None), performance drops a lot. 2 calls to "bufferSubData" that takes 15ms each, on PC. Even after the extra camera is disabled, that performance drop keeps.

    upload_2023-12-9_6-10-24.png

    upload_2023-12-9_6-10-42.png


    Other than that, this is AWESOME!
     
    KamilCSPS likes this.
  18. De-Panther

    De-Panther

    Joined:
    Dec 27, 2009
    Posts:
    590
    KamilCSPS likes this.
  19. De-Panther

    De-Panther

    Joined:
    Dec 27, 2009
    Posts:
    590
    KamilCSPS likes this.
  20. De-Panther

    De-Panther

    Joined:
    Dec 27, 2009
    Posts:
    590
    Done some tests with URP.
    There are no issues of display not refreshing/update.
    Frame rate is ok.
    But there are some artifacts in some cases. On AR it's randomly modify objects to triangles, and on VR it shows one color for an eye when looking on a place with only skybox.
    Also linear color space looks dark when entering XR mode.
     
    KamilCSPS likes this.
  21. KamilCSPS

    KamilCSPS

    Joined:
    May 21, 2020
    Posts:
    449
    Superb work!

    Just wanted to say kudos to you for, again, being a pillar of this community.
     
    De-Panther likes this.
  22. De-Panther

    De-Panther

    Joined:
    Dec 27, 2009
    Posts:
    590
    Released version 0.20.0 of WebXR Export with XR SDK support.

    Couldn't fully integrate the display texture. Had to use some hacks.
    The XR SDK generate a texture for the display each frame, and it allows to get the texture and draw it on the display.
    When drawing the texture, each frame I got warnings like:
    - WebGL: INVALID_VALUE: bufferSubData: buffer overflow
    - GL_INVALID_OPERATION: Active draw buffers with missing fragment shader outputs.
    And if I set the XR SDK to invalidateRenderStateAfterEachCallback, each frame I also got error:
    OPENGL NATIVE PLUG-IN ERROR: GL_INVALID_OPERATION: Operation illegal incurrent state

    With those warnings and errors there were also distortions and artifacts in the texture.

    As the XR SDK draws the content of the texture to the WebGL Canvas, I ended up not using the texture.
    Instead, when in XR mode, I override the WebGL bindFramebuffer function to bind to the XRLayer when trying to bind to the canvas.
    This method also force me to use the transparent hack, which ignores the WebGL clear function when colorMask is only alpha false.

    I also had to drop the Built-in Render Pipeline support, as there were some extra issues, but I think that for at least some of them I can use my own hacks to fix. Others depends on internal Unity stuff that I can't patch (the performance drop).
     
    KamilCSPS likes this.