Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Question Standalone Holographic Remoting to Hololens 2 with XR SDK Package

Discussion in 'AR' started by Ozelotl, Oct 12, 2020.

  1. Ozelotl

    Ozelotl

    Joined:
    Sep 26, 2017
    Posts:
    11
    Hello,

    I want to create a standalone UWP application that can connect to the Hololens 2 Holographic Remoting Player.

    I am using Unity 2019.4.5f1, XR Plugin Management Package 3.2.16, Windows XR Plugin 2.4.1 and Holographic Remoting Player v2.2.3.0.

    I can use the Window>XR>Windows XR Plugin Remoting functionality as intended (connect, then press play, then see the hologram on my Hololens). I can also execute the code from the scripting api or the same call the Windows XR Plugin Remoting window uses. When connecting like this before entering Playmode, everything works as if I had used the Windows XR Plugin Remoting window. When executing the code during Playmode, a connection is established but the screen in the Hololens stays black.

    Additionally, I have found no way to make even the black screen connection work in a standalone UWP desktop application.
    If I check "Windows Mixed Reality" in the XR Plug-in Management settings for UWP, my built application will ask me to connect a MR device when starting.
    If I uncheck it, I can start the application but will get an error upon trying to execute the connection code, as XR dlls are not loaded.

    Unfortunately, I can only find tutorials for getting Holographic Remoting to work with Legacy XR (by including the "none" sdk and then setting "XRSettings.enabled = true;"). I would love to be able to use Holographic Remoting in a standalone application with the new XR Pipeline, or at least know if this is currently not possible.

    Thanks in advance for your help and best regards,
    Stella
     
  2. JimboJonesBL

    JimboJonesBL

    Joined:
    Jul 16, 2020
    Posts:
    19
    Would love to know the exact same thing. I'm using Unity 2019.4.13, Windows XR Plugin 2.4.1, XR Plugin Management 3.2.17, Holographic Remoting Player v24.0.0. Holographic Remoting via Editor Playmode works flawless. But when building a standalone UWP application, the application asks to connect a MR device just like @Ozelotl .

    Settings in Unity:
    Edit->Project Settings-> XR Plug-in Management
    PC, Standalone:
    - Enable Initialize XR on Startup
    - Enable Windows Mixed Reality
    UWP:
    - Enable Initialize XR on Startup
    - Enable Windows Mixed Reality

    Edit->Project Settings-> XR Plug-in Management->Windows Mixed Reality
    PC, Standalone:
    -Enable Shared Depth Buffer (24bit)
    UWP:
    -Enable Holographic Remoting
    -Enable Shared Depth Buffer (24bit)

    Edit->Project Settings ->Player -> Publishing Settings under Capabilities additionally
    - Enable InternetClientServer
    - Enable PrivateNetworkClientServer

    Add the holographic Remoting prefab from the MRTK.Tutorials.PCHolographicRemoting.unitypackage from https://docs.microsoft.com/de-de/wi...orials/mr-learning-pc-holographic-remoting-01

    Settings in Visual Studio:
    -Release
    -x64
    -LocalMachine

    When hitting LocalMachine build while having the Holographic Remoting Player opened, following Error appears:
    "Cant open app
    To use this app, youll need to set up Mixed Reality. Go to Start > Mixed Reality Portal or plug in a compatible headset to get started." I don't even get to the point where I can enter an IP.
    HLRemotingError.png

    PS: I can access Windows Device Portal via WiFi

    Is it possible to have a standalone application with the XRSDK setup? I would really appreciate an answer, as there are no real tutorials or information on this topic.
     
    AndSet likes this.
  3. joejo

    joejo

    Unity Technologies

    Joined:
    May 26, 2016
    Posts:
    958
    >Would love to know the exact same thing. I'm using Unity 2019.4.13, Windows XR Plugin 2.4.1, XR Plugin Management 3.2.17,

    I know it's tangential to the root issue, but 3.2.17 should be used with 2019.4.15f1 or later. XR Plug-In Management 3.2.17 - Unity Forum
     
  4. JasonCostanza

    JasonCostanza

    Unity Technologies

    Joined:
    May 23, 2017
    Posts:
    404
    Hello there,

    Can you try changing your settings to match the following and let me know if it works better, differently, worse?
    upload_2020-12-17_10-11-46.png

    upload_2020-12-17_10-12-1.png

    The reason it's hitting the Headset plug in step is because "Initialize on startup" may be active, if I'm following your descriptions correctly. That will cause the app to look for any valid VR display at launch, but we're not ready to give it one yet so we need to disable that. The loader MUST be enabled because otherwise our remoting API's won't be accessible, as Ozelotl mentioned seeing in their post about the dlls not loaded.

    If you can try the above and let me know if you're still hitting issues, that would be fantastic. I did scour our bug database and found a matching bug, but it has yet to be verified by our internal QA team and due to COVID-19, some of our bug verification times have been longer than usual as we shift bugs and resources around local government lockdowns and such.
     
  5. JimboJonesBL

    JimboJonesBL

    Joined:
    Jul 16, 2020
    Posts:
    19
    Hey guys,

    thank you very much for your fast answers!

    @joejo Stupid me can't read version names. I'm sorry, I'm running XR Plugin Management 3.2.17 with Unity 2019.4.16f1. I double checked the other versions, these are correct. Thank you for pointing out!

    @JasonCostanza Okay, I see progress! Now I'm getting a game preview window with a Connect button and text field to enter the IP address. Unfortunately, the development console gives me a strange error:
    HLRemotingError1.01.png

    Following this error message I researched two things.
    1. Why is the error for "Hololens '1' remoting plugin"
    - In checked the HolographicRemoting prefab from MRTK.Tutorials.PCHolographicRemoting.unitypackage, that is responsible for the remote connection. It has a script attached which has the following lines for connecting:

    Code (CSharp):
    1.  
    2.     public void Connect()
    3.     {
    4.         if (HolographicRemoting.ConnectionState != HolographicStreamerConnectionState.Connected)
    5.         {
    6.             //HolographicRemoting.Connect(IP); //For HL1
    7.             HolographicRemoting.Connect(IP, 99999, RemoteDeviceVersion.V2);
    8.  
    9.         }
    10.     }
    11.  
    Full code in attachment.

    Peeking the definition of RemoteDeviceVersion...
    Code (CSharp):
    1. namespace UnityEngine.XR.WSA
    2. {
    3.     [NativeHeader("Modules/VR/HoloLens/PerceptionRemoting.h")]
    4.     public enum RemoteDeviceVersion
    5.     {
    6.         V1 = 0,
    7.         V2 = 1
    8.     }
    9. }
    10.    
    ... gives probably the answer to why it says "Hololens '1'"?


    2. "Did you enable remoting in Player Settings"
    - Well, because I'm using the new XR Plugin, the options in the Player Settings are greyed out of course. As stated in my first post Holographic Remoting is enabled in the Mixed Reality settings.

    Am I missing something here?
     

    Attached Files:

  6. JasonCostanza

    JasonCostanza

    Unity Technologies

    Joined:
    May 23, 2017
    Posts:
    404
    Hey @JimboJonesBL

    Sorry just saw this, I apologize!

    The option for Holographic remoting is also in the XR Plugin system. If you look in my screenshots, that setting is found under the Windows Mixed Reality loader settings. Go to Project Settings > (under) XR Management > Windows Mixed Reality. Select that and you will get settings pertaining to the WMR loader specifically, in which Holographic Remoting now lives. If that is disabled, that could be the cause of the error you're getting though I do also not like the text you're receiving. I will ask our engineer if he can track down that error and confirm my thinking or not.

    Note, a lot of our engineers have already taken time off for the holidays so it may take us a while to get a confirmation from them as many are not checking emails or messages at this time.
     

    Attached Files:

  7. JimboJonesBL

    JimboJonesBL

    Joined:
    Jul 16, 2020
    Posts:
    19
    Hey @JasonCostanza,

    citing my first post, I guess this already complies with my settings:
    Edit->Project Settings-> XR Plug-in Management->Windows Mixed Reality
    PC, Standalone:
    -Enable Shared Depth Buffer (24bit)
    UWP:
    -Enable Holographic Remoting
    -Enable Shared Depth Buffer (24bit)

    I really appreciate the effort and am looking forward to receiving some news on this after the Christmas break. Happy holidays to you guys :)
     
  8. JasonCostanza

    JasonCostanza

    Unity Technologies

    Joined:
    May 23, 2017
    Posts:
    404
    Oh I apologize I misread that first post. I just got back from holiday so let me get this back on my radar and I'll have updates for you soon. I hope you had a happy holiday and new year!
     
  9. JasonCostanza

    JasonCostanza

    Unity Technologies

    Joined:
    May 23, 2017
    Posts:
    404
    By soon, turns out that means 20 minutes in this case. Updates!

    @JimboJonesBL can you search in your project in Visual Studio for MicrosoftHolographicAppRemoting.dll. If that file is missing then somehow your plugin did not install the remoting stuff you need and you should purge the package from your project fully, then re-download the WMR package and re-enable that holographic remoting checkbox and try again.
     
  10. JimboJonesBL

    JimboJonesBL

    Joined:
    Jul 16, 2020
    Posts:
    19
    Hey @JasonCostanza,

    happy new year! Thanks for getting back on the topic so fast!

    Searching through the built project, I found a dll file with almost the same name (see below). I assume that's the one?

    HolographicAppRemotingDLL.png
     
  11. JasonCostanza

    JasonCostanza

    Unity Technologies

    Joined:
    May 23, 2017
    Posts:
    404
    Hey @JimboJonesBL Let's get you to file a bug report attaching your project and we will take a look to eliminate the back-and-forth exchange. I'm hoping by looking at your project we can give you exact guidance or realize we broke something and will make a fix for you that you can track using your bug report. How's that sound? Provide the bug number here when you have it and I'll go intercept it.
     
  12. JimboJonesBL

    JimboJonesBL

    Joined:
    Jul 16, 2020
    Posts:
    19
    Hey @JasonCostanza,

    I filed a bug report with the project attached. Hopefully, you can tell me where it's going wrong with this!

    It's Case 1304280
     
  13. JasonCostanza

    JasonCostanza

    Unity Technologies

    Joined:
    May 23, 2017
    Posts:
    404
    Thank you, got the bug. I'll take a look at it real soon and post an update here or email you with any info or questions
     
  14. JasonCostanza

    JasonCostanza

    Unity Technologies

    Joined:
    May 23, 2017
    Posts:
    404
  15. JimboJonesBL

    JimboJonesBL

    Joined:
    Jul 16, 2020
    Posts:
    19
    Hey @JasonCostanza,

    is there any progress to report on this issue yet? Unfortunately, the issue status didn't change since it went live.
     
  16. JasonCostanza

    JasonCostanza

    Unity Technologies

    Joined:
    May 23, 2017
    Posts:
    404
    I looked into the status more and the bug is assigned to our engineer for fixing. I am not sure if they have begun work on investigating it yet but it is on their plate.
     
    JimboJonesBL likes this.
  17. keveleigh

    keveleigh

    Official Microsoft Employee

    Joined:
    Jun 19, 2015
    Posts:
    35
  18. JimboJonesBL

    JimboJonesBL

    Joined:
    Jul 16, 2020
    Posts:
    19
    Hey @keveleigh,

    thanks for jumping on this topic.

    I adjusted the connection script with the WindowsMRRemoting class, see attachment. The error from above disappears, unfortunately, another weird error appears:
    HL_OculusError.png
    Checking the log:
    Begin showing splash screen.
    The referenced script (Microsoft.MixedReality.Toolkit.XRSDK.Oculus.Input.OculusXRSDKDeviceManagerProfile) on this Behaviour is missing!
    (Filename: C:\buildslave\unity\build\Runtime/Scripting/ManagedReference/SerializableManagedRef.cpp Line: 199)

    The referenced script on this Behaviour (Game Object '<null>') is missing!
    (Filename: C:\buildslave\unity\build\Runtime/Mono/ManagedMonoBehaviourRef.cpp Line: 334)

    A scripted object (probably Microsoft.MixedReality.Toolkit.XRSDK.Oculus.Input.OculusXRSDKDeviceManagerProfile?) has a different serialization layout when loading. (Read 64 bytes but expected 148 bytes)
    Did you #ifdef UNITY_EDITOR a section of your serialized properties in any of your scripts?
    (Filename: C:\buildslave\unity\build\Runtime/Serialize/SerializedFile.cpp Line: 2324)


    This error did not halt the application though. I still could enter an IP address and click connect. WindowsMRRemoting.isConnected also returns true, but nothing is displayed on the Hololens2.

    Starting the application in Debug mode, I get some exceptions, but they don't give me a direct hint what's going on (Full debug log in attachment). The log also shows, that the Connect() call returns false?!

    Unfortunately, I won't be able to debug this project any longer in depth. But if there is only an error in the connection script I can still check if a solution works.

    Thank you for your help!
     

    Attached Files:

  19. manurocker95

    manurocker95

    Joined:
    Jun 14, 2016
    Posts:
    209
    Sorry for reviving an old thread but does anyone know why HolographicRemoting doesn't exist under UnityEngine.XR, UnityEngine.XR.WindowsMR, UnityEngine.XR.WSA nor UnityEngine.XR.OpenXR? I'm using OpenXR but can't connect to the hololense2 in standalone...
     
  20. JimboJonesBL

    JimboJonesBL

    Joined:
    Jul 16, 2020
    Posts:
    19
  21. manurocker95

    manurocker95

    Joined:
    Jun 14, 2016
    Posts:
    209
    Yeah, it's really confusing because I ended get it working with:

    StartCoroutine(LoadDevice("WindowsMR"));

    IEnumerator LoadDevice(string newDevice)
    {
    XRSettings.LoadDeviceByName(newDevice);
    yield return null;
    XRSettings.enabled = true;
    yield return AppRemoting.Connect(m_remotingConfiguration);
    }

    but there's almost no documentation about this...
     
    JasonCostanza likes this.
  22. JimboJonesBL

    JimboJonesBL

    Joined:
    Jul 16, 2020
    Posts:
    19
    Totally agree! I had the same issue back then, that no further documentation was available...
    I'm glad you got it working :)
     
    manurocker95 and JasonCostanza like this.
  23. JasonCostanza

    JasonCostanza

    Unity Technologies

    Joined:
    May 23, 2017
    Posts:
    404
    Hi there,

    We'll be taking a look into this and hopefully have some more information soon. I have just submitted this to my team to review and see if we can help improve documentation or offer a script suggestion.
     
    JimboJonesBL likes this.
  24. manurocker95

    manurocker95

    Joined:
    Jun 14, 2016
    Posts:
    209
    It's weird to see how "difficult" is to get this working when it should be completely plug and play like VR stuff. The documentation seems outdated as UnityEngine.XR.WSA.HolographicRemoting is no longer working nor supported, so it would be nice to have some template or documentation to take a look ^^.
     
  25. Tanya_Li

    Tanya_Li

    Unity Technologies

    Joined:
    Jun 29, 2020
    Posts:
    104
  26. manurocker95

    manurocker95

    Joined:
    Jun 14, 2016
    Posts:
    209
    Sorry for the bump after a year. I'm facing a new issue that maybe you could help me with:

    - I would like to disable Mixed reality's OpenXR features and start them in runtime but if I don't include Holographic remoting remote app feature group, the assembly is not even added to build and setting to "enabled = false" every feature in UnityEngine.XR.OpenXR.OpenXRSettings.Instance.GetFeatures(); doesn't seem to actually disable the feature, as AppRemotingPlugin feature is instanced when I load OpenXR (manually). I would like to set the features for VR or MR depending on specific configs.

    Specifically, I'm making an app that runs on Desktop pc (Windows standalone and not UWP), that can run VR and MR devices as headsets with openXR. VR headsets work just fine, but Hololense 2 automatically disconnects on build (but not on editor) in the case I add the feature group previous to the build.

    Any idea?