Search Unity

Hololens Remoting Build gestures not recognized

Discussion in 'Windows Mixed Reality' started by MerkSim, Jul 20, 2018.

  1. MerkSim

    MerkSim

    Joined:
    Mar 1, 2018
    Posts:
    12
    Hello,

    I 've been trying to debug this thing for a day and a half and I cant wrap my head around it. It all revolves around the new 2018.2 Remoting feature.
    Here's my process :

    - HolographicRemoting.Connect(IP, bitrate) //connect to the hololens device
    - StartCoroutine(LoadDevice("WindowsMR")); // Loaddevice like the in unity blog
    - recognizer = new GestureRecognizer(); //initialize a gesturerecognizer and subscribe function to tapped
    recognizer.Tapped += (args) => { SendGazePoint(); };
    recognizer.StartCapturingGestures();

    Now while this works complete as expected in the editor, when I build the UWP app and launch it, it never recognizes any tap gestures. Ideas?

    Build Settings:
    UWP , enabled Mixed Reality Remoting
    Player Settings:
    Configuration: RuntimeV : .NET 4.x Equivalent
    Backend: IL2CPP
    Api compatibility: .NET 4.x
    ALL capabilities selected in publishing settings
    Windows Mixed Reality in XR settings
     
    abuklea likes this.
  2. JasonCostanza

    JasonCostanza

    Unity Technologies

    Joined:
    May 23, 2017
    Posts:
    295
    Hello MerkSim,

    would you happen to have a minimum repro project we could take a look at? We're happy to see you choosing to use the new feature, we definitely still love receiving feedback on it.
     
  3. MerkSim

    MerkSim

    Joined:
    Mar 1, 2018
    Posts:
    12
    Hi Jason,

    Thanks for the quick response!
    Here's a minimal repo, if you build it (and use the interface to set the IP of the hololens you re using) and press connect you ll get the remoting going. Then Just tap anywhere. Should get a "Tapped" message but it doesn't. (It does in the editor)

    https://bitbucket.org/MerkSim/remote_tap/src/master/
     
    JasonCostanza likes this.
  4. JasonCostanza

    JasonCostanza

    Unity Technologies

    Joined:
    May 23, 2017
    Posts:
    295
    Thanks MerkSim, I will take a look at this and get back to you soon
     
  5. JasonCostanza

    JasonCostanza

    Unity Technologies

    Joined:
    May 23, 2017
    Posts:
    295
    When I try to run your application you provided I get an error saying VR remoting is not enabled. Are you not seeing that error?
     
  6. MerkSim

    MerkSim

    Joined:
    Mar 1, 2018
    Posts:
    12
    You mean in the editor or after building the VS project? If in the editor (according to https://blogs.unity3d.com/2018/05/30/create-enhanced-3d-visuals-with-holographic-emulation-in-uwp/) :
    "while in Play mode inside Editor. To enable this option, go to Holographic Emulation window and make sure that the Emulation Mode is set to Remote to Device"

    If you mean in build, I dont see that no.
    Here's the output while running the application. Tried x64/ x86, IL2CPP / .NET
     
  7. MerkSim

    MerkSim

    Joined:
    Mar 1, 2018
    Posts:
    12
    Tell you what, can you by any chance provide me with a project (simple 1-2 script example like mine) where tapping while using holographic remoting works?
    Thanks!
     
  8. JasonCostanza

    JasonCostanza

    Unity Technologies

    Joined:
    May 23, 2017
    Posts:
    295
    Hey MerkSim, still working on your issue. Just dropping a note it's on my radar and not forgotten. I was struggling with "technical difficulties" last week with my system, hopefully off to a better start this week.
     
  9. JasonCostanza

    JasonCostanza

    Unity Technologies

    Joined:
    May 23, 2017
    Posts:
    295
    Update: We discovered we're having some issues with the new remoting feature. I've submitted a bug to our developers to have them work on this feature. Sorry for the delay and the not so good news. More issues than just your gesture issue, so hoping we can squash all those bugs in one pass.
     
  10. abuklea

    abuklea

    Joined:
    Mar 2, 2013
    Posts:
    13
    Hi Jason, do you have any further updates on this issue?
     
  11. JasonCostanza

    JasonCostanza

    Unity Technologies

    Joined:
    May 23, 2017
    Posts:
    295
    @abuklea what editor version are you on? I just checked the public release of 18.2.14f1 and gestures appear to be working. If you're on an editor older than that I would recommend updating to the newer version and see if that starts working again for you :)
     
  12. abuklea

    abuklea

    Joined:
    Mar 2, 2013
    Posts:
    13
    Hi @JasonCostanza .. I am currently using 2018.2.11f1

    The remoting from the editor has stopped working altogether for me now, but I've mainly been struggling to get the standalone UWP build (from Unity) to work with remoting. I have disabled all the relevant code and now trying to get it to work in the Editor again so perhaps I have broken something else and can't be sure yet. When it was working from the editor yesterday, I had the issue where the headtracking was working and I could see my app in the hololens, but none of the gestures were working at all. Now the remoting connects to the hololens ok.. and the remoting screen disappears in the device.. but my app does not display anything. I will continue investigating...

    My comment was mainly directed generally at you mentioning you are "having some issues with the new remoting feature"... and I was wondering how that relates to the other issues I am having with standalone Unity UWP app remoting to a hololens (and now the issues I am having in the editor also)

    With my standalone Unity UWP app I am currently getting the error "InvalidOperationException: Enable VR Streaming to allow connection to remote holographic device". I've tried loading the WindowsMR device and connecting to remote service in different orders, but no luck. It is not clear from the Unity or the Microsoft remoting documentations how to build a standalone UWP app from Unity to connect to a HoloLens via holographic remoting.

    Please see the question I have posted here: https://answers.unity.com/questions/1566982/how-to-enable-vr-streaming-unity-uwp-hololens-remo.html

    Currently this question has almost 200 followers, but so far no response... are you able to comment on this issue please?

    Any help would be much appreciated :)
     
  13. abuklea

    abuklea

    Joined:
    Mar 2, 2013
    Posts:
    13
    @JasonCostanza
    I have narrowed the issues I was having down to including the "None" entry in Virtual Reality SDKs list under XR Settings. If I have only the "Windows Mixed Reality" entry in the list then everything works as expected. As soon as I add "None" (either before or after the WindowsMR entry), the remoting connects to the HoloLens, the remoting window disappears on the hololens, however my application does not appear on the device (blank display). If I remove "None" it works again.

    See my full update to my question here: https://answers.unity.com/questions/1566982/how-to-enable-vr-streaming-unity-uwp-hololens-remo.html

    Thanks
     
    Last edited: Nov 1, 2018
  14. JasonCostanza

    JasonCostanza

    Unity Technologies

    Joined:
    May 23, 2017
    Posts:
    295
    Hey Abuklea,

    I'll try to parse through all your information here. This is how I currently understand your setup:
    • Your script can connect to the Hololens with only WMR in the SDK list
    • Your Holographic Emulation Window can connect to the Hololens with only WMR in the SDK list
    • Desired XR SDK list contains in this order:
      • None
      • Windows Mixed Reality
    • If "None" is on the top of the list, your connection goes through but the Hololens remains black
    • If "Windows Mixed Reality" is only in the list, then it works fine and you see your app in the headset
    Questions:
    1. In your configuration where "None" is on top, are you loading device after establishing the connection?
      We expect the headset to be black at this configuration because the app is started in a standard UWP application and no device has been loaded. So you're connected remotely but not switched to the headset with LoadDeviceByName

      We should see the behavior in this order:
      1. App launches on PC (Hololens is at remote connection screen displaying your IP)
      2. Connect to the remoting application (Hololens will go to black to represent you are connected)
      3. Load the MR Device (Hololens will start rendering frames and mirror audio from the PC)

      To Disconnect:
      1. Disconnect from the remoting application on the Hololens using HolographicRemoting.Disconnect()
      2. Confirm Hololens returns to the Remoting application displaying your IP address
      2. Load the "None" device to return to a standard UWP application on your desktop which unloads the MR device
    2. Have you seen our blog post on remoting as well?
      https://blogs.unity3d.com/2018/05/30/create-enhanced-3d-visuals-with-holographic-emulation-in-uwp/
    3. To resolve the error "InvalidOperationException: Enable VR Streaming to allow connection to remote holographic device"
      1. Confirm the checkbox for "WSA Holographic Remoting Support" is checked if you're building the UWP solution
      2. Confirm that if you're simulating in editor, that the Holographic Emulation Window is open when you attempt to initiate the connection. However, do not change or fill in the IP field of the emulation window. Just leave it on "Remote to device" as shown in the blog post above.
    Update us on how that works out and we can diagnose from there if you're still hitting issues :)
     
  15. abuklea

    abuklea

    Joined:
    Mar 2, 2013
    Posts:
    13
    Thanks @JasonCostanza I really appreciate your help :D

    1. I will definitely confirm all of these steps when I get the time here at work.. hopefully this week. I believe I was doing those steps in the order you have mentioned - but will re-confirm. Step 3 (Load the MR device) was only producing a black screen and not rendering frames. Can I confirm that I am loading the device (XRSettings.LoadDeviceByName) using the "WindowsMR" string.. and not "HoloLens" or another string?

    2. Yes I've read, followed and re-read that blog post.. very helpful. The issue seems to be the XR SDK lists but as mentioned I will reconfirm all of the steps

    3. Thank you. Seems to be solved now - I did have all the correct boxes checked. However due to the issue above I was trying different orders of connecting and loading devices, and trying to load the WindowsMR device before connecting to the remoting service caused this error. You have now confirmed for me the correct order

    Cheers!
     
    JasonCostanza likes this.
  16. JasonCostanza

    JasonCostanza

    Unity Technologies

    Joined:
    May 23, 2017
    Posts:
    295
    WIndowsMR is the correct string to load
     
    abuklea likes this.
  17. abuklea

    abuklea

    Joined:
    Mar 2, 2013
    Posts:
    13
    Hi @JasonCostanza

    I have checked my project and code again.. and I simply cannot get the remoting to work in Unity (via code; the holographic emulation in the editor works fine). Neither in the Editor nor in a standalone build can I get it to work via code.

    Remoting connects successfully (HolographicRemoting.ConnectionState == HolographicStreamerConnectionState.Connected),
    Device seems to load (XRSettings.LoadDeviceByName(newDevice); yield return null; XRSettings.enabled = true)
    but I simply get a blank screen in the hololens device
    - app starts
    - remoting connects and green connected shown in the holographic emulation panel
    - WindowsMR device loaded
    - On hololens, remoting player screen (IP address etc) disappears, blank display
    - In editor play window, app is running, no head tracking; camera remains a stationary with no rendering to the device

    Do I need a special camera rig setup, or anything else I'm missing?

    I have simplified the scene to just a camera, a cube, and the script from https://docs.unity3d.com/2018.2/Documentation/ScriptReference/XR.WSA.HolographicRemoting.html
    I have read and followed this https://blogs.unity3d.com/2018/05/30/create-enhanced-3d-visuals-with-holographic-emulation-in-uwp/

    I have tried many variations; order of connecting to remoting and device, waiting for longer between connecting and loading the device etc. Grasping at straws now... any further advice would be awesome.. thanks

    HoloLens: RS5 update.. OS build 10.0.17763.1
    - Holographic remoting player running.. correct IP address confirmed
    Unity 2018.2.14f1 (64bit)
    - UWP platform selected
    - VR supported enabled
    - WSA holographic remoting enabled
    - XR settings (None, Windows Mixed Reality)
    - Holographic emulation panel open (emulation mode = remote to device)... I start the app in disconnected state and the connected status is enabled successfully from script on play (shown in the emulation panel)
    - Tried both IL2CPP and .NET scripting backends
    - Tried both .NET 4.x and .NET Standard 2.0 API levels
     
    Last edited: Nov 14, 2018
  18. GameDevCouple_I

    GameDevCouple_I

    Joined:
    Oct 5, 2013
    Posts:
    1,532
    Probably doesnt help that your running a preview build of the HoloLens OS, RS5 is not officially out yet...
     
  19. abuklea

    abuklea

    Joined:
    Mar 2, 2013
    Posts:
    13
    @GameDevCouple_I True.. I did try it on the previous version before I upgraded - I thought upgrading may solve the issue. I can probably revert one of our devices back to previous non-preview build, and try again if you think it maybe causing the issue. Thanks
     
  20. GameDevCouple_I

    GameDevCouple_I

    Joined:
    Oct 5, 2013
    Posts:
    1,532
    Yeah i think it might be, its always worth checking anyway as the insider builds have had many issues on our end!
     
  21. abuklea

    abuklea

    Joined:
    Mar 2, 2013
    Posts:
    13
    Hi @JasonCostanza

    FYI ... after a suggestion by @GameDevCouple_I - I reverted a hololens back to the April 2018 RS4 Update (10.0.17134.111) from RS5. However this did not fix the problem, and the exact same thing is happening as described previously; where the remoting screen disappears to show a blank display.

    I had already tried this before upgrading to RS5 but since I am completely out of ideas I thought I would try to reinstall RS4 and try it again, based on the suggestion.

    Would you have any further advice on what is going wrong? I have the most simple scene possible and have followed all advice, still with no luck.

    If anyone can provide a working UWP/remoting Unity scene for me to test that would also be very helpful!
     
  22. joschemd

    joschemd

    Joined:
    Sep 16, 2015
    Posts:
    4
    Hi @JasonCostanza

    Any further update on this?
    I too am running issues including the Enable VR Streaming to allow connections to remote holographic device.

    - [Build] Attempting to connect in a build results in the above error message
    - [Editor] Remoting under the holographic remoting tab works when activated by the connect button.
    - [Editor] Setting it to remote to device but not connecting under the tab, then in code to load/connect I get audio streaming ( from desktop browser source) but nothing else but black

    Unity Version 2018.3.2f1
    Visual Studio 2017
    Desktop OS: (17763.288)
    Holographic OS: (17763.134)
     
    MohsenneChaverdie and abuklea like this.
  23. abuklea

    abuklea

    Joined:
    Mar 2, 2013
    Posts:
    13
    Hi all...

    @JasonCostanza
    @Unity_Wesley
    @lukasz_pasek
    @De-Panther
    @francescocagnetta
    @svenbrunner90
    @paulmorganrenaultf1

    Sorry to spam you folks but am running out of steam trying to get this working...
    Does anyone have any further updates or advice about these issues listed?

    Or is there someone else at Unity who is able to respond and provide some advice.. or even better a simple working project? That would be extremely helpful.

    Also see my question and comment posted here: https://answers.unity.com/questions/1566982/how-to-enable-vr-streaming-unity-uwp-hololens-remo.html

    Thanks heaps!
     
    Last edited: Feb 1, 2019
  24. MohsenneChaverdie

    MohsenneChaverdie

    Joined:
    Nov 23, 2016
    Posts:
    32
    same problem here
     
  25. MohsenneChaverdie

    MohsenneChaverdie

    Joined:
    Nov 23, 2016
    Posts:
    32
    Ok, this method works for me.
     
  26. xlfpx

    xlfpx

    Joined:
    Jan 13, 2017
    Posts:
    1
    Hi everyone,

    I'd like to get back to the actual issue this thread was meant to tackle: Gestures cannot be recognized in a UWP-standalone application using holographic remoting.
    I've been facing this issue for quite some time now but I cannot get it to work so I suspect the issue must lie within Unity itself. I tried both the .net and c++ scripting backends, as well as granting all UWP app-capabilities and calling gestureRecognizer.StartCapturingGestures() only with a delay after a holographicRemoting-connection to the Hololens was established and the WindowsMR-device was successfully loaded. None of this helped. All works well when running it in the editor, yet when building a UWP-app gestures simply aren't being recognized.

    @JasonCostanza Is this issue still on your radar, do you know whether it's being looked into, or do you happen to know of project-setup or code-related bad practices that can cause this problem? Maybe you can take a look at my demo-project which produces this issue with minimal code?

    @MerkSim You first brought up this issue. Did you since find a way to get it to work?

    To make sure we're all on the same page and anyone can easily reproduce this issue on their own, I created a minimal demo-project showcasing this problem, including a detailed readme.md explaining how to set up and build the project.
    Anyone should feel free to contribute to this project. You can find it here

    If anyone is still struggling with getting the remoting itself working (regardless of gesture recognition) check out this guide I made.
     
    JasonCostanza likes this.
  27. JasonCostanza

    JasonCostanza

    Unity Technologies

    Joined:
    May 23, 2017
    Posts:
    295
    It is still on my radar. I admit updates have been few and far between, but it is not forgotten.
     
    xlfpx likes this.
  28. MerkSim

    MerkSim

    Joined:
    Mar 1, 2018
    Posts:
    12
    Hey, we didn't put too much weight on an experimental feature (even though it would have been very convenient). I think it was passed to another department and they implemented they own version or something :(
     
  29. Finer_Games

    Finer_Games

    Joined:
    Nov 21, 2014
    Posts:
    26
    Chiming in here on a potentially related issue. I'm able to get remoting working in the editor using 2018.3.11f1, but nothing renders -- not even the splash screen -- in my builds unless I disable the remoting option. Is this the expected behaviour?
     
  30. JasonCostanza

    JasonCostanza

    Unity Technologies

    Joined:
    May 23, 2017
    Posts:
    295
    We actually just implemented a fix for this exact issue a couple days ago but I'm on holding pattern until the fix lands in the main line branch. Then once that happens I plan to open backport requests to both 19.2 and 18.3. Hopefully I can get those opened real soon for you guys.
     
  31. JasonCostanza

    JasonCostanza

    Unity Technologies

    Joined:
    May 23, 2017
    Posts:
    295
    Update! Backports got opened this morning :)