Hololens Remoting Build gestures not recognized

Hello,

I 've been trying to debug this thing for a day and a half and I cant wrap my head around it. It all revolves around the new 2018.2 Remoting feature.
Here's my process :

- HolographicRemoting.Connect(IP, bitrate) //connect to the hololens device
- StartCoroutine(LoadDevice("WindowsMR")); // Loaddevice like the in unity blog
- recognizer = new GestureRecognizer(); //initialize a gesturerecognizer and subscribe function to tapped
recognizer.Tapped += (args) => { SendGazePoint(); };
recognizer.StartCapturingGestures();

Now while this works complete as expected in the editor, when I build the UWP app and launch it, it never recognizes any tap gestures. Ideas?

Build Settings:
UWP , enabled Mixed Reality Remoting
Player Settings:
Configuration: RuntimeV : .NET 4.x Equivalent
Backend: IL2CPP
Api compatibility: .NET 4.x
ALL capabilities selected in publishing settings
Windows Mixed Reality in XR settings

 

Hi Jason,

Thanks for the quick response!
Here's a minimal repo, if you build it (and use the interface to set the IP of the hololens you re using) and press connect you ll get the remoting going. Then Just tap anywhere. Should get a "Tapped" message but it doesn't. (It does in the editor)

https://bitbucket.org/MerkSim/remote_tap/src/master/

 

You mean in the editor or after building the VS project? If in the editor (according to https://blogs.unity3d.com/2018/05/30/create-enhanced-3d-visuals-with-holographic-emulation-in-uwp/) :
"while in Play mode inside Editor. To enable this option, go to Holographic Emulation window and make sure that the Emulation Mode is set to Remote to Device"

If you mean in build, I dont see that no.
Here's the output while running the application. Tried x64/ x86, IL2CPP / .NET

 

Tell you what, can you by any chance provide me with a project (simple 1-2 script example like mine) where tapping while using holographic remoting works?
Thanks!

 

Hi Jason, do you have any further updates on this issue?

 

Hi @JasonCostanza .. I am currently using 2018.2.11f1

The remoting from the editor has stopped working altogether for me now, but I've mainly been struggling to get the standalone UWP build (from Unity) to work with remoting. I have disabled all the relevant code and now trying to get it to work in the Editor again so perhaps I have broken something else and can't be sure yet. When it was working from the editor yesterday, I had the issue where the headtracking was working and I could see my app in the hololens, but none of the gestures were working at all. Now the remoting connects to the hololens ok.. and the remoting screen disappears in the device.. but my app does not display anything. I will continue investigating...

My comment was mainly directed generally at you mentioning you are "having some issues with the new remoting feature"... and I was wondering how that relates to the other issues I am having with standalone Unity UWP app remoting to a hololens (and now the issues I am having in the editor also)

With my standalone Unity UWP app I am currently getting the error "InvalidOperationException: Enable VR Streaming to allow connection to remote holographic device". I've tried loading the WindowsMR device and connecting to remote service in different orders, but no luck. It is not clear from the Unity or the Microsoft remoting documentations how to build a standalone UWP app from Unity to connect to a HoloLens via holographic remoting.

Please see the question I have posted here: https://answers.unity.com/questions/1566982/how-to-enable-vr-streaming-unity-uwp-hololens-remo.html

Currently this question has almost 200 followers, but so far no response... are you able to comment on this issue please?

Any help would be much appreciated

 

@JasonCostanza
I have narrowed the issues I was having down to including the "None" entry in Virtual Reality SDKs list under XR Settings. If I have only the "Windows Mixed Reality" entry in the list then everything works as expected. As soon as I add "None" (either before or after the WindowsMR entry), the remoting connects to the HoloLens, the remoting window disappears on the hololens, however my application does not appear on the device (blank display). If I remove "None" it works again.

See my full update to my question here: https://answers.unity.com/questions/1566982/how-to-enable-vr-streaming-unity-uwp-hololens-remo.html

Thanks

 

Thanks @JasonCostanza I really appreciate your help

1. I will definitely confirm all of these steps when I get the time here at work.. hopefully this week. I believe I was doing those steps in the order you have mentioned - but will re-confirm. Step 3 (Load the MR device) was only producing a black screen and not rendering frames. Can I confirm that I am loading the device (XRSettings.LoadDeviceByName) using the "WindowsMR" string.. and not "HoloLens" or another string?

2. Yes I've read, followed and re-read that blog post.. very helpful. The issue seems to be the XR SDK lists but as mentioned I will reconfirm all of the steps

3. Thank you. Seems to be solved now - I did have all the correct boxes checked. However due to the issue above I was trying different orders of connecting and loading devices, and trying to load the WindowsMR device before connecting to the remoting service caused this error. You have now confirmed for me the correct order

Cheers!

 

Hi @JasonCostanza

I have checked my project and code again.. and I simply cannot get the remoting to work in Unity (via code; the holographic emulation in the editor works fine). Neither in the Editor nor in a standalone build can I get it to work via code.

Remoting connects successfully (HolographicRemoting.ConnectionState == HolographicStreamerConnectionState.Connected),
Device seems to load (XRSettings.LoadDeviceByName(newDevice); yield return null; XRSettings.enabled = true)
but I simply get a blank screen in the hololens device
- app starts
- remoting connects and green connected shown in the holographic emulation panel
- WindowsMR device loaded
- On hololens, remoting player screen (IP address etc) disappears, blank display
- In editor play window, app is running, no head tracking; camera remains a stationary with no rendering to the device

Do I need a special camera rig setup, or anything else I'm missing?

I have simplified the scene to just a camera, a cube, and the script from https://docs.unity3d.com/2018.2/Documentation/ScriptReference/XR.WSA.HolographicRemoting.html
I have read and followed this https://blogs.unity3d.com/2018/05/30/create-enhanced-3d-visuals-with-holographic-emulation-in-uwp/

I have tried many variations; order of connecting to remoting and device, waiting for longer between connecting and loading the device etc. Grasping at straws now... any further advice would be awesome.. thanks

HoloLens: RS5 update.. OS build 10.0.17763.1
- Holographic remoting player running.. correct IP address confirmed
Unity 2018.2.14f1 (64bit)
- UWP platform selected
- VR supported enabled
- WSA holographic remoting enabled
- XR settings (None, Windows Mixed Reality)
- Holographic emulation panel open (emulation mode = remote to device)... I start the app in disconnected state and the connected status is enabled successfully from script on play (shown in the emulation panel)
- Tried both IL2CPP and .NET scripting backends
- Tried both .NET 4.x and .NET Standard 2.0 API levels

 

Probably doesnt help that your running a preview build of the HoloLens OS, RS5 is not officially out yet...

 

@GameDevCouple_I True.. I did try it on the previous version before I upgraded - I thought upgrading may solve the issue. I can probably revert one of our devices back to previous non-preview build, and try again if you think it maybe causing the issue. Thanks

 

Yeah i think it might be, its always worth checking anyway as the insider builds have had many issues on our end!

 

Hi @JasonCostanza

FYI ... after a suggestion by @GameDevCouple_I - I reverted a hololens back to the April 2018 RS4 Update (10.0.17134.111) from RS5. However this did not fix the problem, and the exact same thing is happening as described previously; where the remoting screen disappears to show a blank display.

I had already tried this before upgrading to RS5 but since I am completely out of ideas I thought I would try to reinstall RS4 and try it again, based on the suggestion.

Would you have any further advice on what is going wrong? I have the most simple scene possible and have followed all advice, still with no luck.

If anyone can provide a working UWP/remoting Unity scene for me to test that would also be very helpful!

 

Hi @JasonCostanza

Any further update on this?
I too am running issues including the Enable VR Streaming to allow connections to remote holographic device.

- [Build] Attempting to connect in a build results in the above error message​

- [Editor] Remoting under the holographic remoting tab works when activated by the connect button.
- [Editor] Setting it to remote to device but not connecting under the tab, then in code to load/connect I get audio streaming ( from desktop browser source) but nothing else but black

Unity Version 2018.3.2f1
Visual Studio 2017
Desktop OS: (17763.288)
Holographic OS: (17763.134)

 

Hi all...

@JasonCostanza
@Unity_Wesley
@lukasz_pasek
@De-Panther
@francescocagnetta
@svenbrunner90
@paulmorganrenaultf1

Sorry to spam you folks but am running out of steam trying to get this working...
Does anyone have any further updates or advice about these issues listed?

Or is there someone else at Unity who is able to respond and provide some advice.. or even better a simple working project? That would be extremely helpful.

Also see my question and comment posted here: https://answers.unity.com/questions/1566982/how-to-enable-vr-streaming-unity-uwp-hololens-remo.html

Thanks heaps!

 

same problem here

 

Ok, this method works for me.

 

Hi everyone,

I'd like to get back to the actual issue this thread was meant to tackle: Gestures cannot be recognized in a UWP-standalone application using holographic remoting.
I've been facing this issue for quite some time now but I cannot get it to work so I suspect the issue must lie within Unity itself. I tried both the .net and c++ scripting backends, as well as granting all UWP app-capabilities and calling gestureRecognizer.StartCapturingGestures() only with a delay after a holographicRemoting-connection to the Hololens was established and the WindowsMR-device was successfully loaded. None of this helped. All works well when running it in the editor, yet when building a UWP-app gestures simply aren't being recognized.

@JasonCostanza Is this issue still on your radar, do you know whether it's being looked into, or do you happen to know of project-setup or code-related bad practices that can cause this problem? Maybe you can take a look at my demo-project which produces this issue with minimal code?

@MerkSim You first brought up this issue. Did you since find a way to get it to work?

To make sure we're all on the same page and anyone can easily reproduce this issue on their own, I created a minimal demo-project showcasing this problem, including a detailed readme.md explaining how to set up and build the project.
Anyone should feel free to contribute to this project. You can find it here

If anyone is still struggling with getting the remoting itself working (regardless of gesture recognition) check out this guide I made.

 

Hey, we didn't put too much weight on an experimental feature (even though it would have been very convenient). I think it was passed to another department and they implemented they own version or something

 

Chiming in here on a potentially related issue. I'm able to get remoting working in the editor using 2018.3.11f1, but nothing renders -- not even the splash screen -- in my builds unless I disable the remoting option. Is this the expected behaviour?

 

A year later, I wonder if you have solved this problem? My project also encountered the same problem, unable to detect gestures in remote mode. I am using 2019.1.14. If you have solved in which version, please let me know

 

Sorry, I forgot to reply to you. Please see my last message