VR Desktop Mirror asset - x Windows and HTC Vive

This asset allow viewing and interacting with Windows Desktop (mainly Unity Editor) within VR.
Useful for example for tuning variables in inspector without need to leave the VR headset.

All source-code (C# Unity and C++ DLL) is released on GitHub under a LGPL licence: https://github.com/Clodo76/vr-desktop-mirror

Requirements

• HTC Vive or Oculus Rift CV1/DK2
• Windows 64 bit
• SteamVR asset if used with HTC Vive

How to

1. Install this package: Download (41 KB!) - v1.0
   If you have a compilation error about missing SteamVR_Controller, comment the first line
   
   Code (CSharp):

   1. #define VDM_SteamVR
   This will remove the support of HTC Vive.
   
   
2. Drag "VR Desktop Mirror > Prefabs > VR Desktop Mirror" in project tree.
   
3. Start PlayMode, pick your VR headset.
   
   With Keyboard / Mouse / Oculus Rift:
   - Press the left control to show the desktop. Quick press to hide. Hold press to move. During hold press, use the scroll wheel to adjust distance.
   - Press the left alt to switch to zoom mode (for better aiming). Use the scroll wheel to adjust the zoom distance.
   Important: If you interact with Unity Editor UI, the detection of keyboard for show/zoom don't work. Re-give the focus to the Game View window.
   Note also that it reflect the mouse event in the REAL screen position, NOT the mouse over the plane/texture.
   
   With HTC Vive:
   - Click the Grip button on any controller. It will show the desktop.
   - Point with the controller the desktop and retain press the Grip for positioning.
   - Point and short-click the Grip to hide.
   - Interact with the desktop with any controllers.
   - Trigger act as mouse left click.
   - Thumbpad act as mouse left or right click, depend on the side pressed.

Properties of the asset have tooltip with specific documentation.

The BIG issue

Some functionality of Unity (for example, menu and context menu) pause the playback, and so the VR environment. I consider this issue an Unity bug, because occur even if "Play in background" is enabled.
I opened a topic in the Editor forum, and i also fill a bugreport.

Until the bug is fixed, the asset detect and close any menu and context-menu to avoid the VR environment lock. This can be disabled by unchecking the property "HackUnityBug".

The following it's a list (not exaustive) of action not allowed:

• Unity Editor Menu (for example File, Window etc).
• Any Context Menu (for example right click on a gameobject).
• Any enum property in inspector (that are managed with a context menu).
• Removing a component (that are managed with a context menu; Adding a component work).
• etc.

A little issue

Unity for a Input.GetKeyDown(KeyCode.X) return always false also if the X key is pressed, but Game View in Play Mode don't have the focus. This issue don't occur with HTC Vive buttons. If anyone know how to resolve this, please write a post. Thanks.

Faq

Q- What if i use "Drag" or "Touchpad click" in my game?
A- To avoid collisions, in your script check the bool "VdmDesktopManager.ActionInThisFrame", and skip your behiavour if is True (Input accepted for interaction with the desktop).

Q- Future of this asset?
A- There are a lot of possible improvements:

• Support of multiple monitors
• More buttons configurations preset
• Simulation of mouse wheel or center mouse click
• Virtual keyboard

But honestly the Unity bug described above it's serious, i hope that someone can help me to point out to Unity team the issue, otherwise it seem useless to improve this project.

Thanks for any feedback.

 

Hey there! I was searching around for something like what you've created and found your post. I'm working with the DK2 and I was trying to have the screen as a material on a plane, rather than a pop-up menu that can only be reached through the controllers. Do you think this would be possible with your script? It looks really cool so far.

 

This is a new way to see the technology. It actually make virtual reality enjoyable and impressive. This is a part of standardize experience to the customer's home.

 

Hi, i read your comment, i will implement support for Oculus (and an interface that not require Vive controller) as soon as possible. Thanks for the feedback.

 

Project updated:
- Oculus DK2/CV1 support. HTC Vive support can be removed by commenting a #define directives.
- Support of mouse / keyboard for seated headset (Oculus)
- Minor improvement and bugfixes

Topic rebuilded after the forum rollback.

p.s. i cannot update the title of the topic. Can a moderator change it to

? thanks.

 

So I tried this in Unity 5.3.5 and the virtual desktop plane is white. I can use the controller as a mouse but I can't see anything in game.

I'm using Windows and the HTC Vive.

 

Just wanted to say this is awesome and I've been looking to implement something similar in one of my projects! I sent you a message about it.

-Thanks again

 

Download updated to fix a compilation error if used without SteamVR.

Sorry, i can't reproduce that. You are sure you use the Unity 64 bit edition with this asset? You don't have any log error?

 

is it possible instead of having the desktop rendered in a texture, to render different application frames opened in desktop, in different textures?

 

Any update on this for 5.4?

 

I found the problem. I'm using a laptop and I guess it captures the desktop differently. Any ideas on how to get around this?

 

have a 4970k and an ASUS GeForce GTX 970 STRIX, and I experience zero stuttering in those games. All stock, no special configurations. Edit: No idea how much of a difference it makes but I have 32GB of RAM.

 

There is a Error , Can anyone help me.....

Its occurs when I Import asset there was 3 error 2 was of about //#define VDM_SteamVR and 3 one is this
Kindly help me I really like your asset


Assets/VR Desktop Mirror/Scripts/VdmDesktopManager.cs(386,16): error CS1501: No overload for method `IssuePluginEvent' takes `2' arguments

 

How to use it Can Anyone help me?

 

@abrar-robotics I'm not the developer but I got it working by importing the package as normal, then importing the SteamVR plugin from the asset store, then deleting the camera and using the prefab steam rig and steamvr into the empty scene. It then worked for me.

 

@Clodo I've sent you a message relating to this plugin. I make a player called Whirligig and I'd like to use this plugin to allow people to mirror their desktop to watch content like youtube etc. I think I've got it working the way I want but still need to implement it and also have some other questions. I've sent you a private message but I thought I'd leave a message here as I can't think of any other way of getting in touch

Great work on this by the way. This should be an Asset store plugin rather than just given away free.

 

Hi to all, sorry for the delay.
I updated the .unitypackage for Unity 5.4.1f1. Download: https://github.com/Clodo76/vr-desktop-mirror/raw/master/Output/VrDesktopMirror.unitypackage
I released all the source-code (C# Unity and C++ DLL) on GitHub, under LGPL. https://github.com/Clodo76/vr-desktop-mirror
I added multiscreen support.
I added some property related to multiscreen and colorspace.

Previous version works only with Unity 64 bit, the new version have both 32/64 dll.

Please note:
- Require at least Windows Vista. DON'T WORK with Windows 7. It use Windows API that are available only from Vista and above, nothing i can do about this.
- I cannot detect if SteamVR/Vive asset is installed or not. See the first post if anyone have a compilation issue.

Honestly, the desktop mirror feature maybe, but this Asset aim to be a VR interface for Unity, but the Unity bug described in first post are too much trouble.

 

How to use the multiscreen function?
I imported the new package but I have no idea how I can use it

 

Is it possible to add a function that can not only show the desktop on the unity, it also show the specified window like folder or chrome on the unity? It just like the AR screen from Leap Motion.

0:35

 

You need multiple screens plugged in to your PC.

It's possible to 'fake' it using a headless hdmi adaptor such as: https://www.megamac.com/products/newertech-hdmi-headless-video-accelerator-nwtadp4khead

 

Is there anyway to render the desktop on a gameobject?

 

you will need multiple ports and screens to plugs in.
check out http://galacticbrief.com/index.php/shop/
for more detail and items to help connect everything.

 

You can modify the scripts in the 'VR Desktop Mirror' prefab so it doesn't re-position/resize when you enable/disable the display.

VdmDesktop.cs -> if you comment out the code in Update you can probably get closer to what you want to acheive by positioning the prefab where you need it in the scene and playing with the scale in VdmDesktopManager.cs so it fits better.

 

*Edit

Found the option by myself, already implemented

 

When I build the game I get an Access Violation error?

DesktopCapture.dll caused an Access Violation (0xc0000005)
in module DesktopCapture.dll at 0023:61ba11c0.

 

This is a frequent question.
No, sorry. My asset use Windows Duplication API, that have the advantage of highest framerate, but don't allow to capture a single window.

 

Please help me here. I'm developing a game and this asset is a core part of it. The game was supposed to be released last week but this keeps crashing in the build. It works perfect in the editor, but after about 30 seconds in the build it crashes, and when I debug it says 'ntdll.dll could not be loaded'.

 

Hey! Great work! Do you have any idea why inside the editor I only get "white screen", but in built code it works as expected?

Thanks!

 

Can I just say that you are an absolute angel for creating this? I had the idea (in the shower, where all the best ideas come to you) to mirror my desktop while working on my game and... Looking up possible tutorials I find this; exactly what I'm looking for. I was wondering if you could mirror your desktop to a game object, I'd love to put a handle on it and carry it with me.

My vive is on the way and I am very excited to test this out!

 

@Clodo I have got this setup and its great. However I can see the colour depth is pretty poor. Blacks are really washed out and it seems like it is 16bit or similar. Any way to get a better colour depth? Is it a quality setting or something. Thanks!

 

I am having the same trouble it works great but crashes at around the 30 second mark.

 

Could @Clodo give some answers? I really want to use this but i get the error @CEOManDan and @Echoshard1 describe and the issue @dandoVR mentioned. Also when i comment the #define i get a random bool that is not defined somewhere in the code. This works perfectly for what i want to do and i really would like it fixed!

 

Anyone know how to do this for Mac OS??

 

How can I get a Texture2D from the script? I want to make a screenshot and put it to custom plane.

 

Hello, does anyone know if this prefab still works? I tried dragging the prefab onto my project tree and using my mouse and keyboard but all i get is an empty white screen

 

It works with HTC Vive for me. You can try next:

Code (CSharp):

1. // #define VDM_SteamVR

 

Im trying this out with windows mixed reality headset, specifically acer. I'm using Holotoolkit with Unity because it allows me to program for the headset and test scenes directly onto the headset. Its strange how this desktop mirror works without the headset but once I plug in my headset I only get a white screen. Maybe it has to do with which graphics card is being used. I heard desktop duplication api doesnt work with Nvidea. However I have to use Nvidea for windows mixed reality headsets

 

Hi there,
Please I want to use this with the Oculus Touch .. Would you please guide me through the steps to do so?
Also, is there a guide on the functions used in the DLL so that I can create further utilities on top of this great asset?

 

Wow , Impressive work! Thanks
I am a beginner in HTC Vive, and I am trying to do a simple thing that you already have done here:
I have a embedded webpage ( youtube search page ) inside my unity3D game, and I want to be able to use the HTC Vive controller and click on a certain video to play, I am not sure how to use the controller as a mouse !
Can you please help me through ?


Thanks alot

 

did you ever get this issue fixed? I'm having similar problems when building the project and running from the executable.

 

How to actually get a capture of a single window? its the core part of my AR project and I can't seem to make it work! Help please

 

Hey. Not sure if this asset is still being worked on, but im trying it out and it works fine on an external monitor but not on my laptops screen. I just get a white screen, but im able to move the mouse around. Has this something to do with it being the integrated graphics card? is there a work around? Best regards.

 

Did you get this work? I got a white screen also!

 

Hello, are you able to fix the white screen? If YES, could you please tell how to do it to fix this problem!

 

I get an error saying that the type or namespace name 'SteamVR_TrackedObject' could not be found. And when I try to comment the first line saying #define VDM_SteamVR, I get a lot of errors saying 'VRSettings' is obsolete: 'VRSettings has been moved and renamed. Use UnityEngine.XR.XRSettings instead (UnityUpgradable)'

 

me as well! really need help in that! using Unity 2019 with Oculus Rift S

 

For all who are having white screen issue, probably because there are 2 graphics cards (usually on a laptop), when I tried on my Desktop which only has one graphics card, it works.

Then I tried again in my laptop, but make sure to set unity to use the integrated graphics, and it works.

So perhaps this might help to all who experiencing the white screen issue.

 

I don't mean to revive a dead forum post, but does anyone have any updated information on this? I got it working in my program I was making which allows people to present to each other using PUN2 etc, so I got it working locally first then went to test it with PUN2 and found that even without the added extras I put in for it to work over a network, the actual gameobjects themselves seem to cause my built programs to crash when you enter a scene with them in, I found online people had similar symptoms to do with using shaders but as of yet I'm unable to find any solution because it being the build and all I don't know how to get a log of the issue it encountered while running.

Any help would be massively appreciated I am trying to get this program working for this week and this additional function would really make it exceptional. It's crazy that it is still very hard to find a good way of sharing your screen to unity.

 

im not a programmer. its saying "m.controllerAttach" doesnt exist

 

Need to move it outside of the #IF statement it is in. Currently, the variable is only created if Steam VR is enabled and if not then causes error of not exisiting