Search Unity

  1. Read here for Unity's latest plans on OpenXR.
    Dismiss Notice

VR Desktop Mirror asset - x Windows and HTC Vive

Discussion in 'AR/VR (XR) Discussion' started by Clodo, May 26, 2016.

  1. Clodo

    Clodo

    Joined:
    Dec 27, 2013
    Posts:
    40

    This asset allow viewing and interacting with Windows Desktop (mainly Unity Editor) within VR.
    Useful for example for tuning variables in inspector without need to leave the VR headset.



    All source-code (C# Unity and C++ DLL) is released on GitHub under a LGPL licence: https://github.com/Clodo76/vr-desktop-mirror

    Requirements
    • HTC Vive or Oculus Rift CV1/DK2
    • Windows 64 bit
    • SteamVR asset if used with HTC Vive

    How to
    1. Install this package: Download (41 KB!) - v1.0
      If you have a compilation error about missing SteamVR_Controller, comment the first line
      Code (CSharp):
      1. #define VDM_SteamVR
      This will remove the support of HTC Vive.

    2. Drag "VR Desktop Mirror > Prefabs > VR Desktop Mirror" in project tree.
    3. Start PlayMode, pick your VR headset.

      With Keyboard / Mouse / Oculus Rift:
      - Press the left control to show the desktop. Quick press to hide. Hold press to move. During hold press, use the scroll wheel to adjust distance.
      - Press the left alt to switch to zoom mode (for better aiming). Use the scroll wheel to adjust the zoom distance.
      Important: If you interact with Unity Editor UI, the detection of keyboard for show/zoom don't work. Re-give the focus to the Game View window.
      Note also that it reflect the mouse event in the REAL screen position, NOT the mouse over the plane/texture.

      With HTC Vive:
      - Click the Grip button on any controller. It will show the desktop.
      - Point with the controller the desktop and retain press the Grip for positioning.
      - Point and short-click the Grip to hide.
      - Interact with the desktop with any controllers.
      - Trigger act as mouse left click.
      - Thumbpad act as mouse left or right click, depend on the side pressed.
    Properties of the asset have tooltip with specific documentation.

    The BIG issue

    Some functionality of Unity (for example, menu and context menu) pause the playback, and so the VR environment. I consider this issue an Unity bug, because occur even if "Play in background" is enabled.
    I opened a topic in the Editor forum, and i also fill a bugreport.

    Until the bug is fixed, the asset detect and close any menu and context-menu to avoid the VR environment lock. This can be disabled by unchecking the property "HackUnityBug".

    The following it's a list (not exaustive) of action not allowed:
    • Unity Editor Menu (for example File, Window etc).
    • Any Context Menu (for example right click on a gameobject).
    • Any enum property in inspector (that are managed with a context menu).
    • Removing a component (that are managed with a context menu; Adding a component work).
    • etc.
    A little issue

    Unity for a Input.GetKeyDown(KeyCode.X) return always false also if the X key is pressed, but Game View in Play Mode don't have the focus. This issue don't occur with HTC Vive buttons. If anyone know how to resolve this, please write a post. Thanks.

    Faq

    Q- What if i use "Drag" or "Touchpad click" in my game?
    A- To avoid collisions, in your script check the bool "VdmDesktopManager.ActionInThisFrame", and skip your behiavour if is True (Input accepted for interaction with the desktop).

    Q- Future of this asset?
    A- There are a lot of possible improvements:
    • Support of multiple monitors
    • More buttons configurations preset
    • Simulation of mouse wheel or center mouse click
    • Virtual keyboard
    But honestly the Unity bug described above it's serious, i hope that someone can help me to point out to Unity team the issue, otherwise it seem useless to improve this project.

    Thanks for any feedback.
     
    Last edited: Nov 6, 2016
  2. LasseRB

    LasseRB

    Joined:
    May 29, 2016
    Posts:
    4
    Hey there! I was searching around for something like what you've created and found your post. I'm working with the DK2 and I was trying to have the screen as a material on a plane, rather than a pop-up menu that can only be reached through the controllers. Do you think this would be possible with your script? It looks really cool so far. :)
     
  3. TheTwitt

    TheTwitt

    Joined:
    Apr 6, 2016
    Posts:
    5
    This is a new way to see the technology. It actually make virtual reality enjoyable and impressive. This is a part of standardize experience to the customer's home.
     
  4. Clodo

    Clodo

    Joined:
    Dec 27, 2013
    Posts:
    40
    Hi, i read your comment, i will implement support for Oculus (and an interface that not require Vive controller) as soon as possible. Thanks for the feedback.
     
  5. Clodo

    Clodo

    Joined:
    Dec 27, 2013
    Posts:
    40
    Project updated:
    - Oculus DK2/CV1 support. HTC Vive support can be removed by commenting a #define directives.
    - Support of mouse / keyboard for seated headset (Oculus)
    - Minor improvement and bugfixes

    Topic rebuilded after the forum rollback.

    p.s. i cannot update the title of the topic. Can a moderator change it to
    ? thanks.
     
  6. noelmk

    noelmk

    Joined:
    Jun 18, 2016
    Posts:
    5
    So I tried this in Unity 5.3.5 and the virtual desktop plane is white. I can use the controller as a mouse but I can't see anything in game.

    I'm using Windows and the HTC Vive.
     
  7. abrownbag

    abrownbag

    Joined:
    Jul 17, 2016
    Posts:
    3
    Just wanted to say this is awesome and I've been looking to implement something similar in one of my projects! I sent you a message about it.

    -Thanks again
     
  8. Clodo

    Clodo

    Joined:
    Dec 27, 2013
    Posts:
    40
    Download updated to fix a compilation error if used without SteamVR.

    Sorry, i can't reproduce that. You are sure you use the Unity 64 bit edition with this asset? You don't have any log error?
     
  9. dannox

    dannox

    Joined:
    Oct 20, 2010
    Posts:
    12
    is it possible instead of having the desktop rendered in a texture, to render different application frames opened in desktop, in different textures?
     
  10. jinhaifeng

    jinhaifeng

    Joined:
    May 12, 2016
    Posts:
    6
    Any update on this for 5.4?
     
  11. noelmk

    noelmk

    Joined:
    Jun 18, 2016
    Posts:
    5
    I found the problem. I'm using a laptop and I guess it captures the desktop differently. Any ideas on how to get around this?
     
  12. JasmineJas

    JasmineJas

    Joined:
    Jun 2, 2016
    Posts:
    9
    have a 4970k and an ASUS GeForce GTX 970 STRIX, and I experience zero stuttering in those games. All stock, no special configurations. Edit: No idea how much of a difference it makes but I have 32GB of RAM.
     
  13. abrar-robotics

    abrar-robotics

    Joined:
    Feb 16, 2016
    Posts:
    4
    There is a Error , Can anyone help me.....

    Its occurs when I Import asset there was 3 error 2 was of about //#define VDM_SteamVR and 3 one is this
    Kindly help me I really like your asset


    Assets/VR Desktop Mirror/Scripts/VdmDesktopManager.cs(386,16): error CS1501: No overload for method `IssuePluginEvent' takes `2' arguments
     
  14. abrar-robotics

    abrar-robotics

    Joined:
    Feb 16, 2016
    Posts:
    4
    How to use it Can Anyone help me?
     
  15. phileday

    phileday

    Joined:
    Feb 8, 2014
    Posts:
    89
    @abrar-robotics I'm not the developer but I got it working by importing the package as normal, then importing the SteamVR plugin from the asset store, then deleting the camera and using the prefab steam rig and steamvr into the empty scene. It then worked for me.
     
  16. phileday

    phileday

    Joined:
    Feb 8, 2014
    Posts:
    89
    @Clodo I've sent you a message relating to this plugin. I make a player called Whirligig and I'd like to use this plugin to allow people to mirror their desktop to watch content like youtube etc. I think I've got it working the way I want but still need to implement it and also have some other questions. I've sent you a private message but I thought I'd leave a message here as I can't think of any other way of getting in touch :)

    Great work on this by the way. This should be an Asset store plugin rather than just given away free.
     
  17. Clodo

    Clodo

    Joined:
    Dec 27, 2013
    Posts:
    40
    Hi to all, sorry for the delay.
    I updated the .unitypackage for Unity 5.4.1f1. Download: https://github.com/Clodo76/vr-desktop-mirror/raw/master/Output/VrDesktopMirror.unitypackage
    I released all the source-code (C# Unity and C++ DLL) on GitHub, under LGPL. https://github.com/Clodo76/vr-desktop-mirror
    I added multiscreen support.
    I added some property related to multiscreen and colorspace.

    Previous version works only with Unity 64 bit, the new version have both 32/64 dll.

    Please note:
    - Require at least Windows Vista. DON'T WORK with Windows 7. It use Windows API that are available only from Vista and above, nothing i can do about this.
    - I cannot detect if SteamVR/Vive asset is installed or not. See the first post if anyone have a compilation issue.


    Honestly, the desktop mirror feature maybe, but this Asset aim to be a VR interface for Unity, but the Unity bug described in first post are too much trouble.
     
  18. liching

    liching

    Joined:
    Nov 18, 2016
    Posts:
    2
    How to use the multiscreen function?
    I imported the new package but I have no idea how I can use it
     
  19. liching

    liching

    Joined:
    Nov 18, 2016
    Posts:
    2
    Is it possible to add a function that can not only show the desktop on the unity, it also show the specified window like folder or chrome on the unity? It just like the AR screen from Leap Motion.

    0:35
     
  20. cknowles425

    cknowles425

    Joined:
    Nov 14, 2016
    Posts:
    2
  21. dandoVR

    dandoVR

    Joined:
    Jul 24, 2016
    Posts:
    3
    Is there anyway to render the desktop on a gameobject?
     
  22. brandon2321

    brandon2321

    Joined:
    Nov 16, 2016
    Posts:
    5
  23. cknowles425

    cknowles425

    Joined:
    Nov 14, 2016
    Posts:
    2
    You can modify the scripts in the 'VR Desktop Mirror' prefab so it doesn't re-position/resize when you enable/disable the display.

    VdmDesktop.cs -> if you comment out the code in Update you can probably get closer to what you want to acheive by positioning the prefab where you need it in the scene and playing with the scale in VdmDesktopManager.cs so it fits better.
     
  24. hrforecast

    hrforecast

    Joined:
    Nov 27, 2016
    Posts:
    1
    *Edit

    Found the option by myself, already implemented :)
     
    Last edited: Dec 13, 2016
  25. dandoVR

    dandoVR

    Joined:
    Jul 24, 2016
    Posts:
    3
    When I build the game I get an Access Violation error?

    DesktopCapture.dll caused an Access Violation (0xc0000005)
    in module DesktopCapture.dll at 0023:61ba11c0.
     
  26. Clodo

    Clodo

    Joined:
    Dec 27, 2013
    Posts:
    40
    This is a frequent question.
    No, sorry. My asset use Windows Duplication API, that have the advantage of highest framerate, but don't allow to capture a single window.
     
  27. CEOManDan

    CEOManDan

    Joined:
    Dec 17, 2016
    Posts:
    2
    Please help me here. I'm developing a game and this asset is a core part of it. The game was supposed to be released last week but this keeps crashing in the build. It works perfect in the editor, but after about 30 seconds in the build it crashes, and when I debug it says 'ntdll.dll could not be loaded'.
     
  28. zvidrih

    zvidrih

    Joined:
    Jan 31, 2017
    Posts:
    4
    Hey! Great work! Do you have any idea why inside the editor I only get "white screen", but in built code it works as expected?

    Thanks!
     
  29. Talashowl

    Talashowl

    Joined:
    Apr 25, 2015
    Posts:
    5
    Can I just say that you are an absolute angel for creating this? I had the idea (in the shower, where all the best ideas come to you) to mirror my desktop while working on my game and... Looking up possible tutorials I find this; exactly what I'm looking for. I was wondering if you could mirror your desktop to a game object, I'd love to put a handle on it and carry it with me.

    My vive is on the way and I am very excited to test this out!
     
  30. TimGM

    TimGM

    Joined:
    Mar 28, 2017
    Posts:
    1
    @Clodo I have got this setup and its great. However I can see the colour depth is pretty poor. Blacks are really washed out and it seems like it is 16bit or similar. Any way to get a better colour depth? Is it a quality setting or something. Thanks!
     
  31. Echoshard1

    Echoshard1

    Joined:
    Nov 26, 2013
    Posts:
    2
    I am having the same trouble it works great but crashes at around the 30 second mark.
     
  32. Deleted User

    Deleted User

    Guest

    Could @Clodo give some answers? I really want to use this but i get the error @CEOManDan and @Echoshard1 describe and the issue @dandoVR mentioned. Also when i comment the #define i get a random bool that is not defined somewhere in the code. This works perfectly for what i want to do and i really would like it fixed!
     
  33. emathew

    emathew

    Joined:
    Jul 31, 2017
    Posts:
    13
    Anyone know how to do this for Mac OS??
     
  34. dNazarik

    dNazarik

    Joined:
    Feb 6, 2016
    Posts:
    6
    How can I get a Texture2D from the script? I want to make a screenshot and put it to custom plane.
     
  35. MosesOh

    MosesOh

    Joined:
    May 16, 2017
    Posts:
    2
    Hello, does anyone know if this prefab still works? I tried dragging the prefab onto my project tree and using my mouse and keyboard but all i get is an empty white screen
     
  36. dNazarik

    dNazarik

    Joined:
    Feb 6, 2016
    Posts:
    6
    It works with HTC Vive for me. You can try next:

    Code (CSharp):
    1. // #define VDM_SteamVR
     
  37. MosesOh

    MosesOh

    Joined:
    May 16, 2017
    Posts:
    2
    Im trying this out with windows mixed reality headset, specifically acer. I'm using Holotoolkit with Unity because it allows me to program for the headset and test scenes directly onto the headset. Its strange how this desktop mirror works without the headset but once I plug in my headset I only get a white screen. Maybe it has to do with which graphics card is being used. I heard desktop duplication api doesnt work with Nvidea. However I have to use Nvidea for windows mixed reality headsets
     
  38. tareksalah

    tareksalah

    Joined:
    Jan 28, 2017
    Posts:
    16
    Hi there,
    Please I want to use this with the Oculus Touch .. Would you please guide me through the steps to do so?
    Also, is there a guide on the functions used in the DLL so that I can create further utilities on top of this great asset?
     
  39. karbasi

    karbasi

    Joined:
    May 27, 2016
    Posts:
    2
    Wow , Impressive work! Thanks
    I am a beginner in HTC Vive, and I am trying to do a simple thing that you already have done here:
    I have a embedded webpage ( youtube search page ) inside my unity3D game, and I want to be able to use the HTC Vive controller and click on a certain video to play, I am not sure how to use the controller as a mouse !
    Can you please help me through ?


    Thanks alot


     
  40. adgarcia92

    adgarcia92

    Joined:
    Jan 11, 2018
    Posts:
    1
    did you ever get this issue fixed? I'm having similar problems when building the project and running from the executable.
     
  41. Joe789

    Joe789

    Joined:
    Jan 5, 2017
    Posts:
    2
    How to actually get a capture of a single window? its the core part of my AR project and I can't seem to make it work! Help please
     
  42. TobiasSL

    TobiasSL

    Joined:
    Mar 11, 2019
    Posts:
    2
    Hey. Not sure if this asset is still being worked on, but im trying it out and it works fine on an external monitor but not on my laptops screen. I just get a white screen, but im able to move the mouse around. Has this something to do with it being the integrated graphics card? is there a work around? Best regards.
     
  43. saharaseeri

    saharaseeri

    Joined:
    Jul 1, 2017
    Posts:
    11
    Did you get this work? I got a white screen also!
     
  44. saharaseeri

    saharaseeri

    Joined:
    Jul 1, 2017
    Posts:
    11
    Hello, are you able to fix the white screen? If YES, could you please tell how to do it to fix this problem!
     
  45. EmuBroileri

    EmuBroileri

    Joined:
    Jan 31, 2019
    Posts:
    2
    I get an error saying that the type or namespace name 'SteamVR_TrackedObject' could not be found. And when I try to comment the first line saying #define VDM_SteamVR, I get a lot of errors saying 'VRSettings' is obsolete: 'VRSettings has been moved and renamed. Use UnityEngine.XR.XRSettings instead (UnityUpgradable)'
     
  46. JKR137

    JKR137

    Joined:
    Feb 12, 2020
    Posts:
    1
    me as well! really need help in that! using Unity 2019 with Oculus Rift S
     
  47. romi-fauzi

    romi-fauzi

    Joined:
    Aug 16, 2013
    Posts:
    152
    For all who are having white screen issue, probably because there are 2 graphics cards (usually on a laptop), when I tried on my Desktop which only has one graphics card, it works.

    Then I tried again in my laptop, but make sure to set unity to use the integrated graphics, and it works.

    So perhaps this might help to all who experiencing the white screen issue.
     
    Last edited: Apr 1, 2020
unityunity