Stream Desktop to in-game object

Hi

I'm relatively new to Unity, but I'm an experienced C# developer. I'm trying to render a stream of the Desktop onto a screen within the game. I've found the following posts :

http://forum.unity3d.com/threads/so...dow-with-opaque-contents-lwa_colorkey.323057/

http://answers.unity3d.com/questions/869378/viewing-desktop-in-scene.html

but as they're not quite what I'm after, I'm having a problem putting the concepts together. From what I can work out, I'd need to use Win API's to grab the Desktop, then somehow assign that to a material, and then update it using OnRenderImage()? Anyone got any advice, or done something similar?

Thanks

 

Hi,

Using those links, made a quick test,
it grabs screenshot and saves as bitmap (using System.Drawing.Graphics.FromImage())
https://github.com/UnityCommunity/UnityLibrary/blob/master/Scripts/Helpers/Screenshot/GrabDesktop.cs

See this for more info about using those system DLL's with unity
http://answers.unity3d.com/answers/253571/view.html

Not yet sure how to convert the bitmap into unity texture2D (then you could use it as usual texture on any material)
And how to take screenshot of desktop (not your app which is at front..)

 

Thanks mgear - that's interesting

I could maybe combine that with this other link I found which assigns a screenshot to a RenderTexture :

http://unity.grogansoft.com/in-game-security-camera-using-render-texture/

I think I may have to try Win32 APIs to maybe grab specific Desktop Windows (rather than the app which is at the front as you say). My concern is how quick it will be as it is generating a PNG each time - I'll give it a go!

 

If you can convert that screenshot Bitmap into bytearray, then can create Texture2D from that, should be fast enough..

https://docs.unity3d.com/ScriptReference/Texture2D.html

 

I'm having a bit of a problem working out how to update a Render Texture from a image/Texture2D. Most examples I've seen go the other way

 

Not sure where you need rendertexture, if just want to display it on a mesh? (as an material maintexture)

 

So I would just create a Texture2D from the bytearray, and then assign it like this? :

MyObject.GetComponent<Renderer>().material.mainTexture = MyDesktopTexture;

and do this in the Update() method of the mesh?

 

OK - I managed to get this working but it seriously crunches my PC. I guess because every frame it is creating a bitmap, does CopyFromScreen(), converts to a PNG byte array and loads into a Texture2D.

Think I'm going to have to try and optimize

 

The PNG conversion is probably where you're wasting cycles. You're compressing and decompressing once per copied frame, and if the system can't compress faster than you can read you have a bottleneck right there. And if you're basically doing a double copy, you need to find a way to cut down that step too

Perhaps you need to look outside Unity's APIs? Here's one thing I found on the subject:
https://msdn.microsoft.com/en-us/library/windows/desktop/hh404487(v=vs.85).aspx

 

Could also place the capture loop in coroutine and add delay to make it slower than update..
Or even separate thread might work.. then use the byte array from that thread.

 

ya, you'll want to do your image processing in an IEnumerator as Update() will run way to many times per frame...

unfortunately, there is only so much performance you're going to be able to squeeze out of it... the BMP to buffer conversion is going to be slow... not sure what decompressor it would be running on it. As an example, I was loading 2K JPG image on iPad and at best, the decompression takes 300ms... then it has to be pushed to the buffer.

 

There are still a lot out there trying to get Windows or other programs (Browser, Notepad, etc) to run inside an object in Unity during runtime. There are many approaches, although most seem to involve long, processor-intensive methods. For example, many use the screenshot method where the app takes constant screen shots of the web-browser and 'feeds' it to a texture in Unity. I have gotten emulators to work in Unity, but a Windows 7/10 emulator would be super-slow and complicated (however Windows 10 Mobile Edition might work). You could make a simple C# or java Web Browser, I did one for a hand-held system before, basically it 'reads' the *.html page at the web address and looks for HREF links, IMG pictures and text and displays them. Then use www in Unity to download the pics, and put the text into a GUI object. (This is basically making an app that runs inside Unity) I've seen a few people trying to get NotePad or WinCalc to run on a Unity object, but it might be easier to write your own text editing app/calc that runs inside Unity. There's currently(2017) an asset called Awesomium, although it is dependant on the C++ Awesomium SDK, really complicated to set up, and it's newest version is not free. However, it is supposed to render a real-time browser window onto an render-texture object. I haven't tried it yet so I don't know what browser it uses but...

 

That is the best way I found: Clean, easy and fast.
https://github.com/hecomi/uDesktopDuplication

 

Your are a god! Thankz!