How to get the raw camera stream data from the Hololens?

I would like to get the raw pixels from the camera of the hololens 2. I also want that this data contains the real world pixels and the rendered 3D content.

I know there is this class to get the raw data of a single frame:
https://docs.unity3d.com/2019.3/Documentation/ScriptReference/Windows.WebCam.PhotoCapture.html

This class here can enable you to grab a video directly into a file:
https://docs.unity3d.com/2019.3/Documentation/ScriptReference/Windows.WebCam.VideoCapture.html

The problem with VideoCapture is that you can not get the raw pixels

I would like to know if there is a solution to this?

Also is there any difference for solutions for Unity 2019.3 and 2020?

What I want to achive:
I would like to stream the real word video stream with the rendered 3d content over the network.
I want to use a propritary protocol.
It should be low latency and about 30FPS.

 

Hey,

You're correct, the VideoCapture and PhotoCapture APIs do not provide the actual frame data. Instead you'll want to use the WebCamTexture API, which use Media Foundation to capture frames from the video stream and writes them to a Texture. However, I'm fairly certain this does not include the rendered "holographic" content.

To get the full MRC video you'll probably need to implement the WinRT Media Capture API yourself and enable the MixedRealityCaptureVieoEffects. This is basically what Unity's VideoCapture API does.

Finally, have you checked if Microsoft's Mixed Reality Toolkit (MRTK) includes this functionality? Or maybe there's something on the Asset Store?

 

did you find the solution?