Video Streaming content from a remote instance to an Android app

I am working on a project which involves streaming the content of a Unity instance running on a desktop as a real time video to another Unity instance running on an Android device (specifically, an Oculus GO).

My first naive attempt was based on grabbing screens and relaying them to the device through TCP, but I soon realized I had hit the first block: even if I could send a lot of packets through the network, the Android device had to lock up for (too many) milliseconds in order to push the bytes into GPU space with Texture2D.Apply(). This results in a cap of 45~ frames per second, hardly reached with too many tradeoffs such as resolution and image quality.
Another side-effect of this solution was that I was wasting too much bandwith, since I was sending full pictures (basic I-frames all the way), which prompted me to search for any kind of codec that might be used in order to lower the amount of bytes pushed through the network.

My next attempt involved using FFMpeg externally with keijiro's FrameCapturer, since it's also a good way to sidestep the GPL restrictions (more on this later); I've managed to get a video stream out which I could be played in VLC and I thought I was set, but then I was stumped again on how to play that video stream inside the Unity/Android app: VideoPlayer doesn't work, since seems like it only works on http/https URLs and I don't even know if it even works for "on the fly" generated videos.
In addition, the issue about pushing frames on the screen still stands though because even if I could manage to zip the stream to the maximum I would still have to deal with the lag induced by Apply(), so I tried to see if I could access the hardware-accelerated decoder on the device but I don't even know from where I can begin, all the libraries and codec I've found work either on Windows or there's some license-sidestepping shenanigans that I can't use on this system.

Things I have tried that don't fit my use case / don't work:

• AVPro, OfflineRenderer, FrameRecorder, Unity Recorder, etc... : I don't need to record a video, I need to generate a video from a Unity instance and stream it as it's getting captured in real time to another Unity instance over the network.
• FFMpeg (through a modified FrameCapturer): it actually creates a stream in real time, but I have/know no way to play it back in a Unity app.
• Using libvpx/openh264: They can be compiled to run on Windows, but as libraries they are not available through C#/Unity on Android (and I read that libvpx conflicts with the one included statically in Unity).
• Unity's own MediaEncoder class: the scripting manual and the API is lackluster and not sufficiently explained, assumes it only works in the domain on traditional video streaming with the VideoPlayer component.

Things I might try, given enough time:

• Writing a native Android full-screen app (PITA since it must work on Oculus GO) that uses native library calls for hardware-accelerated decoding of video streams.
• Write my own barebones GOP decoder (still doesn't solve Apply() though).
• S̶w̶i̶t̶c̶h̶ ̶t̶o̶ ̶U̶n̶r̶e̶a̶l̶ ̶a̶n̶d̶ ̶h̶o̶p̶e̶ ̶i̶t̶'̶s̶ ̶p̶o̶s̶s̶i̶b̶l̶e̶ ̶t̶o̶ ̶d̶e̶c̶o̶u̶p̶l̶e̶ ̶t̶h̶e̶ ̶h̶t̶t̶p̶ ̶s̶t̶r̶e̶a̶m̶ ̶f̶r̶o̶m̶ ̶t̶h̶e̶ ̶p̶l̶a̶y̶e̶r̶ ̶w̶e̶b̶p̶a̶g̶e̶.

Any help would be greatly appreciated!

PS: About licensing, the issue is that this is not an open-source project and I can't infect it with a viral license, calling FFMpeg as an executable file on Windows works but I can't do the same on Android/Oculus GO.

 

Ideas:
Use Spout - http://spout.zeal.co
and/or
Syphon http://syphon.v002.info
https://github.com/keijiro/KlakSyphon

 

Thanks, they are pretty interesting, but they don't behave nicely on Android.

 

@Tibor0991 Hi there! Did you end up finding a good solution to your problem?