[Release] FMETP STREAM: All-in-One GameView + Audio Stream + Remote Desktop Control(UDP/TCP/Web)

In a native iOS app I can stream video to a Wowza, Adobe, and Unreal Media server using .264/RTMP video stream. I was hoping to be able to this from a Unity app for iOS & Android. So what I would like to stream a real phone camera. The other use case would be to stream AR video. It seems you have VR case handle.

 

RTMP protocol is not supported in current version. But you can still stream your camera or phone webcam to your own website via our WebSocket HTML demo.

 

[version 1.120] updated
- Game View Capture Mode: RenderCam, MainCam, Full Screen(with UI Canvas)
- Improved Inspector Layout

 

As many customers asked for Node.js Setup with FMWebSocket, we made a step-by-step tutorial.
It's a setup on Mac, but the commands are also compatible on Windows10.

 

Hi,
I am currently considering to buy your asset.
I understand, that you are using the Texture2D.EncodeToJPG() function. The strange thing is that if I am running this function on my gaming notebook with an 640*480 image, it takes rougly 50ms to run this function once.
Your Oculus Go though only took 5 ms (as you mentioned on a previous post) so I wonder if you have found a way to optimize the call of "EncodeToJPG()" beside simply running it async (since putting Unity Functions into another Thread is not possible).
I am only in need to optimize encoding raw images to JPG, so this asset would only make sense for my problem if you improved the usage of the EncodeToJPG function.
Thanks in advance!

 

It’s real pain that Unity3D doesn’t provide a better jpg encoder option. In fact, we are in a very similar situations.

We’ve tried our best to optimise the whole process(capture texture & encoder) with some trial and error testings. It improved a bit in different combination of priority.

In current version, we still focus on Unity3D default function in order to make it compatible on all platforms.

There are some native async jpg encoder solution, which will improve the performance too in limited platforms. You should do some research about this topic if you want a faster encoder.

From my experience, another option is streaming raw pixels directly between devices. We did some experiments on Macbook & PC in LAN, but the raw data traffic is monster for mobile.

We’ve tried many methods to balance the processing speed, streaming traffic, and compatibility.

My little advice:
If you are targeting PC and Android, you may try some experiments related to async jpg encoder solution first. As far as I know, there are some native or paid solutions on github or store.

 

Thanks a lot for your honest reply. I really appreciate it.
I agree with you that default functions are to be prefered.
Before going on with the EncodeToJPG I will run some tests with openCV and check the performance difference.
Thanks again.

 

Hello, I use "Demo WebSocket Streaming" and I have a latency with the microphone which increases with time. How can I improve it?

 

If your delay keep increasing for running a certain period, please try reducing your sample rate and set the channel as mono.

In general, there is ~0.5 second playback delay generated in Unity3D audio playback system. This requires native audio playback solution for your build platform, which is not included in our package.

If you have further questions, could you please send us an email for technical support? Thanks.
thelghome@gmail.com

 

Hi - I need to stream an Unity screen image to a web client. Can this be done using this product?

 

Yes, you can stream image/video to a web client/browser. You may check this Web Browser Live Streaming Demo.

 

Great - can this be done from an iOS device to a web client/browser?

 

Yes, it's compatible. It's a cross platform solution on iOS/Android/Mac/PC/Linux/VR/AR...etc.

 

Sold! Thanks very much for your quick response.

 

I'm confused as to how to make this work from a running iOS client to a web client. I think the iOS end is failing, because the local index.js (socket.io) serves okay, and I see 'user connected' when I start it.

The iOS screen is on another IP address though. How do I modify the scripts to have that example work?

Thanks.

 

The demo index.js is an example of Server-Clients structure.
Unity3D(init as Server) -> Node.js Server(index.js) -> Unity3D(init as Client) or HTML demo

In this logic, you have to init Unity3D(Server) too. Otherwise, the data will be filtered by our index.js demo script.

Meanwhile, you can also override our default Server-Clients structure with your customised logic.
To do this, you need to create a event listener via FMSocketIOManager.instance.On("event name", data);
you can also override the emit via FMSocketIOManager.instance.Emit("event name", data);

Hope it give you idea for troubleshooting, and please feel free for write us an email for technical support too.
email: thelghome@gmail.com

 

Hello I bought your product, and trying to compile it i get this error, can someone help please?

To install it i just dragged the folder into my project, i didnt import it from the package directly, and i didnt import anything from the project settings folder from the package because i didnt want to overwrite my project settings with the one from your package. I dont know if thats causing it. but heres the error:

Assets\FM_ExhibitionToolPack\FMWebSocket\Scripts\WebsocketSharp\WebSocket.cs(367,23): error CS0121: The call is ambiguous between the following methods or properties: 'WebSocketSharp.Ext.IsNullOrEmpty(string)' and 'WebSocketSharp.Ext.IsNullOrEmpty(string)'

Assets\FM_ExhibitionToolPack\FMWebSocket\Scripts\WebsocketSharp\HandshakeRequest.cs(102,39): error CS0121: The call is ambiguous between the following methods or properties: 'WebSocketSharp.Ext.Contains(System.Collections.Specialized.NameValueCollection, string, string)' and 'WebSocketSharp.Ext.Contains(System.Collections.Specialized.NameValueCollection, string, string)'


.....Many more similar, cant compile....

Seems you made extension to several classes and that clashed with something else?

 

Do you include other networking features like photon in your original project? They may import the WebSocket library already. In this case, you can simply remove the WebSocket folder which cause error, as it’s duplicated in your project.

 

You may remove this one with ease, as it’s duplicated in your project.
Assets\FM_ExhibitionToolPack\FMWebSocket\Scripts\WebsocketSharp

 

I removed the one from Plugins and it builds now. Thanks

 

Hello,

I bought this product, i want to use it to stream live video from a worldspace RenderTexture or gameObject surface, to Linux Ubuntu server. Then other clients can connect to that server and watch the scream on a texture in 3D worldspace. I would like to stream the audio too.

Can someone give me the steps to get that working please?

Thanks so much!

 

please refer to this Youtube Tutorial(you could ignore the AR setup).

The AudioEncoder works similar to GameViewEncoder, which you can check out our FMNetworkStream demo scene.

If you have specific setup, please contact us for technical support via email.
Support email: thelghome@gmail.com

 

Hi,

We are having good success with your assets - thank you! Do you have an example like the 'TestServer' index.html that shows how to capture and send audio data from javascript?

Here is the set up:

1) Unity server app sends screen image and audio to the server;
2) HTML app sends audio image back to unity app.

 

Glad to know that it works for you!
It's possible, but we don't have exact example of your requested feature.
The easiest way would be another WebGL app instead of pure HTML & Javascript.

If you are looking for a pure javascript solution:
step1, you can refer to our GameViewEncoder & AudioEncoder/MicEncoder for data structure.
step2, you need to implement your own javascript solution of getting correct PCM audio data and jpeg data in HTML.
step3, refer to our index.js example for matching the event name of socket emit().

Hope the suggested solution make sense to you.

 

HI,

I bought your product few days ago, i just have time to use it now, this is what i need it to do...

1. From my mobile Unity app, send live stream video/audio to server
2. Several other unity apps can connect to server and watch and listen to audio/video stream

This is what i need, can your product do that?

Thanks

 

Thanks for your purchase.
Yes, it's possible.
You can start with FMWebSocket Streaming demo, which I assumed that you may need Internet connection.
Or please feel free to email us for technical support: thelghome@gmail.com

 

I sent you an email to that address can you check it please? thanks

 

sure, replied via email. Hope it give you direction

 

We did some testing by replacing the default Unity3D encode jpg method, which you can implement on your side too.
This demo shows you the possibility of getting 60+FPS in Oculus GO with this setting, as proof of concept.
The testing network environment is using a 4 years old low-end Router TPLink AC750.

 

[Coming in version v1.130]
[New!] Fast Mode: Improved Encode/Decode Performance on Windows 10 & Mac OSX
100+% performance boosted with native solution
Supported platform in this version: Win10 & Mac OSX

 

Hi, could this be used to live stream camera and sound from one computer to many webgl clients over the internet?

 

Yes, it's possible. You may find more demo / tutorial videos on our Youtube Channels.
https://www.youtube.com/channel/UC1ADZ8xSCKDDizmStC5vkow

 

[v1.140] submitted
[ Import notes: *before update to v1.140, please Backup your project, remove the root folder of our plugin. ]
The fast encode/decode supports iOS now! Let'e see how it works on iPhone 5, 720P(24+fps) Live Streaming!

 

Does this streaming support controls in the streaming device? for example controlling a unity game from browser

 

This is a very common question and the answer is Yes.
Although our html web demo is a passive viewer, you can still create your own JavaScript button for sending command and trigger events as controller.

You may refer to some basic usage of socket.io, and it won’t be difficult to implement on JavaScript & HTML yourself.

 

[v1.200] Solved the bottleneck issue on Android platform, with "Fast Mode" support now.
Meanwhile, we do another VR device test. It's a Vive Focus with Hand Tracking SDK from official htc demo. Personally, I do not recommend Vive Focus, it's too heavy and not comfortable. Quest should be better option as Android VR Headset.

 

I want to stream the actual desktop of a phone or Mac / PC even while running in the background. Such as, Google Meet Present View does. Is there a way to support that or a suggestion for me to go hack it in?

 

Usually you need to work with some native capture library for device desktop.
It's not an easy task for phone or Mac, but Windows 10 has its own library for doing this.

https://twitter.com/FrozenMistDev/status/1255466954992713729
In our latest project, we successfully stream Win10 Desktop into a 360 projection system which are driven by multiple old PC.

Although this feature is not included in current version(v1.200), we may still consider adding a simple Win10 Desktop streaming feature in near future. It's already working in progress with successful projects, and just need some time to simplify it with nice inspector for Unity3D.

 

I'm pretty good at hacking in native code. Maybe something like this or this or this? I need to pass iOS screen displays to other devices. And then there's all of this HLS from Apple.

Where would I pass the captured image(s) to hook into the streaming of FMETP?

 

Once you can capture the screen and display it in Unity3D, FMETP STREAM will do the rest for you.
You can use our default RenderCam setup without coding, or you may clone the logic in GameViewEncoder & create your own mapper for streaming.

And glad to know that you are good at hacking in native code, awesome man! I'd like to know your progress too.

Just some tips for your native capture part: try to get the raw capture data and pass them into Unity3D, instead of passing an complete image data. But all your reference links are good starting point.

support email: thelghome@gmail.com

 

[v1.210] update submitted
- GZip Mode: optimised network traffic (reduce up-to 50% file size, depends on captured content)
- Tested with HoloLens 2 x HTML 5 Web browser

 

Hey @thelghome I'm looking for a plugin to help with some webstreaming I'm doing.

I'm streaming live video from a non-unity application on my local network via a webcam. Would your plugin allow me to view this stream in Unity over a network? I wanted to use Unity's VideoPlayer to do this, but it needs a URL rather than an IP address and a Port number!

 

Thanks for your interest. It's a YES or NO answer, which depends on your source format and setup.
FMETP STREAM is designed for customised Unity3D GameView Streaming in LAN or Internet. If your external application can match our meta data structure, it may work.
You could send us an email for further technical advice/discussion.
support: thelghome@gmail.com

 

Cheers, I'll send you an email now!

 

Hi, updated to latest version which has the 'Fast' mode. (Which makes me wonder if you are gonna make a 'slow' mode too )
Application goal: streaming the view from the Quest to a PC unity build.
Enabling/disabling the the "Game View Encoder" component still gives a difference of 15-20 FPS (52vs70) in the Quest! Naturally, that is way too much, any tips on how to reduce this?

I'm using the Game View Encoder as component ON the OVR 'CenterEyeAnchor', capturing 'main cam', resized 'quarter', quality '10', streamFPS '5'.
And yes, I have the 'fast' mode enabled on Quest and PC side.

Thanks

 

Hi there,

I saw that a client can share a view to the server and vice versa. I assume that even several clients can share their views at the same time to the server. However, I want to know if a client can see the shared views of the other clients through the server's view.

The other question is related to the interaction of clients with the streamed content. Is this possible right now with your plugin? If not, can this be achieved easily with some small additions?
I figured out that the answer to this question is Yes. I think I just need to define the possible interactions for each platform to make it work.

How many users can concurrently start the streaming?

Thank you in advance for your reply.

 

Just recap our email conversation for everyone who are interested, here is the trick.
In GameViewEncoder Settings: switching from "MainCamMode" to "RenderCamMode"
52fps -> 66fps

Fast Mode off: it uses default Unity3D function for Encoder.
Fast Mode on: it uses native method for Encoder.

 

Sorry for the late reply, as I didn't receive the forum notification.

It depends on few things.
You can theoretically have unlimited number of clients with their own views, by defining "Label ID" in your encoder/decoder.
But in reality, there are some limit like bandwidth, and processing power.

With FMNetworkUDP for local streaming, you can either interact with other game view sharing by labelling them.
or, you can just stream your view to a target IP address, which can save lots of bandwidth.

With FMWebSocket for Internet streaming, you may be aware the bandwidth of server when all the data will stream to your server first. As Target IP streaming only support FMNetworkUDP in current version.

If you have other urgent question, you could always drop me a message on twitter or email us directly.

 

[ver1.302] submitted
You can now share your Windows 10 Desktop to any platforms!
This is one of example: Win10 Desktop to HTML 5

 

Hi. I have a problem . After installing your asset, my unity is not working. I mean I can launch 2-3 times, then is is freezing and does nothing (need to stop unity with ctl-alt-del to restart it).