[Release] FMETP STREAM: All-in-One GameView + Audio Stream + Remote Desktop Control(UDP/TCP/Web)

We had a couple of interns for a while that used Macs. We're still cleaning up their line endings.

 

I think I'm going to buy this asset but it's borderline, I'm a little nervous.

My main use is for video chat. Agora isn't quite right, users need to enter a session, I'd rather just have a webcam texture with audio, not sessions. Plus Agora is potentially expensive; you have to use their server service in addition to your regular server for other uses.

I looked at just doing it by hand with Mirror and webcam textures + audio, but there's tutorials or anything for that and I'm not a strong enough programmer to go into that with confidence.

FM Stream looks perfect but it could be better. I'd be more confident if there was tutorial for integrating the encoders with Mirror. Also, when I buy an asset I always look for a strong community on Discourse or similar user forums, which FM doesn't seem to have.

Anyway, I'm not writing this to criticize the asset or company, I just though it might be useful to see what a perspective customer is watching for.

 

Thanks for sharing your perspective to us, and we totally agreed your points.
We'll surely find a better way to gather all discussion and questions from all users, which can help each others to troubleshoot immediately, instead of emails.

At the moment, we have a small facebook group for this purpose.
https://www.facebook.com/groups/275354490106676

We sometimes post plugin related topic, including coming features, testing, and bugs discussions. But it's still in a small scale.

 

Hi, i need help for this project. I need use OBS player for streaming live (video and audio) on the unity, for android and WebGL.With FMETP STREAM is posible? thanks

 

I think it's a duplicated question from your email, I tried to re-cap my email's answer here:
FMETP STREAM doesn't support OBS player directly, though you can still capture Win10's desktop into Unity without system sound.

Edited: there are some softwares can redirecting system sound into Unity as Mic Input, then it may work as well.

 

Hi, I want to stream UDP from OBS to Unity for a project to be deployed on the Hololens 2, is this package capable of that?

 

Please be reminded that it’s not a wrapper for OBS.

but if you are streaming motion jpeg in udp without sound, then it might work. As we don’t have h264 stream decoder for this.

 

Hi with this plugin is it possible to stream to a unity app on another PC with IP? Also in addition to stream is it possible to stream other data?
Thank you

 

For Internet Streaming, please refer to this old demo. and you can send byte[] or string in bidirectional.

 

[v1.329] submitted to store
-bug fixed: Panorama Capture mode bug on build.

 

I want to stream the screen of HoloLens2 to Windows10.

https://www.dropbox.com/s/ub54pomjwi6zklv/Test BG Debug.mp4?dl=0
=> test video Link

Windows10(Server)-[Demo_FMNetworkStreaming] Scene
HoloLens2(Client)-[QRCode Demo] Scene

I checked that streaming works well, but there was a big error.

The background, other than UI and controller rendering, remains black.

What settings do I need to set up for real-world cameras to be streamed in the background?

I have confirmed that both MainCamera mode and Full mode are not available.

The streaming background on HoloLens 2 should be displayed as a webcam (real world) background, not black.

Among the things I tested, I used the "[ Enable for Test] WebcamDemo" object in the Demo Scene, but the symptoms were the same.

Is there a way to display the webcam as the background in HoloLens 2?

 

it's a known issue since Microsoft updates.
There are some webcam stream/access issues between Unity3D and UWP.

Some customers implement their own solution to get the webcam back in HL2 with FMETP STREAM.

There was a solution replied by Microsoft, but I didn't have a chance to try it yet.
https://docs.microsoft.com/en-us/wi...and-apis/mixed-reality-capture-for-developers

Another solution is via Vuforia class, which should work, but no guarantee since so many months.
https://stackoverflow.com/questions/58499314/hololens-how-to-get-webcam-texture-2d-from-vuforia

Sorry for any inconvenience caused, and hope this can be fixed in newer version of Unity in future.

Edit: thanks for one of customers(Chris), who shares below solution on HL2:
"unity web texture loaded with a mesh plane. It works as well with latest unity 2019.4.14f"
and you will need to set a separated rendercam with GameViewEncoder: RenderCam Mode.

 

Thank you for the quick reply. This guide gives me hope.

When setting this mode after proceeding with the guide [GameViewEncoder: RenderCam Mode]

https://www.dropbox.com/s/ub54pomjwi6zklv/Test BG Debug.mp4? dl = 0
Basically, the general objects featured in the link above should appear well.

At the same time, it should be streamed using the HoloLens 2's webcam as the background.

Is it possible if I set it according to the guide?

 

You may follow below setup, but we haven't tested further on HL2 webcam yet.
1) create mesh and assign webcam texture manually yourself, and fill it in the far clipping plane distance.
2) create a layer and assign it to your webcam texture mesh, let's say "BackgroundCam"
3) in your HL main cam, please ignore "BackgroundCam" layer
4) create a render cam, and parent it to your main cam, and render everything including "BackgroundCam" layer.
5) add a script "TargetProjectionMatrix" on render cam, and refer to your main cam. This will help you match HL2's projection matrix.

If you succeed, please let us know too

 

Unfortunately, one of the default settings, HoloLens2's Webcam failed to read.

<Demo_FMNetworkStreaming> Demo scene <[ Enable for Test] WebcamDemo> I copied and set the game object configuration as it is, but the following error occurs on HoloLens2.

[Error Log]: Error: Operation has failed with error 0xc00d36e6: The requested attribute was not found.
(C:\buildslave\unity\build\PlatformDependent\MetroPlayer\MetroWebCamTexture.cpp:WebCamTexture::Update at 613)

What could be the reason?

 

For troubleshooting, would you please try Unity's default example for getting webcam texture in unity 2019.4.14f?
https://docs.unity3d.com/ScriptReference/WebCamTexture.Play.html

Our FMETP STREAM webcam example script is checking all available webcams, not sure if this is allowed on HL2.

 

One more awesome experiment.
StereoPi VR Streaming to Quest 2, with FMETP STREAM.
https://twitter.com/StereoPi/status/1334837087049306113

So Excited to share this demo video!

 

Is it possible to stream game view to mobile in non local network?

 

We have FMWebSocket demo for Internet Streaming, which should also work for public network without internet.

Other alternative networking systems like Photon, Mirror and UNet..etc should still be compatible via their bytes streaming feature.

 

I mean i wanted to stream game view implemented from my PC to Mobile of external network without port forward.
it is possible?

 

In this case, you will have to host your node.js server via other cloud service provider. like Amazon..etc.

 

um...
If I use port forward, is it the same?
Should I host a server like Amazon?

 

In general, you have to connect to a node.js server for communication between any network.
The node.js server can be hosted locally with port forwarding, or a cloud server with Internet.

It depends on your project scale and budget, either of above solution will work for your case.

 

I'm thinking about buying this asset, but first I need to know if this asset has the capability I'm looking for.
I need to display in my app a live camera feed (live highway, street, and shopping mall cameras).
My setup consists of an Android/iOS Unity app, NodeJS server, and a Raspberry Pi hooked to a camera, where the app should display the live video stream captured by the RPi camera.

1. Can this be done with this asset?
2. Do you have any code examples for achieving this goal?
3. Can UI elements can be displayed on top of the live video stream?

 

Thanks for reply.
You were a big help to me.

 

Hi, i just saw your product on Unity Asset store, and I want to buy, but first have question...

While in my Unity Game, can i stream live from the phones (Android/IOS) CAMERA AND MICROPHONE to a server, and then others (WebBrowser/Android/ios app) can connect to the server to view the live video and audio?

Also I need it to work with multiple streams, so many different people can be streaming to different set of people at the same time. Meaning person A can be streaming to person B,C,D,E,F while at the same time person X is streaming to persons J K L M. So people should be able to select which stream to watch based on the URL or something like that.

Basically a webinar type setup where one person streams their phone camera and mic, and others can listen. But many streams from different people coming in at the same time, going out to different people as well as described above.

Does it use HLS, or DASH, RTMP, or some other streaming LIVE streaming protocol?

Thanks

 

FMETP STREAM allows you to stream Game Views between Unity apps.
For your case, it seems likely you are looking for some IP cam solution instead.

We did some experiment for StereoPi for VR streaming, but I have to wait for their official announcement before telling you script details.

The good news is, we have MJPEG fast decoder in the package. As long as you have mjpeg stream from RPi to Unity, our decoder can help you do the rest.

Once you receive live video within Unity, we have a "Full Screen" capture mode, which will capture with UI canvas. Then, you can stream the captured result to your node.js server via socket.io with ease.

 

You can stream your webcam & mic to others with our asset.

But, according to your large scale streaming need, our local hosting FMWebSocket solution might have performance impact.

Photon with FMETP STREAM might be an alternative, as they have GameRoom system with UDP protocol option, which should fit your use case. It's been tested by other customers, though we didn't include Photon demo scene.

We are still working in progress for standard live streaming platform support in future, but still need more time and some hard work.

 

Hello I am using this asset well.

In the current demo scene, we succeeded in receiving a string value using <Receiver> in FMNetworkManager.cs.

However, I want to call the same function from the server to the client, not to receive the string value when the UI button is pressed.

How can I implement it?

 

We provide option of SendToAll, Server, Others.
Could you please specify your issue?

I assumed that "SentToOthers()" would be your choice, if you don't want to receive the string sent from yourself.

 

Hello I am trying to use microphone streaming.

The volume of the mic is very low. Is there a way to increase the volume of the mic on the cord?

(This is a situation where you can no longer turn up the volume on the device.)

 

We don't have this option in current version.
But since our captured data is in PCM raw data, you may hack our MicEncoder a bit, by multiplying the value of raw data.

 

Can I sync the Win10 Monitor to HoloLens2？
I need to share my desktop on Hololens2 in real time. All the methods I have found are about synchronizing the HoloLens2 screen to the PC side instead of the PC side to HoloLens2.

 

Yes, it should be compatible.

 

Thank you,I will try it

 

Hello! I am trying to stream a video from a webcam into Oculus Quest 2. I've bought the package but I'm not sure from where to start now. I am guessing: create a UDP client in Unity (which will be deployed to Oculus) and a UDP server from the PC with the webcam. Is this correct? If so, is there any example code for the UDP server that I need to implement in the PC?
Thanks!

 

For beginning, you can first try our demo scene "Demo_FMNetworkStreaming".
The core components are FMNetworkManager, and GameViewEncoder/Decoder.
GameViewEncoder -> bytes -> FMNetworkManager -> bytes -> GameViewDecoder

you may also refer to this AR tutorial, similar setup should also work on VR headset.

 

I will try, thank you! And thank you very much for the super fast reply!! Happy New Year!

 

Happy New Year! We just want to make sure customers saved their time, by rapid reply

 

Just bought this today, and can't get the demos to work consistently. The echo test works fine. But the Demo_WebSocketNetwork scene claims that it can connect, whether to my localhost running the NodeJS example, or to my Amazon EC2 instance running the same - but neither socket server shows any console confirmation of a connection, and even with multiple applications claiming to be connected, there is no evidence that any information is being transmitted. Did I miss something obvious?

 

I guess it's a known issue of socket.io version, and here is a temporary fix.
Please type this command in your terminal, for reinstalling the correct version.
npm install socket.io@2.3

hope this can solve your problem.

 

Hi, I just purchased this asset and I can happily say it works really well. I'm just currently having trouble with Websockets on WebGL mobile and getting the following error:
ReferenceError: io is not defined

It runs perfectly on desktop browsers and standalone though. Did I possibly miss a step or something?
Thank you in advance.

 

It seems likely the wrong IP input, please check if you are testing with localhost on your desktop browser? If yes, you should change it to your node.js server IP.

If your mobile is on public network, please try forwarding your port to your node.js server IP.

 

Please tell me how to reconnect if connection is lost during UDP connection.

The configuration is as follows:
Server: UWP (HoloLens2)
Client 1: UWP (HoloLens2)
Client 2: PC (Windows 10)

 

Hello Again!

I've made progress based on your previous explanation. I have 2 unity projects, one handling the webcam (sending into a RawImage) and the other one on Oculus. When I run both on my computer, the image from the webcam is transferred between projects, however, after deploying the app into the Oculus (Oculus Quest 2), the image is not shown. Is there any permissions or flags I should set in the manifest to allow the connection of the FMNetwork package? If that is not it, what else could be the cause of this problem? Thanks in advance!

 

In general, you can simply disable and re-enable the GameObject with FMNetworkManager.cs.
It will automatically stop and clear all previous connection, and try a new connection.

If you have some unknown errors related to connection, please check if you have duplicated Label ID for Encoder/Decoder between those devices. Each Label ID should work as a pair, if there are more than one pair, the received data will be interfere though it's still connected.

 

If you are working on PC, please don't forget to disable the firewall as possible. and please try enabling the network discovery feature in your Win10 OS, which should be under networking settings in OS.

This is the most common situation, and solution.

 

Thank you for quick response!

I want to connect two platforms of HoloLens2/PC (Windows 10) in the same Wi-Fi environment. How do I do it?
(When I tested it on Android and PC, it was well connected.)

 

below is my suggested setup:
FMNetwork Server: PC
FMNetwork Client: HL2

some tricks, it might help:
-Do not connect your USB cable to your PC when running the app
-Try switching on/off "UseMainThreadSender" on all platform
-Try remove all "AudioEncoder/MicEncoder", see if it's related to network traffic..etc

For technical support, I recommended that you can write us an email: thelghome@gmail.com

 

Solved it, thank you for the suggestion. My google app engine server was unreachable due to a DNS issue with my ISP provider. (Still very weird it only happens on mobile). The error was:
DNS_PROBE_FINISHED_NXDOMAIN

Got it to work by manually setting a private DNS (dns.google) in Android connection settings, and (8.8.8.8) in iOS
Now everything works in all platforms - WebGL (Desktop/Mobile), iOS and Android

Thank you for such an amazing asset!