[Release] FMETP STREAM: All-in-One GameView + Audio Stream + Remote Desktop Control(UDP/TCP/Web)

- 1 server (Tab S8/Android)
- 20 clients (Oculus2/Android)

I just applied the SendToOthersReliable() function in the above environment, and the result was communication with only 1 out of 20 clients.

Is this 1:1 only?

If you debug the string value of other clients that did not receive it, "???" is output.


FMNetworkManager.cs in Client and Server

Should I enable the 'UserMainThreadSender' option?

 

As a quick question generally what gives the best image quality at 60fps? Network or websocket? And is there a tutorial on setting it up a camera in unity from start to finish with it

 

The reliable option is still in experiment, ideally is one to one confirmation. I am afraid that it may not work with one to many, just in case one of clients was disconnected during the whole loop, it won't work then.

We suggested that you may implement similar method on a higher level. For example, you may adding a sequence id in front of you important message, buffer it in the first send. Only remove that buffer message when all the clients confirmed "received".

It might take some time to implement.. but that's also one of disadvantage with UDP.

 

Performance with FMNetwork 2.0 and FMWebSocket 2.0 are very similar. Only different is UDP and TCP(WebSocket).
For new user, we recommend below videos for different networking systems:

 

Does this patch fix the issue when checking "ShowLog"?
When I unchecked ShowLog, no such error actually occurred.

However, if you still communicate with String via UDP, all String communication will be paralyzed after about 100 communications. However, the connection between Server and Client continues to work well.

So if you restart the server, communication works fine again.

The above problem is not caused by an error, but continues to occur separately from the error.

Is there any other solution besides resetting the server?
Or is there a reason you can guess?

 

The latest update should fix the "ShowLog" related error.
Do you have an example scene for us to reproduce this issue?
Meanwhile, if you stream byte[] instead of string, will it work?
100 communications means 100 string message to clients, or do you mean 100 clients? and what's the average length of each string message you sent?...etc
I might need further info to reproduce the bug you mentioned. If you have a minimal demo scene, we can investigate it further.

 

[News!] Remote Desktop Control for Mac and PC
Our Work-In-Progress feature will arrive soon, as a new AddOns template.
Here is the latest test on Quest 2.

Please let us know your thoughts

 

hello @thelghome I just bought 2.0 asset. However I see a multiple assembly error on the console when import the asset. I use Pun2 and want to build my app on Web. Thus, when I delete Pun2 Websocket file, the app does not work on Webgl because of the lacking of the Pun2 socket. If ı delete your file then your asset won't work. What am I supposed to do now? Please suggest me a solution.

 

For PUN2 users, you may remove either our websocket-sharp lib, or the complete FMWebSocket folder(if not needed).
FYI: https://frozenmist.com/docs/apis/fmetp-stream/troubleshooting/

The error is very straight forward. It is simply caused by the duplicated websocket-sharp library, which is a very common library used by other plugins too.

 

hello again @thelghome , I am getting the error that I have mentioned above. I deleted FMWebSocket folder completely. I use pun2 networking system and I have done everything same in your tutorial. Then built my scenes. You can see the errors on the console for both server and receiver scenes. My app's gonna work on WebGL. Thus, I should not use FMUDPNetworkmanager. I have to use FMWebsocket instead, I guess. But you told me that I can delete all FMWebsocket file. I tried to change protocol to TCP and WebSocket in photonserversettings but nothing changed. Could you please suggest me a solution ? Thanks.

 

If you have WebGL scene build, please make sure that there is no FMNetworkManager system in your WebGL scene.
If there is any udp related function calling in WebGL, it will cause errors as expected.

 

I see but how to send my app datas to clients? I need a FMmanager to send my datas from my app right?

 

Even your demo scene does not work. Received result is empty ? Did not change anything and does not work. Weird asset. I also tried everything on your tutorial videos, nothing works in this asset and cannot find any solution anywhere.

 

You have PUN2 in your project, why don’t use PUN2 network system for data stream?
There are many ways to stream data via PUN2, this is just one of examples verified by customers: https://frozenmist.com/docs/apis/fmetp-stream/pun2-example/

In your last post’s screenshot, there is no connected client.
In general, you may disable firewall for testing, as it may block the UDP stream with unregistered app by default.

 

Hi,

I am after a solution that would enable to stream a 360 webcam stream (or video) from a PC to many Oculus Headsets using webRTC to achieve best performances. Is this plugging capable of helping with that ?

 

In our package, the Encoder and Network systems are two modules.
The encoder will help you convert the GameView or Desktop into byte[] stream.
Our package includes demos for two networking systems, FMNetwork(UDP) and FMWebSocket.

It’s also compatible with other networking systems with byte[] stream support, like PUN2, UNet, Mirror or your own networking solution.

If you have WebRTC setup in your project, you may stream our encoded byte[] through the network.

 

[New Feature] FM Remote Desktop Add-on
-Desktop Capture(Mac/PC)
-Remote Control(All Unity platforms)
Asset Store link: https://assetstore.unity.com/packages/slug/242857

 

Is there anyway i can use the package on previous unity versions like 2019？

 

Yes, it should be compatible with Unity 2018 LTS or above.
You may ignore the import warning if you are using older version.

 

I am actually trying to apply the package to a Vuforia AR project using 2019.1.10f1. And every time after I import the package, it will automatically stop all the part of Vuforia. I'm not really sure what is going on.

 

And this is one of the problems i got
Assets\FMETP_STREAM\FMWebSocket\Scripts\SocketIO\FM_IO\Decoder.cs(43,25): error CS0433: The type 'MessageEventArgs' exists in both 'websocket-sharp, Version=1.0.2.59611, Culture=neutral, PublicKeyToken=5660b08a1845a91e' and 'websocket-sharp, Version=1.0.2.39901, Culture=neutral, PublicKeyToken=5660b08a1845a91e'

 

websocket-sharp.dll is a very common library, you may remove the duplicated one.

 

I'm trying to build an Android App with FMETP, however, when I try to build this, it gives me some errors. One is
Assets/FMETP_STREAM/FMCore/Plugins/FMTurboJpeg/Android/arm64-v8a/libturbojpeg.so () and Assets/FMETP_STREAM/FMCore/Plugins/FMTurboJpeg/Linux/libturbojpeg.so (). Assign different architectures or delete the duplicate.
But after I delete the duplicate files, it gives me
The Assembly UnityEditor is referenced by FMTurbojpegWrapper_Editor ('Assets/FMETP_STREAM/FMCore/Plugins/FMTurboJpeg/FMTurboJpegWrapper/FMTurbojpegWrapper_Editor.dll'). But the dll is not allowed to be included or could not be found.
This error. I'm not really sure what I should do to build this.

 

Please check if you have the same import settings as below.
Sometimes Unity may assign a wrong import settings if your project corrupted during import.
You may re-import our package from Asset store, or change to correct library import settings.

 

It still does not work. I tried to change the setting and change the API Compatibility Level to NET 4.x but still give me the same error. I also tried to reimport it but still the same.

ArgumentException: The Assembly UnityEditor is referenced by FMTurbojpegWrapper_Editor ('Assets/FMETP_STREAM/FMCore/Plugins/FMTurboJpeg/FMTurboJpegWrapper/FMTurbojpegWrapper_Editor.dll'). But the dll is not allowed to be included or could not be found.

 

And when I tried to reimport the package it gives me an error for all of the platform and FMTurboJepg which is
could not find the expected key at line 14

 

You may reach us via email, I could send you a clean example as reference.
thelghome@gmail.com

 

Please be reminded that you have to Restart Unity Editor after import/update any native libraries.

 

I have tried to restart it but still does not work. I don't really know why and I have sent you an email

 

received, and sent you the Unity2019.1.10f1 full project.

Edited:
-solved in v3.347
-this was a Unity Editor bug in Unity 2019.1.10f1, solved by creating an additional "Editor" folder for the dll library which involves "unity editor" class.

 

Looks nice! I have some questions

* Do you re-use the byte array which is send over network, to prevent GC
* How do you create the byte array, e.g. in an async way, or does this stall the main thread?
* Do you use some special codec for encode/decode the screen, or is it a tiny/small-jpg

 

Hi @thelghome! I'm looking at several streaming assets to replace our existing solution, and so wanted to ask a few questions to ensure that it will solve our needs.

We currently configure and launch OBS from our Unity build, using NDI to pipe the video and audio to OBS, which then streams a camera or render texture (including post-production effects on the camera) to Twitch or YouTube. However, the NDI assets between platforms tend to be inconsistently updated and maintained, leading to frequent breaks + fixes. With that in mind, I had a few questions regarding your FMETP 3.0 asset:

1. If we already have an address to send to (Youtube or Twitch RTMP server destination), do you see much additional work that would need to be done on our end to stream out to those locations with your asset? We already authenticate and create a stream on a user's behalf, so we just need a way to send their RTMP destination the a/v data.
   
2. What is performance like vs. OBS for streaming? Have you done any tests to determine if it is comparable? We'd be disappointed if our build took a large performance hit switching solutions.
3. If you can stream to browser (this might be an ignorant question!), there is a limit to the number of people who can go to the browser to view, correct? Since it's not set up to broadcast?
4. Does your asset have a solution for receiving and rendering an incoming stream? For another feature in particular, we're looking for a way to render an RTMP stream on a texture in our build, too, or a YouTube or Twitch video...

Thank you for your time and help here!

 

1. the byte[] are store in queue, and we have to make sure that they are sent in correct data.
2. the queue will be shared between threads
3. libjpeg-turbo is involved in the current version, other encoders(mpeg2, h264...etc) will be coming soon in 2023.

Our super low latency streaming solution has been used by many companies, research labs.
The stability has been proven by many public exhibitions, museum and art gallery. Some galleries reported that our live streaming applications can run smoothly for 3 months without any reboot.

 

Cool! Would the streaming work greate from one client to many (8) others?

 

About RTMP related topic, we've been working in progress with some functional internal demo(Android -> Youtube..etc).
However, it's not "Plugin ready" yet, and not supported in the current version.
We may release more details in this Summer as possible.

1. All-in-One solution for RTMP is still working in progress, but some customers' route our mjpeg/raw video stream into ffmpeg for Youtube stream support.
2. It's hard to do comparison, it's different product. Our original focus is low-latency network stream in C#.
3. Our web browser stream demo is based on WebSocket via node.js server. The number of people limit is related to total bandwidth.
4. RTMP encoder/decoder solution will be coming soon.

PS: Overall, we'd like to include all those popular features. Our development speed is also depending on the Asset Store incomes, fundings and donations. Again, thanks for everyone who purchased our products, you are the part of funders for boosting our development. Please don't forget to check out our recent big feature FM Remote Desktop, a complete remote desktop control C# solution like Teamviewer, Anydesk.

 

As long as you have good wifi between devices, it's surely no problem.

 

Hi,
Great plugin to stream video from unity application to another clients, and especially to a website (what I need for my project). I use the your web page that you include as example because I'm not an expert on the web side of development and I customize it to my needs and design.

But I have a question is possible to send back audio (from microphone) from the website to the application in unity? Something like a video chat, where several people can login to this webpage and chat via voice with the player on the unity side.

If you have any advice on how to accomplish this with FMETP Unity and a webpage, I would like any tips to guide me on the correct direction (mostly because I'm not an expert on web side of things), or if is not possible using this solution, know from the begning and start searching in another place.

Thanks in advance, for any help on this subject.

 

Yes, the Mic Input from HTML has been integrated in the example "TestServer_v2.2.4". (in FMETP STREAM 3.0)
It's purely javascript, which you may check it out.

 

Thanks, I will try it. Thanks again

 

Hi,
I'm not familiar with Video Streaming but I wanna implement screen sharing feature in my app.
Is it possible to stream entire screen or each window from PC or Mac to multiple clients?
Could I use streaming services of AWS or Google Cloud for that?

 

Hello. I am currently using your product well.
Thank you for creating such a useful asset.
I am currently working on a project using Hololens 2.
I am wearing Hololens 2 and streaming the screen and sound to the web.
I have encountered a problem during the work.
When I streamed the sound, there was a lot of noise and cutting off.
I was able to fix the noise by lowering the StreamFPS value (20->2.5), but I could see that the synchronization between the Hololens 2 screen and the web screen was slightly off.
I am curious if there is any other way to solve the noise and synchronization problems.
I will attach the audio encoder options I am currently using.
Thank you for your answer

 

Desktop capture(PC/Mac/Linux) is supported, and you can stream it through the Internet via node.js hosting on AWS EC2 or other cloud hosting services, by following our FMWebSocket setups.

Desktop Capture is an option in our GameViewEncoder, you may choose different screen index to switch, or stream multiple screens with different label ids to pair different decoders.

 

I suggested that you may untick “Match System SampleRate” option, and assign a lower sample rate like 24000.

As it is purely on web via default JavaScript pcm input on single thread, the noise is due to the gap of reading data. Longer buffer time may help, but it will give you more delay. It’s kind of web limit in the current version. It should be solvable via 3rd party JavaScript plugin instead using default JavaScript class.

 

Thank you very much for your answer.
I was able to implement the desired sound state by combining the StremFPS option and other options.
I am leaving this message because I have one more question.
In HoloLens 2, I transmitted the screen and sound to the web and accessed it from PC/Android, and I confirmed that both the video and sound were well output.
However, when accessing from the Safari/Chrome browser of IOS (mobile/Pad), the sound was not output.
Is there a specific format that I need to send from HoloLens to the web server in order for sound to be output on IOS (mobile/Pad), or do I need to do additional work on the web server side?
I have experienced something similar in the past when working with Unity and WebGL.
At that time, I thought it was an IOS security policy issue and had to find another way.
I apologize for not being able to find or study it myself due to time constraints.
Thank you for your answer.

 

iOS safari is always tricky about sound playback. By default, it requires button input to enable sound playback.
In past, we were able to put a simple html button to by-pass this policy.

I believed that's an iOS web specific topic instead of Unity part. If my origin suggestion didn't work, you might have to do some research for the latest iOS solution, which should be a very common issue for all web developers.

 

Hello @theIghome , me and my team are working on a webgl project where we need to stream a user's screen on a theatre like screen. Basically we want to represent the stream on a world canvas. Our question is whether the stream is possible on 3d canvas/world canvas because that's the sole reason why we want to purchase this asset.
Thank you!!!

 

Our decoder will output the stream as Texture, which you may assign it to material, UI Image..etc.
Please be reminded that Desktop capture feature requires standalone build(Mac/Linux/PC), WebGL can stream In-Game view, but not system desktop view.

 

Yes , we need the game view , thanks for clarifying @theIghome

 

Hellon,

v3.347
-minor bug fixes with FMNetwork

Can you tell us what this release update is about, exactly?

 

Question about desktop capture on Mac and WIndows,
Could I capture entire desktop or select one app window?
If I can select, is it possible to get the list of desktop (for each monitor) and the list of app window with the name and the thumbnail of each window?