[Release] FMETP STREAM: All-in-One GameView + Audio Stream + Remote Desktop Control(UDP/TCP/Web)

For development testing on the same computer, there is only one server + one client connection would be allowed.
It's due to IP address limit, when communicating via UDP.

 

Hi all, the image I receive in HTML is mirrored, flipped, I don't know why, hope someone can help me

 

In GameViewEncoder, there should be "Flip X" option under the "Desktop Capture Mode" settings.

 

[引用=“thelghome，帖子：8307420，成员：1157023”]在GameViewEncoder中，“桌面捕获模式”设置下应该有“Flip X”选项。[/引用]
哇！非常感谢！我刚刚接触到这个插件，这个问题困扰了我两天，我以为是我电脑显卡的问题，谢谢！

 

Can point cloud only be pushed locally? How to change the server settings of point cloud

 

It's simply a pair of encoder and decoder.
By switching different networking system for byte[] streaming, you can get either local and Internet Point Cloud Streaming result.

You may also pair them directly without networking system for kind of real time point cloud visualisation.

 

Can the gameview encoder GC be avoided

 

Thanks for pointing out the issue, we will review & investigate the related codes.

 

I compared several situations, which seems to have a great impact on the frame rate

 

I am a little bit confused by your zfbrowser ms test.
Thus, I did a quick test shown below, with screenshots.

[Below is testing for FMETP STREAM V2]
# Empty scene without GameViewEncoder: ~1ms
# GameViewEncoder with AsyncGPUReadback enabled should give you the best performance.
In this case, most of time GC contributes ~1ms workload and occasionally reach 4ms.

 

I am using FMETP v2, Connecting Pico VR and Huawei Android Pad. VR and Pad connect the same Wifi, Sometimes FMNETWORKMANAGER can connect, mostly FMNETWORKMANAGER cannot connect(As Below). Someone can help me solve this problem.

 

For mobile devices, we suggest that you may reload the scene, whenever your app is paused or resumed from background (or accidentally). Because in some specific Android phones/VR device, it didn't kill the existing connection correctly when the app is running in background. It will occupy the port when your app is resumed.

PS: According to our testings, the Unity behaviour of "OnPause()/OnResume()" on mobile devices could be very different in some phones & OS version, as some of them are triggered in main thread, while some of them are triggering in different timing unexpected.

 

Thanks, I solved this problem. I think it's caused by the wifi router limit.

I want to use "Main Camera" capture mode in my Pico VR headset.
But there is 1 error in unity Editor,
How can i solve this problem

 

I retested it. It seems to be related to the richness of the picture. The richer the picture, the larger the encoded size, and the lower the frame rate of the game

 

And how to use desktop capture mode

 

I am using "Main Camera" capture mode, but there are 3 errors. Someone can help me solve this problem.

 

This should be missing materials, usually reset or re-add the component should fix it.

 

Simply change it to desktop capture mode.
For V2 users, if it returns null empty image, it's related to GPU options referring to "FM Desktop Stream issue" part.
https://frozenmist.com/docs/apis/fmetp-stream/troubleshooting/

This known bug should be fixed in V3.

 

In general, URP main cam capture mode should be compatible, according to our testing result.
This might be related to URP with Android VR headset, as they have a hardware specific GPU for mobile VR purpose.

 

It seems to be related to the richness of the picture？

 

This would be related to FMSocketIOManager part in FMETP STREAM V2, it has one more layer of encoding in the networking part. (it's a known issue about performance impact with huge data stream)

We've done some optimisation in V3 which should come soon in August.
Meanwhile, we are also working on alternative of FMWebsocket solution.
We will try to integrate the new method into V3.xx as soon as possible, though more time and more fund are needed for our long term development.

 

FMETP STREAM 3.0 is finally released!

Desktop Capture for PC, Mac & Linux now, and better performance on HoloLens 2 with Mixed Reality Stream, and more!
Let's check it out!

 

As the price is savage (doubling the v2 premium price), will 2.0 be updated or will it be deprecated?

 

We understood your concern about recent major upgrade.
If you've owned our Beta 3.0 test version, please feel free to keep it for your own project usage.

FMETP STREAM 2.0 would be considered as LTS for at least one year from now, with minor bug fixes.
While FMETP STREAM 3.0 would be our main focus in development, especially for new features and performance.

In reality, we've been trying very hard to keep the development cost as low as possible for FMETP STREAM V1 & V2, which does help many students, researchers and startups.

However, as a small team here, we still have lots of bills to pay...

PS: It’s also a difficult choice for us to raise the price, in order to maintain our development in long term.

 

As student, even the half price (regular v2 price) is really high for an upgrade. I'll keep the beta until I can afford the v3 release. Will those fixes you mentioned (already made for v3) be added to v2?

 

Hi, I would like to know how stable this plugin is on Linux on its 3rd version. I would especially need the GameView to WebSocket Streaming and camera inputs via WebSocket. It would be nice to know if there is some kind of trial before spending 134€ blindly, or at least to be sure that these two features will work correctly and smoothly.

 

There are some big changes between V2 & V3, especially on FM Network 2.0 and GameViewEncoder.
It's hard to keep them in sync for V2, as the core design has been changed. Thus, V2(as LTS) would only focus on bug fix and testing on newer Unity versions.

For students with V2(especially who tested our V3 beta), you may reach us directly via email if you have specific projects with V3 feature needed. We could still grant you access for non-commercial projects for study & research purpose.

 

Good point, we just made a quick demo shot with Ubuntu for your concerns.
However, we no longer provide trial version, as it's very hard to prevent illegal usage.

 

Hello. I plan to buy your plugin. There are several questions.
I develop a game for a browser on WebGL, requires a capture video from desktop then stream to several players in the room on the ui canvas(Texture2D). Players are connected via Photon Network. It is necessary to send the sound from the microphone and the video p2m as Live Stream on WebGL.

1) Is it possible to start capture desktop video from browser then broadcast this to other players?

2) Is it possible to capture mobile device screen video then broadcast this to other players?

3) Can your plugin implement voice comunication between players in the same room via mic?

4) Is a dedicated server required to broadcast the video stream?

5) Which version of your plugin is better for my purposes 2.0 or 3.0

Thanks!

 

Let me simplify the answer for above questions.
Desktop capture is only available for PC, Mac, and Linux standalone build, as it's too complicated to grant permission from WebGL.

If you are using Photon/PUN2, you can stream the desktop data via PUN2 directly without extra server.
https://frozenmist.com/docs/apis/fmetp-stream/pun2-example/

As PUN2 also supports Mic, it would be easier to use their SDK instead.
If Desktop Capture is your key feature, FMETP STREAM 3.0 would be recommended.

 

Thanks for the answer. PUN2 Voice not support webgl((

 

Unfortunately there is also same limitation on webgl build too. The default mic input class is based on Unity, which they didn't allow access via webgl.

For general solution, it requires native mic input sdk on webgl via javascript, usually you will get the raw PCM array as input. Then convert them into byte[] for PUN2 streaming. There should be some potential solution on github..etc.

 

try to find out, thx

 

There is an interesting use case requested by customers, we released an update for ffmpeg playback support, via ffplay command. It's also available in our api page:
https://frozenmist.com/docs/apis/fmetp-stream/unity-to-ffmpeg/

Code (CSharp):

1. // + GStreamer Commands(Example):
2. // + Playback source: Unity GameViewEncoder(Output Format: MJPEG)
3. // + with FMNetworkManager(DataStream, Sender mode, port 3001)
4. ffplay "udp://127.0.0.1:3001"
5. // + Playback source: Unity MicEncoder(Output Format: PCM16, mono, sample rate 11025)
6. // + with FMNetworkManager(DataStream, Sender mode, port 3001)
7. ffplay -f s16le -ac 1 -ar 11025 "udp://127.0.0.1:3001"

 

Hi, is it possible to stream 2 difference Client-Cameras to one Server with 2 RAWImages on one CANVAS Display???
I mean 1 to 1 works very well over a node Server. But how can i display two different views on one Display side by side.

 

You can setup two encoders with different label IDs. On your display app, you should setup two decoders too.

 

Thank you. Works Perfect.

 

@thelghome The Computer Desktop Capture part for V3 is intriguing. What is that all about? Can't find any details on it anywhere.

 

The Desktop Capture mode is for streaming your Computer's System Desktop, which is compatible on PC/Mac/Linux.
It's very straight forwarding solution and we would include more description on our webpage soon.

 

Hi,
is it possible to send 2 or more Strings to other client?
My Client has the following script:

Code (CSharp):

1.         string latitude = Input.location.lastData.latitude.ToString();
2.         string longitude = Input.location.lastData.longitude.ToString();
3.         FMSocketIOManager.instance.SendToAll(latitude);
4.         FMSocketIOManager.instance.SendToAll(longitude);

And my client, which should receive the messeges, has the code:

Code (CSharp):

1.     public void Action_ProcessStringData(string _string)
2.     {
3.         debugText.text = "Received  string: " + _string;
4.     }

If i send one string, that works. But with 2 Strings, the receiver APP shows the last String on the Line. (longitude)

I used the FMSocketIOManager.

 

For debugText demo, it's only showing the last text on UI Canvas.
The simple solution could be this:

Code (CSharp):

1. debugText.text = debugText.text + "\n" + "Received  string: " + _string;

 

[V3.100] FM WebSocket 2.0 Major update is included in FMETP STREAM 3.0 too!
It's been a while, but we've made progress for improving performance on WebSocket!
We re-create the core part of Server-Clients system via websocket, without the need of socket.io anymore!
The overall encoding performance should be faster on mobile devices

more demo and detailed documentation will be coming soon.

 

Ok, im sorry, but i didnt understand your answer.
With this, i also get the last string and not both.

I mean in my sendData script, there are exists two strings "latitude" and "longitude:

Code (CSharp):

1.         string latitude = Input.location.lastData.latitude.ToString();
2.         string longitude = Input.location.lastData.longitude.ToString();
3.         FMSocketIOManager.instance.SendToAll(latitude);
4.         FMSocketIOManager.instance.SendToAll(longitude);

The problem is, how can i get both strings in my getData Script:

Code (CSharp):

1.  public void Action_ProcessStringData(string _string)
2.     {
3.         latitude.text = _string; <--------- First String
4.         longitude.text = ?????? <-------second String
5.     }

 

In your case, you send twice and it will trigger Action_ProcessStringData() twice. It will override your previous value as expected, according to how C# works.

You could simply group both of variable together, and split them later on the receiver. For example, use "," for separation purpose. If you have advanced data group, you may consider json format instead.

Code (CSharp):

1. FMSocketIOManager.instance.SendToAll(latitude+","+longitude);

Code (CSharp):

1. public void Action_ProcessStringData(string _string)
2. {
3.   string[] chunks = _string.split(',');
4.   latitude.text = chunks[0];
5.   longitude.text = chunks[1];
6. }

 

OK thank you. Can i use the Socket ID, which is a part of the FMSocketIOManager, to send a data specially to this SocketID?
Is there an any API Doc where can i look, what are the functions of each components is?

 

It's possible with FM WebSocket 2.0, which use FMWebSocketManager instead of FMSocketIOManager.
https://frozenmist.com/docs/apis/fm-websocket/fm-websocket-manager/

The WSID will be assigned once connected, via below command:

Code (CSharp):

1. FMWebSocketManager.instance.Settings.wsid

 

But how can i set the ID?
Like this???:

Code (CSharp):

1. FMWebSocketManager.instance.Settings.wsid = "23232"

And both, client app and the server app, should have the the same line in their script????
Do you have an example code?

 

In current version, the wsid is generated from node.js server side, which you shouldn't change it manually.
It will also return the value of your own wsid via "OnReceivedWSIDEvent()". It will be invoked when you connect to the node.js server successfully.

If your network type is Server, OnClientConnectedEvent() will return the new connected client wsid on server.

What's your use case, are you going to create nick name for each user?

 

I have a node.js server which runs on a raspberryPi4.
Additonally i have a Android Tablet and Android Smartphone. They have AR-Apps which tracks the GPS Data and send the Camera View to a Windows Applikation over the node.js Server.
The Stream from the Camera-Views of the devices works very well with the FMSocketIOManager and the GameViewDecoder/GameViewEncoder.
But additionally i will display the GPS Data (lon/lat) from each device under the Views.

If i send only the Data from one device, it works. But i want to stream the GPS Data fro two different devices to my Windows Application.

I thought, i can use two different FMSocketIOManager in my WindowsAPP and give each Canvas in my Screen different SocketID´s. So i thought i can send from the Android devices over the socket id different GPS Data. But i was wrong.

 

I have a quick question:
I've been using V2 for a long time to send camera view over my local network, but I had another idea:
I envision a system where my unity desktop app can "host" a live stream that any other person with a special access code can view and interact with through my website.
Is this possible?
And if so, do you have a recommended way of handling it?

Thanks!