[Release] FMETP STREAM: All-in-One GameView + Audio Stream + Remote Desktop Control(UDP/TCP/Web)

I am using Photon Network to create a Unity project. I have successfully implemented streaming of the MainCamera between Quest 2 and PC, as well as between Quest 2 and Quest Pro using FMNetworkStream. Now, I would like to implement MainCamera streaming between different networks of Quest devices using FMNetworkStream and PUN2 Example. Where should I look in the PUN2 Example for guidance?

 

If you've already implemented the streaming via PUN2 example, it should be compatible for your need of public networks.
https://frozenmist.com/docs/apis/fmetp-stream/pun2-example/

 

Does remote desktop view work on Windows 11? I just saw this

Maybe i just need to update?

 

We have major changes in V3, which supports Mac, Linux and Win11.
The old method in V2 has some known limitation related to system settings related to GPU.

 

So you gotta pay full price for the new version just to get windows 11 support? That seems kinda screwy, especially since i'm only using FMETP for the desktop share. Is there an upgrade option?

 

About your screen sharing issue in V2, please refer to this troubleshooting page "FM Desktop Stream issue":
https://frozenmist.com/docs/apis/fmetp-stream/troubleshooting/

Let me explain in this way:
In V2, this is a known issue and it's solvable by changing Window System display(graphic) settings.
In V3, this known issue has been fixed by a completely different implementation, including new support on Mac and Linux Desktop.

Overall, we didn't force any user to upgrade and pay more, we didn't take away any features in V2.
We are spending time on the development and noticed that we have to make major changes for improvement.
Some core scripts can't be upgraded easily because there are completely different structures and libraries involved between V2 and V3.

Please try our suggested method for V2, it might also solve your issue on Win11.
PS: Win11 doesn't exist at the time we release V2. Thus, it might work, but we haven't verified it yet.

 

So, the only solution is to have every user go into their graphics settings and change to the integrated chip by default? Is there any other way? That sounds like it would be a pain to tell every user to use the integrated chip by default, not to mention it wouldn't be good for performance.

 

Please be reminded that this has been fixed in V3, we've re-worked on the core part, major difference from V2(months of effort in development).
For V2 users, there is 50% off discount when you login with V2 account via Asset Store.

 

Hi,

We acquired the asset with the goal of streaming a VR headset (Focus 3) to a browser frontend (React) running on a different device. For an initial proof of concept, we implemented the streaming from Unity to the browser frontend, running on different devices (PC to PC). For this proof of concept, we employed Websockets, which we managed to get working, so being able to display the streamed images in a simple frontend. However, we ran into issues when trying to stream from a VR headset (Focus 3) to the same frontend. For the setup we used the documentation and the demo video & project for the Quest VR Stream.

The issue is that we receive a gray / black image when we set the Capture Mode of the GameViewEncoder to Main Cam, the Main Cam being the camera of our Rig. When we inspect the Texture right before the Encoding in Unity (by saving it locally), the texture is already gray / black there. Therefore, we concluded that the problem should not be related to our Websocket setup. Some googling resulted in there being a known issue with the Blit() function not working correctly with Unity URP and Single Pass Instanced VR, resulting in the aforementioned gray / black image. From what we could gather from the code, the Blit() function is used in the GameViewEncoder to get the current texture of the camera, which would explain our encountered issue. We tried different Unity versions to see if it got fixed but had no luck. We also tried it on another VR headset (Pico 4) and had the same issue.

As an alternative we also tried the Render Cam Capture Mode. This seemed to be working. However, if we interpret our testing correctly, this mode re-renders the entire scene and uses that result for streaming. (The streamed image had a different field of view as the VR headset with a slightly off camera position). Here, we had the concern that re-rendering the scene would incur high performance cost, especially considering we are using standalone VR headsets.

Accordingly, we have the following questions:

1) As we saw from the video, you made it working with the Oculus Quest and the Main Cam Mode. Do you have some ideas what we could try, or what we might be missing?
2) Might you have further insights regarding performance comparison of the different Capture Modes?
3) Do you have any further advice on how we could / should proceed with dealing with the encountered issue?

 

It's URP hardware specific issue with VR headset.
Our VR main camera stream demo is running on the default render pipeline, not URP.

Of course there will be performance impact, when you have duplicated camera. At least the drawcalls count will be double in this case.

Depending on your project scale and visual quality:
According to my personal experience in museum, exhibition and gallery art standard, the default standard render pipeline can achieve almost the same quality of URP if you have good knowledge in 3D modelling, texturing, lighting and rendering skill.

However, most of VR projects nowadays are lead by programmers without "good 3D skills", and 99% of them will just jump into URP immediately without second thought. They are just obsessed with the soft-shadow and post-processing enabled by default.

If you ask me personally, will you benefit much with URP(in terms of quality and performance)?
My personal answer will be "NO" for most of VR projects I've seen so far.

I've been working on many computer systems, and does some in-house benchmark too.
For the same quality of rendering, you will have almost zero benefit from URP's default settings. Sometimes it's even worse with URP default settings, in comparison to standard pipeline. For URP and HDRP, those are actually designed for programmers who are good at writing their own render pipeline.

Meanwhile, still, the URP is still not "100%" stable for VR development, because the URP was developed by many different teams in Unity Team. The original name is light-weight render pipeline, but it turns out "bad performance on some devices". Somehow they changed the name to URP since too many bad reviews...

If live streaming and stability are your first priority, please consider standard pipeline again.

Hopefully above explanation can give you some idea and direction. Otherwise, I have to write a book to further discuss~
Good luck!

 

Thanks for the great feedback and insights! We will definitely consider this. A book considering this discussion sounds interesting ...

 

I've just gotten the FM Network 3.0 asset and I'm having a lot of trouble connecting one device to another. Usually I'm able to have the other devices connect to the server by using the servers IP - is there something that I'm missing?

 

It seems likely that the auto network discovery wasn't working properly.
99% is related to firewall rules, which might block the udp broadcast packet, while 1% will be related to public wifi, campus wifi or company wifi rules.
fyi: https://frozenmist.com/docs/apis/fmetp-stream/troubleshooting/

 

Hello Dear Developer.
I've recently been using your plugin for the Hololens2 UWP platform and I've noticed a significant drop in performance (from 60FPS to 15FPS) after enabling the script. I'm very confused I tried a lot of things but it doesn't work. If possible, would it be handy to see if there is an error in my plugin settings that is causing the performance drop?

Here's the Unity Profiler trace data, I've hidden the parts that have less of an impact on performance.


These are the components in my scene:


This is my plugin settings, I hide my public server IP (I use a remote server)


If you see any details or clues from this that can help me improve the efficiency of my program's operation, please let me know, thank you very much.

 

In the TextureEncoder, you forgot to enable the Fast Mode, Async and AsyncGPU Readback options.

Beside, you may consider using GameViewEncoder(MainCam mode) instead, which will have better performance than TextureEncoder.

Lastly you can do some local network test for comparison.

 

I have set up streaming from one device to another and thats working great. At some point I want to stop the stream, take a screenshot and send that image to the other device. How would I best do that? I'm not having any luck so far.

 

You can stop the stream by disabling the Encoder.
If you have other stream input like Texture, you can use TextureEncoder instead, usually you can set another pair of GameViewDecoder with the same label ID of TextureEncoder.

 

Thank you for your reply and guidance, when I enable FastMode for encoder and decoder, the app frame rate increases to about 30FPS, but I can't enable "Async GPU Readback", the app crashes while running on Hololens2 when it's enabled.
For the suggestion of the GameViewEncoder component, I need to use a web camera so I may not be able to use it.

 

In the MainCam mode, you can enable the "Mixed Reality" option to include webcam view.

Btw, "AsyncGPUReadback" should be compatible with MRTK 3.0, is there any error log from your side?

 

Sure, sorry I spent some time looking for the crash log file, for GameViewEncoder's suggestion I will adjust the code and pick up structure and test it, thanks again for your help!

 

Here is the memory dump file when the crash happened.
https://drive.google.com/file/d/1oPedHW0-QwWRCn1E85_0Nfn6zqbWZVaa/view?usp=drive_link


FILE_IN_CAB: HoloLive.exe.8144.dmp
NTGLOBALFLAG: 0
PROCESS_BAM_CURRENT_THROTTLED: 0
PROCESS_BAM_PREVIOUS_THROTTLED: 0
APPLICATION_VERIFIER_FLAGS: 0

CONTEXT: (.ecxr)
x0=0000000000000380 x1=0000018612d83aa0 x2=00000186c303a890 x3=0000000000000000
x4=0000000000000002 x5=00000186c303ad70 x6=0000000000000430 x7=00000186c303a948
x8=0000000000000000 x9=0000000000000038 x10=0000000000000006 x11=0000000000000006
x12=00000186c264d050 x13=00000186c26c77a0 x14=00000186c26c7b20 x15=0000000000000001
x16=00007ff9576615c0 x17=ffffa0eb5d519553 x18=0000000000000000 x19=0000000000000002
x20=0000000000000380 x21=0000000000000000 x22=00000186c303a890 x23=00000186c26c9820
x24=00000186c26ca6a0 x25=00000186c26c7b20 x26=00000186c264d090 x27=0000000000807fff
x28=0000000000000000 fp=00000047c0cff080 lr=00007ff948177954 sp=00000047c0cff020
pc=00007ff9481779bc psr=20000000 --C- EL0
turbojpeg+0x79bc:
00007ff9`481779bc 4c400020 ld4 {v0.16b,v1.16b,v2.16b,v3.16b},[x1]
Resetting default scope

EXCEPTION_RECORD: (.exr -1)
ExceptionAddress: 00007ff9481779bc (turbojpeg+0x00000000000079bc)
ExceptionCode: c0000005 (Access violation)
ExceptionFlags: 00000000
NumberParameters: 2
Parameter[0]: 0000000000000000
Parameter[1]: 0000018612d83aa0
Attempt to read from address 0000018612d83aa0

PROCESS_NAME: HoloLive.exe

READ_ADDRESS: 0000018612d83aa0

ERROR_CODE: (NTSTATUS) 0xc0000005 - 0x%p 0x%p %s

EXCEPTION_CODE_STR: c0000005

EXCEPTION_PARAMETER1: 0000000000000000

EXCEPTION_PARAMETER2: 0000018612d83aa0

STACK_TEXT:
00000047`c0cff020 00007ff9`48192914 : 2b331d2f`96464c8b 14d16b2f`800054cd 00000047`c0cff060 00007ff9`48192b64 : turbojpeg+0x79bc
00000047`c0cff0f0 00007ff9`4818d20c : 00000047`c0cff170 00007ff9`4818d20c 000001f8`00000008 00000186`c303ac40 : turbojpeg+0x22914
00000047`c0cff170 00007ff9`4817c928 : 00000047`c0cff1b0 00007ff9`4817c928 00000186`c28c86d0 00000000`000001f8 : turbojpeg+0x1d20c
00000047`c0cff1b0 00007ff9`481b1e64 : 00000047`c0cff1f0 00007ff9`481b1e64 00000186`00000006 00000000`00000001 : turbojpeg+0xc928
00000047`c0cff1f0 00007ff9`462d9550 : 00000047`c0cff2b0 00007ff9`462d9550 00000000`c28c8601 00000380`00000e00 : turbojpeg!tjCompress2+0x25c
00000047`c0cff290 00007ff9`462d926c : 00000186`00000002 00000186`00000032 00007ff9`00000002 00007ff9`47d8d000 : GameAssembly!DllGetActivationFactory+0xab640
00000047`c0cff390 00007ff9`4629b2bc : 00000000`00000002 00000000`00000000 00000047`c0cff3f0 00007ff9`4629b2bc : GameAssembly!DllGetActivationFactory+0xab35c
00000047`c0cff3f0 00007ff9`462855dc : 00000047`c0cff440 00007ff9`462855dc 00000186`c2b49150 00000000`00000000 : GameAssembly!DllGetActivationFactory+0x6d3ac
00000047`c0cff440 00007ff9`46ddda68 : 00000047`c0cff4a0 00007ff9`46ddda68 00000000`00000000 ffffffff`fffffffe : GameAssembly!DllGetActivationFactory+0x576cc
00000047`c0cff4a0 00007ff9`46dd9dec : 00000047`c0cff4d0 00007ff9`46dd9dec 00000186`1359cf50 00000186`bf7394e0 : GameAssembly!DllGetActivationFactory+0xbafb58
00000047`c0cff4d0 00007ff9`46ddd9a4 : 00000047`c0cff590 00007ff9`46ddd9a4 00000000`00000000 00000186`bf7394e0 : GameAssembly!DllGetActivationFactory+0xbabedc
00000047`c0cff590 00007ff9`45f22e10 : 00000047`c0cff5d0 00007ff9`45f22e10 00000186`c2b49120 00007ff9`47c7ddd0 : GameAssembly!DllGetActivationFactory+0xbafa94
00000047`c0cff5d0 00007ff9`46de2c74 : 00000047`c0cff5f0 00007ff9`46de2c74 00007ff9`47d91000 00007ff9`47c7ddd0 : GameAssembly!mono_field_set_value+0x268
00000047`c0cff5f0 00007ff9`46de97dc : 00000047`c0cff750 00007ff9`46de97dc 00000184`ec000100 00000186`c2b49120 : GameAssembly!DllGetActivationFactory+0xbb4d64
00000047`c0cff750 00007ff9`4602e784 : 00000047`c0cff770 00007ff9`4602e784 00000047`c0cff790 00000000`00000000 : GameAssembly!DllGetActivationFactory+0xbbb8cc
00000047`c0cff770 00007ff9`461e9534 : 00000047`c0cff7a0 00007ff9`461e9534 00000184`f02206e8 00000184`f01fdd20 : GameAssembly!mono_image_is_dynamic+0xba8cc
00000047`c0cff790 00007ff9`461b07c4 : 00007ff9`4618adf0 00000000`00000000 00000047`c0cff7f0 00007ff9`461b07c4 : GameAssembly!WriteZStream+0x107f4
00000047`c0cff7f0 00007ff9`4618ae4c : 00000047`c0cff880 00007ff9`4618ae4c 00000186`13579850 00000186`7d1fdf30 : GameAssembly!SystemNative_UTime+0x4c7c
00000047`c0cff880 00007ff9`461ab508 : 00000047`c0cff8d0 00007ff9`461ab508 00000186`7d0f2701 00000186`13577b40 : GameAssembly!mono_jit_info_get_method+0x2c81c
00000047`c0cff8d0 00007ff9`4621e2c4 : 00000047`c0cff920 00007ff9`4621e2c4 ffffffff`fffffffe 00000000`00000000 : GameAssembly!mono_type_size+0x3d48
00000047`c0cff920 00007ff9`759201c8 : 00000047`c0cff950 00007ff9`759201c8 00007ff9`461ab3c0 00000186`7d0f2720 : GameAssembly!WriteZStream+0x45584
00000047`c0cff950 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x58

SYMBOL_NAME: turbojpeg+79bc
MODULE_NAME: turbojpeg
IMAGE_NAME: turbojpeg.dll
STACK_COMMAND: ~66s; .ecxr ; kb
FAILURE_BUCKET_ID: INVALID_POINTER_READ_c0000005_turbojpeg.dll!Unknown
OS_VERSION: 10.0.22621.1120
BUILDLAB_STR: ni_release_svc_sydney_rel_prod
OSPLATFORM_TYPE: arm64
OSNAME: Windows 10
IMAGE_VERSION: 0.2.0.0
FAILURE_ID_HASH: {9425f581-b08f-bbb1-9c76-6ef1a865436c}
Followup: MachineOwner

 

Do you have screenshots of your Unity Scene's Hierarchy and GameViewEncoder Settings?
You may also reach us via email, for the ease of following your case.
technical support: thelghome@gmail.com

 

Hi! anyone have successfully worked with a Magic Leap? I'm getting this error as soon as I try to compile for Magic

Assets\FMETP_STREAM\FMCore\Scripts\Features\WebcamManager.cs(8,13): error CS0246: The type or namespace name 'WebCamDevice' could not be found (are you missing a using directive or an assembly reference?)

2020.3.48f1 LTS
FMETP V2

 

In v3:
You may add scriptable symbols FM_WEBCAM_DISABLED in order to disable the webcam permission, if your devices isn’t allowed.

In v2:
You have to remove the webcam related script manually, if your device is not allowed.

Please refer to this page: https://frozenmist.com/docs/apis/fmetp-stream/

 

Hello @thelghome ! I have finished my multipeer connection with your system, it's amazing !
But I have a problem, my game is multiplatform and i have a lot of lags on VR (like 2 fps and Quest 2 freeze when FMETP is in my scene).
On VR, I use the GameViewDecoder and I tried Fast Decode Async etc ...
Do you have an issue ?
Thanks you !

 

What's your networking system(PUN2 or FMWebSocket)? It shouldn't be 2 fps when decoding on VR. (sounds likely sth else problem).
You may please reach us via email, which we can follow your case easily and we may need more details from your side.
technical support: thelghome@gmail.com

 

[V3.400] Updates and bug fixes
-Fixed the GameViewEncoder Preview flickering issue on URP
-Fixed GameViewDecoder texture size bug on Unity 2023

 

Hi is there anyway of getting the latency or ping between clients to server in ms ?
I want to show the info on UI and I went through the docs, but I couldn't find any helpful info of how to get this data any suggestions?

 

There is no ping info unfortunately, but it won't be too difficult to write your own function to check the ping.
You can simply send a dummy string message to server and send it back to your client. Then, you can compare the timestamp on client side.

 

Thanks for your previous answer

I have another question, right now, we are facing an issue where the audio is so glitchy, the server instance is running on Amazon AWS EC2 free tier.
when the mic is shared and two people are trying to communicate, there is always a disruptive very annoying noise and there is a huge gap!
any idea what is the best course of action to have a better rate in video audio transmission?

Do you have samples that audio works best?
and andy suggestions to use infrastructure?

 

Are you testing audio playback on WebGL or HTML?
If yes, the audio playback is through javascript's default audio class without advanced buffering.

Some customers integrate other 3rd party audio playback plugins, to improve the audio stream playback on Web.

PS: our main development is on Unity side and doesn't include advanced Web audio player in the latest version yet.

 

I am trying it on Android, non of above, my platforms are desktop, android, and iOS.

 

Do you have screenshot of your Encoder settings?
Please try adjusting the samplerate to 22khz for mobile if possible, and reduce the StreamFPS for larger buffer.

 

Hello, can I play the IP camera RTSP stream with your asset?

 

This is still working-in-progress, and not supported in the current version yet, unfortunately.

 

I tried to play around with the settings for a while here is what I have

When I put the sample rate to 22050 what I hear from other devices is a baby voice xD the pitch will be increased!
here is my decoder :


one very weird thing is, whatever i say it gets echoed back to me, sometimes, I even hear back what i said 1 minute ago multiple times from the origin device!

What should i do?

 

The echo is usually due to your testing environment, your microphone might capture your speaker playback in the same room in reality.
The output sample rate shouldn't be changed in runtime.
If you write us an email for technical support, we can follow your case easier: thelghome@gmail.com

 

I have a question about AudioDecoder and MicEncoder.
Two people can hear each other fine. When I try to communicate with more than two people, we can hear each other, but it's choppy and stuttery. What can I do to improve the quality?

 

In your case, you should assign different label ID for each MicEncoder(each user), and create multiple AudioDecoder for different label IDs.

It’s interference issue(not quality issue) when you are feeding multiple MicEncoder to one AudioEncoder with same Label ID, which is a wrong setup.

 

I don't understand, but can you give me an example of your setup? I have more than one person communicating and I want them to be able to hear each other.

 

Assuming 3 people talking to others, each user should have unique MicEncoder, plus multiple AudioDecoder for others.
App(User1):
-MicEncoder(Label ID 2001)
-AudioDecoder1(Label ID 2001)
-AudioDecoder2(Label ID 2002)
-AudioDecoder3(Label ID 2003)

App(User2):
-MicEncoder(Label ID 2002)
-AudioDecoder1(Label ID 2001)
-AudioDecoder2(Label ID 2002)
-AudioDecoder3(Label ID 2003)

App(User1):
-MicEncoder(Label ID 2003)
-AudioDecoder1(Label ID 2001)
-AudioDecoder2(Label ID 2002)
-AudioDecoder3(Label ID 2003)

 

So how do you recommend tweaking the values of SteamFPS and Output Sample Rate to avoid stuttering?

 

Please test my suggested setup of AudioDecoder first and see if it helps. As the chopping sound is obviously due to wrong setup, which mixed multiple mic inputs from multiple users.

The output sample rate can help reducing the bandwidth. Lower streamFPS should have larger buffer for smoother playback.

 

Hello, I have a question about FMNetworkManager. I'm curious about the minimum and maximum values of ServerListenPort and ClientListenPort.

 

Should be the same as UDP/TCP port range.

 

What I'm wondering is the range of Port numbers. For example, from 1 to 3000

 

We don't have that limit, please refer to the general answer.

 

Thanks to this, testing is going well. I have another question though, I set the [Output Sample Rate] for [MicEncoder] to 8000 and the [StreamFPS] to 2.5. There is a slight delay, should I lower the values further, and if so, by how much each?

 

You may increase the streamFPS a bit, 2.5FPS means that the data will stream 2.5 times per second. You can increase it to 10~20 streamFPS.
Meanwhile, there is ~0.5 sec delay in Unity Audio playback, which is one of the limitation using Unity Audio System.
If you want to reduce extra 0.5 sec delay, you need to use 3rd party native audio playback system instead of Unity3d audio system.

 

Hi, @thelghome
We are working on something that requires screen sharing, audio streaming (for each individual player) and video streaming. Using Fishnet, we got the byte array streaming working. However, on WebGL, the audio modification capabilities of Unity are lacking.

Your plugin seems to be solving several of our problems. So, we need to know before making a decision on the purchase"
1. If audio streaming is supported on WebGL? We need spatial audio, i.e., separate audio for each player. (Our solution works fine on all platforms except WebGL)
2. If video streaming, and whether multiple resolutions can be streamed (for bad connections, crowded network, etc).
3. for desktop sharing. What are the supported platforms?
4. for how long will it be supported and if we can request new features?