[Release] FMETP STREAM: All-in-One GameView + Audio Stream + Remote Desktop Control(UDP/TCP/Web)

Oh. Ok I found the issue. Seems like it's about HTTPS (or rather wss). I'll see about that. Thanks for the answers tho! Appreciate it

Edit: A question, does the server zip file you provide support wss? I'm not sure if it does...

 

the demo node.js server from zip is a minimal demo for local hosting, as an example purpose.

You have to modify it for wss need. Since there are many different tricky solutions for hosting, we only focus on Unity3D support with minimal node.js demo. But you can surely contact us via email, if you need more help or advice.
technical support: thelghome@gmail.com

 

I found out that by default, when ssl is enabled, the port value is not passed to the url request in WebGL build. Is this intentional or considered a bug?

 

We've received your email and thanks for pointing out this issue, seems likely a bug.
We will investigate this issue, and we may need more info from you via email.

 

Hello,

We are using your asset for streaming desktop/game view over a multiplatform program we're making. We did a prototype first and it worked great - it's awesome how easy it is to set up

The issue I'm having now is that we're trying to use UDP instead of TCP, and this has caused issues with the streaming. For context we are using TNet for our networking. I'm looking into it further, cause I think most of the problem is mine, but where it stops right now is in in GameViewDecoder, under Action_ProcessImageData. I'm printing right before this if check:

Code (CSharp):

1. if (receivedLength == dataLength)

and I'm seeing very different numbers between the two. dataLentgth being in the 50-60000 ranger, but receiveLength closer to 1000. This means in turn that ProcessImageData is not called.

I don't have a very thorough understanding of what's going on with the unpacking of the byte array, so I come here for help Do you have any ideas what might be causing this discrepancy?

Ole

 

In our Encoder, we basically split bytes data into small chunks for UDP stream, and we found this method does optimise the network performance a bit too.

In the Decoder, it simply collects all chunks into one complete byte array for decode.

I am not familiar with TNet, but as long as you follow below structure, it should fit almost any kind of networking system.
Encoder -> bytes -> Network manager -> bytes -> Decoder

We suggested that you should test from our UDP demo first, which may give you some idea.
There might be some chance that you have wrong setup, or interfering by duplicated label ID for different encoder/decoder. We would still like to help, and you are welcome to write us email for technical support: thelghome@gmail.com

 

I recognize that I am lost.

I have a VPS on windows.
I have installed node, socket.io, express, etc ...

I have compiled the examples they provide. I open it on the server and there is a connection. The command "node index.js" opens "listening 3000" perfectly and everything on the server works fine.

The problem is when I put a client (my computer, for example) to connect with that server. I suppose that if I open the client on a computer (not on the server) as a client and I introduce the IP of the server (which is open and connected) a communication should take place. Is that so?

Everything works perfectly locally, but I have not found a way to do it through a server. I have tried with windows and with linux and there is no way. So I am doing something wrong.

I have opened ports, removed firewall .... but nothing.

I've been in for many hours (and test server costs) and haven't found the solution. I have tried it. So I have no choice but to turn to you. The same is something very simple that I am not able to see.

Thank you.

 

It seems likely there might be some settings issue on either your VPS, or maybe you need some modification on our node.js demo.

There was a very kind suggestion by one of reviewers, I try to repost his solution. It might or might not be related, but it may give you some idea.
ref: https://assetstore.unity.com/packages/tools/network/fm-websocket-155542#reviews


According to different feedbacks from customers, they always have different settings, or some modification based on our demo node.js script. Because there are just too many different setups of server hosting, and we can only provide the minimal local hosting example.

Thus, in any case that you are struggling, please feel free to email us. We are always happy to share idea and give advice.
Technical support: thelghome@gmail.com

 

I'm going to try it by other means, to see what I get.

The communication through UDP and TCP I have it solved for other software that I developed and I suppose it will be similar. After all, it's C #.

Thank you anyway.

A greeting.

 

So here's my next question.

Do you have any kind of buffering implementation? To eliminate any stutter and/or to be able to transfer large screens or multiple videos?

 

In order to keep the lowest latency, we didn't implement any buffer.
We understood your concerns, we may consider your idea for our future development as an optional feature in settings.

 

Alrighty then. Thanks for the answer. I think I'll implement my own algorithm for that then. With an acceptable latency, more fluid flow would be preferable for a number of reasons.

 

If I want to pause a stream temporarily, is the best way to disable the component?

 

Yes, the simplest way is disabling the Encoder. It will resume when you enable it.

 

[Share]
We've already tested on Quest 2, it works well with FMETP STREAM!

 

I would like to stream Oculus Quest game(which I have been developing) to the PC side over Internet remotely. Is it possible? Can I have some similar tutorials on how to achieve this?

We would also like to enable talking, so that people wearing Oculus can talk with person in front of the PC. Do you have audio support?

You mentioned Photon is compatible with this package? I don't understand why you need Photon? And how can I place Photon with your package?

In addition, in our oculus quest game, one camera view is based on what avator sees, and we also would like to have another camera view to see the avator. Is it possible to stream both two camera views to the PC?

 

We may prepare Quest WebSocket tutorials in future, but you can always follow our current WebSocket setup.

We provided WebSocket examples for the stream over Internet. You need to host the node.js server yourself or rent any cloud hosting.

Photon is just an alternative. You can always use your own networking system instead of ours, as long as they support bytes streaming and follow below logic:
GameViewEncoder-> bytes-> Network Manager-> bytes-> GameViewDecoder

You can capture multiple camera views for streaming, with simply different Label IDs.

We have MicEncoder for audio streaming, and decode the stream using AudioDecoder on the receiver.

If you have good knowledge in node.js server hosting, you may use FM WebSocket with master controls of your game server. As alternative, it may save you some time if you are familiar with Photon in your existing projects.

But please be reminded, we don’t have Photon example scene and you will need to write some simple scripts to send your bytes data by following the networking logic mentioned above.

In case that you have trouble in setting up the scene, you can still reach us via email.
Technical support: thelghome@gmail.com

 

Thanks for your reply. In this video tutorial, you are testing it locally, however, I would like to have it stream through the Internet.

If I want to use Photon as Server, how should I implement it?

 

Other users can reach the node.js server via its public IP. For example, you can use port-forwarding in your home router.

All our Encoders will generate byte[], and you have to stream our byte[] via Photon, and use our Decoders on the receiver.
For Photon users, you may begin with

Code (CSharp):

1. stream.SendNext(bytes);

Code (CSharp):

1. bytes = (byte[])stream.ReceiveNext();

above just an example found online, you may need to refer to Photon libraries for example codes of streaming byte data.

Photon is actually kind of idea of node.js server hosting on cloud. But it's highly optimised for Gaming.

 

Hello
I get errors on Unity 2019.4.12f
Maybe I need to import a package ?


Library\PackageCache\com.unity.package-manager-ui@2.0.13\Editor\AssemblyInfo.cs(7,12): error CS0246: The type or namespace name 'UxmlNamespacePrefixAttribute' could not be found (are you missing a using directive or an assembly reference?)
and many more...

Cheers, stay safe

 

Seems likely there is something wrong with your package manager.
The best way to solve it -> Try "Untick" our project settings when you import our assets.

Reason: some errors occur, when our default project settings override yours.
*Our default project settings are simply adding demo scenes in build settings using Unity2018LTS.

 

Thanks for the fast answer
The problem is that the error is coming even before I can choose to untick the preferences.

In order I get this :

 

Since the errors are all related to package manager, I suggested that you should press "Skip" in the warning.

 

I think I found out !
At least no errors now :
If I press skip and then I import everything even the ProjectSettings it seems to work.


Thanks again for your fast answers.
PS: I wrote beore your answer, you are totally right

 

Another little question.
On Demo_FMNetworkBasic
When I do a Server on the Mac or on Windows (in Unity Editor) nobody can connect to it.
But when I do a server on the Iphone then Mac can connect to it. (Not Windows)

Summary :
- Mac client can connect to Win server
- Mac client can connect to Iphone server

- Iphone client can connect to Mac server
>>> Iphone client cannot connect to Win server <<<

- Win client can connect to Mac server
- Win client can connect to Iphone server

I guess there is some Networking permissions that I have tweak somewhere ?
- I did rules for unity editor in the firewall to make incoming and outgoing communication free.
- I accepted the local network permission on the Iphone :

 

Have your ever tried disabling firewall completely, instead of setting rules?
It seems most likely the UDP broadcast message is blocked, which is used for auto IP detection purpose.

In the worst case, you will need to type your server IP manually in FMNetworkManager's Client Settings.

 

Ok thanks good tip
I just noticed that the scenes are very basic do you provide scenes like the FMDemo_AR visible on the documentation ?

[EDIT]
I found scenes like ''Demo_FMNetworkStreaming'', I guess this is a similar base to then do the AR app on top with AR Foundation ?
Can you provide more "extended demo scenes" as well ?

I guess this is to keep the Asset flexible and not dependent from other third-party libraries that would be updated and could break your demo scenes etc...
Smart :-D

 

For AR scene with best performance, we suggested this:
1) create a RenderCam with GameViewEncoder
2) added TargetProjectionMatrix and referring to AR Cam

and the rest setup should be almost as same as our FMNetwork demo scenes.

We have Vuforia AR related tutorial:

The setup can actually apply for any VR or AR and HoloLens headsets.

 

Hi @thelghome, related to my previous question, we found the root of the issue. While trying to use UDP, we were still sending the byte arrays as single packets, which were way bigger than the recommended 1000-1500 packet size recommended for UDP. We implemented our own function to split the packages into pieces, and I looked at some of your scripts and saw you had something similar. I'll attribute this one to our lack of knowledge of network coding and standards, we're still figuring things out

 

Now the issue I have currently, is related to audio streaming (Desktop streaming works flawlessly now with UDP working properly). Because we have a setup with multiple different audio sources, I wanted to use just one single AudioDecoder and send data from that to all the audio sources.

At the moment I'm doing it by invoking an Action<AudioClip> loaded with the audioClip that's created in the decoder. However this is not working as I would expect, and from what I can tell that's because the AudioDecoder only runs correctly once, after which it stops. It seems like it's stopping in the CreateClip function, at the very first line;

Code (CSharp):

1. if (samplerate != (int)SourceSampleRate || channel != SourceChannels)

and when printing these values they are always 48 000/48 000 and 2/2. Would you have any clue what could be the problem?

EDIT: I see I've misunderstood the purpose of CreateClip, let me debug the callback functions instead and see what I can find!

 

there is a chunksize variable in our encoder, which you can adjust the size of each packet.

 

You can reduce the audio samplerate in your sender app, which can be adjusted via MicEncoder, or AudioSettings(under Project settings) for AudioEncoder.

If you have multiple audio streams, please be reminded that label ID should be different. Otherwise, Decoder may try decoding all streams from wrong sources as they shared same label ID.

 

Thank you, that's good to know! I suppose the same goes for GameDecoder/Encoder?

 

Yes, all encoder and decoder work as a pair of couple, with different label ID.

 

Alright, so I see the issue. What I want to do is to stream a windows desktop, with audio. But the AudioEncoder only encodes audio from within Unity. So if I play a video on my desktop, that sound is not encoded. Is there something I'm missing or this something you don't support?

 

Windows’ Audio is not supported yet, unfortunately.

 

Thank you for confirmation, I'll try to find an alternative

 

Hello, I want to implement the following functions with the Stream function of this asset. Please review whether it is possible.

1. We plan to record gameplay video from Android (Galaxy Tab s7 / Client) to PC (Windows10 / Server) and transmit the microphone of the video and the sound of the game in real time.

2. Android games must use ARCore. Could this asset and ARCore work together?

3-1. The PC will capture the video received in real time and save the video file.

3-2. Can I transfer the original video recorded on Android to the Server PC after streaming is finished?

There must be a video file. Which method is safer, 3-1 or 3-2? Or is there a better way to save the streamed video by sending it to the server?

 

When the <Capture Mode> is set to <Full Screen> from Android to Windows 10, the sound quality of the microphone streamed from Windows 10 is severely noise.

How can I fix it?

 

1) opt1: You can either setup MicEncoder and AudioEncoder(for game sound) with different Label ID. You will need two decoders on the other side.
opt2: You can playback the Mic in your Game using Unity's example, and capture everything including In-Game sound using AudioEncoder. This workflow can reduce the network traffic.

2) ARCore should work, we recommended that you should setup a separated render cam and clone the projection matrix of your main cam. However, you can also simply capture the result using MainCam Mode or Full Screen Mode in GameViewEncoder.

3) FMETP STREAM doesn't include the function of saving video file, thus, I suggested that you should use other asset to save your video as mp4 or mjpeg on PC instead of phone.

 

In general, the performance in these Modes(Render Cam mode is recommended):
Render Cam > Main Cam > FULL SCREEN

The audio may or may not related to GameViewEncoder, but you may compare the audio result in different render cam mode.
In case that you have quality issue on audio, please try to reduce the sample rate of Mic, or Audio Quality Settings in Unity. You may email us for technical support if necessary: thelghome@gmail.com

 

working with Unity WebGL??

 

Yes, it supports Unity WebGL.
You may refer to this old demo:

 

I came to the forum because I get the exact same errors importing the asset into a new blank project in Unity 2019.4.9f1. I think you guys must have accidentally included some stuff in the plugin that is incompatible with it. I urge you to test it from a scratch Unity 2019.4.x version.

 

Sorry for any inconvenience of importing in this version. We would definitely submit different version of package to store in coming few days.

As all assets are submitted via Asset Store plugin with Unity 2018 LTS, they might have changed some settings during our submission. We didn't intend adding any package dependencies. Thus, we will surely submit another version for Unity 2019.

In current version, please Press "Skip" when there is any warning related to package dependencies, which suggested by @satolas.

 

Thanks for the reply. Yes, it did work when I chose skip and unchecked all the project settings folder. I just wanted to let you know that it wasn't just one persons screwed up settings but more likely a wider problem with your asset with 2019.4.

One thing I'd request as a developer, though, is if you could clean up your warnings. There are 35 warnings, a lot of which are about inconsistent line ends but other things are stuff like variables declared/assigned but not used. Those are easy things to fix on your side, and they make it nicer for developers because they can concentrate on only their own warnings.

 

Thanks for your suggestion, we just submitted an update for these changes in v1.328.
-Cleanup warnings, and exclude dependencies for the ease of import

PS: We used to work on Mac OS, and we will keep checking those warning on Win10 once v1.328 available on store asap.

 

Hi,

I'm developing a VR application for Oculus Quest2, there will be a '''screen sharing'' feature in my app where the user is able to overlay their desktop screen to a e.g. tv screen in my VR app. Is this possible with this plugin?

 

Thanks for reaching out to us.
You can stream your Win10 Desktop screen into your Quest 2. It's one of our experimental features.

 

That's amazing to hear, thank you!