Search Unity

  1. Welcome to the Unity Forums! Please take the time to read our Code of Conduct to familiarize yourself with the forum rules and how to post constructively.
  2. Join us on Dec 8, 2022, between 7 am & 7 pm EST, in the DOTS Dev Blitz Day 2022 - Q&A forum, Discord, and Unity3D Subreddit to learn more about DOTS directly from the Unity Developers.
    Dismiss Notice
  3. Have a look at our Games Focus blog post series which will show what Unity is doing for all game developers – now, next year, and in the future.
    Dismiss Notice

[Release] FMETP STREAM: All-in-One GameView+Audio Stream (UDP/TCP/WebSockets/HTML)

Discussion in 'Assets and Asset Store' started by thelghome, Apr 30, 2019.

  1. Flamacore

    Flamacore

    Joined:
    Dec 17, 2013
    Posts:
    133
    Oh. Ok I found the issue. Seems like it's about HTTPS (or rather wss). I'll see about that. Thanks for the answers tho! Appreciate it

    Edit: A question, does the server zip file you provide support wss? I'm not sure if it does...
     
  2. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    597
    the demo node.js server from zip is a minimal demo for local hosting, as an example purpose.

    You have to modify it for wss need. Since there are many different tricky solutions for hosting, we only focus on Unity3D support with minimal node.js demo. But you can surely contact us via email, if you need more help or advice.
    technical support: thelghome@gmail.com
     
  3. Flamacore

    Flamacore

    Joined:
    Dec 17, 2013
    Posts:
    133
    I found out that by default, when ssl is enabled, the port value is not passed to the url request in WebGL build. Is this intentional or considered a bug?
     
  4. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    597
    We've received your email and thanks for pointing out this issue, seems likely a bug.
    We will investigate this issue, and we may need more info from you via email.
     
  5. olemidt1

    olemidt1

    Joined:
    Mar 22, 2018
    Posts:
    11
    Hello,

    We are using your asset for streaming desktop/game view over a multiplatform program we're making. We did a prototype first and it worked great - it's awesome how easy it is to set up :)

    The issue I'm having now is that we're trying to use UDP instead of TCP, and this has caused issues with the streaming. For context we are using TNet for our networking. I'm looking into it further, cause I think most of the problem is mine, but where it stops right now is in in GameViewDecoder, under Action_ProcessImageData. I'm printing right before this if check:
    Code (CSharp):
    1. if (receivedLength == dataLength)
    and I'm seeing very different numbers between the two. dataLentgth being in the 50-60000 ranger, but receiveLength closer to 1000. This means in turn that ProcessImageData is not called.

    I don't have a very thorough understanding of what's going on with the unpacking of the byte array, so I come here for help :) Do you have any ideas what might be causing this discrepancy?

    Ole
     
  6. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    597
    In our Encoder, we basically split bytes data into small chunks for UDP stream, and we found this method does optimise the network performance a bit too.

    In the Decoder, it simply collects all chunks into one complete byte array for decode.

    I am not familiar with TNet, but as long as you follow below structure, it should fit almost any kind of networking system.
    Encoder -> bytes -> Network manager -> bytes -> Decoder

    We suggested that you should test from our UDP demo first, which may give you some idea.
    There might be some chance that you have wrong setup, or interfering by duplicated label ID for different encoder/decoder. We would still like to help, and you are welcome to write us email for technical support: thelghome@gmail.com
     
    olemidt1 likes this.
  7. Barritico

    Barritico

    Joined:
    Jun 9, 2017
    Posts:
    363
    I recognize that I am lost.

    I have a VPS on windows.
    I have installed node, socket.io, express, etc ...

    I have compiled the examples they provide. I open it on the server and there is a connection. The command "node index.js" opens "listening 3000" perfectly and everything on the server works fine.

    The problem is when I put a client (my computer, for example) to connect with that server. I suppose that if I open the client on a computer (not on the server) as a client and I introduce the IP of the server (which is open and connected) a communication should take place. Is that so?

    Everything works perfectly locally, but I have not found a way to do it through a server. I have tried with windows and with linux and there is no way. So I am doing something wrong.

    I have opened ports, removed firewall .... but nothing.

    I've been in for many hours (and test server costs) and haven't found the solution. I have tried it. So I have no choice but to turn to you. The same is something very simple that I am not able to see.

    Thank you.
     
  8. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    597
    It seems likely there might be some settings issue on either your VPS, or maybe you need some modification on our node.js demo.

    There was a very kind suggestion by one of reviewers, I try to repost his solution. It might or might not be related, but it may give you some idea.
    ref: https://assetstore.unity.com/packages/tools/network/fm-websocket-155542#reviews
    Screenshot 2020-10-14 at 9.18.26 PM.png

    According to different feedbacks from customers, they always have different settings, or some modification based on our demo node.js script. Because there are just too many different setups of server hosting, and we can only provide the minimal local hosting example.

    Thus, in any case that you are struggling, please feel free to email us. We are always happy to share idea and give advice.
    Technical support: thelghome@gmail.com
     
  9. Barritico

    Barritico

    Joined:
    Jun 9, 2017
    Posts:
    363

    I'm going to try it by other means, to see what I get.

    The communication through UDP and TCP I have it solved for other software that I developed and I suppose it will be similar. After all, it's C #.

    Thank you anyway.

    A greeting.
     
  10. Flamacore

    Flamacore

    Joined:
    Dec 17, 2013
    Posts:
    133
    So here's my next question.

    Do you have any kind of buffering implementation? To eliminate any stutter and/or to be able to transfer large screens or multiple videos?
     
  11. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    597
    In order to keep the lowest latency, we didn't implement any buffer.
    We understood your concerns, we may consider your idea for our future development as an optional feature in settings.
     
  12. Flamacore

    Flamacore

    Joined:
    Dec 17, 2013
    Posts:
    133
    Alrighty then. Thanks for the answer. I think I'll implement my own algorithm for that then. With an acceptable latency, more fluid flow would be preferable for a number of reasons.
     
    thelghome likes this.
  13. olemidt1

    olemidt1

    Joined:
    Mar 22, 2018
    Posts:
    11
    If I want to pause a stream temporarily, is the best way to disable the component?
     
  14. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    597
    Yes, the simplest way is disabling the Encoder. It will resume when you enable it.
     
  15. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    597
    [Share]
    We've already tested on Quest 2, it works well with FMETP STREAM!
    1dc9338f-2421-42a0-83f1-796fb83252b7.jpeg
     
    olemidt1 likes this.
  16. jerermyzhou

    jerermyzhou

    Joined:
    Aug 24, 2020
    Posts:
    4
    I would like to stream Oculus Quest game(which I have been developing) to the PC side over Internet remotely. Is it possible? Can I have some similar tutorials on how to achieve this?

    We would also like to enable talking, so that people wearing Oculus can talk with person in front of the PC. Do you have audio support?

    You mentioned Photon is compatible with this package? I don't understand why you need Photon? And how can I place Photon with your package?

    In addition, in our oculus quest game, one camera view is based on what avator sees, and we also would like to have another camera view to see the avator. Is it possible to stream both two camera views to the PC?
     
  17. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    597
    We may prepare Quest WebSocket tutorials in future, but you can always follow our current WebSocket setup.


    We provided WebSocket examples for the stream over Internet. You need to host the node.js server yourself or rent any cloud hosting.

    Photon is just an alternative. You can always use your own networking system instead of ours, as long as they support bytes streaming and follow below logic:
    GameViewEncoder-> bytes-> Network Manager-> bytes-> GameViewDecoder

    You can capture multiple camera views for streaming, with simply different Label IDs.

    We have MicEncoder for audio streaming, and decode the stream using AudioDecoder on the receiver.

    If you have good knowledge in node.js server hosting, you may use FM WebSocket with master controls of your game server. As alternative, it may save you some time if you are familiar with Photon in your existing projects.

    But please be reminded, we don’t have Photon example scene and you will need to write some simple scripts to send your bytes data by following the networking logic mentioned above.

    In case that you have trouble in setting up the scene, you can still reach us via email.
    Technical support: thelghome@gmail.com
     
  18. jerermyzhou

    jerermyzhou

    Joined:
    Aug 24, 2020
    Posts:
    4
    Thanks for your reply. In this video tutorial, you are testing it locally, however, I would like to have it stream through the Internet.

    If I want to use Photon as Server, how should I implement it?
     
  19. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    597
    Other users can reach the node.js server via its public IP. For example, you can use port-forwarding in your home router.

    All our Encoders will generate byte[], and you have to stream our byte[] via Photon, and use our Decoders on the receiver.
    For Photon users, you may begin with
    Code (CSharp):
    1. stream.SendNext(bytes);
    Code (CSharp):
    1. bytes = (byte[])stream.ReceiveNext();
    above just an example found online, you may need to refer to Photon libraries for example codes of streaming byte data.

    Photon is actually kind of idea of node.js server hosting on cloud. But it's highly optimised for Gaming.
     
  20. satolas

    satolas

    Joined:
    Feb 13, 2017
    Posts:
    21
    Hello :)
    I get errors on Unity 2019.4.12f
    Maybe I need to import a package ?
    upload_2020-10-22_18-13-14.png

    Library\PackageCache\com.unity.package-manager-ui@2.0.13\Editor\AssemblyInfo.cs(7,12): error CS0246: The type or namespace name 'UxmlNamespacePrefixAttribute' could not be found (are you missing a using directive or an assembly reference?)
    and many more...


    Cheers, stay safe
     

    Attached Files:

    Last edited: Oct 22, 2020
  21. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    597
    Seems likely there is something wrong with your package manager.
    The best way to solve it -> Try "Untick" our project settings when you import our assets.

    Reason: some errors occur, when our default project settings override yours.
    *Our default project settings are simply adding demo scenes in build settings using Unity2018LTS.
     
    satolas likes this.
  22. satolas

    satolas

    Joined:
    Feb 13, 2017
    Posts:
    21
    Thanks for the fast answer :)
    The problem is that the error is coming even before I can choose to untick the preferences.

    In order I get this :
    upload_2020-10-22_18-42-11.png
    upload_2020-10-22_18-42-20.png
    upload_2020-10-22_18-42-30.png
     
  23. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    597
    Since the errors are all related to package manager, I suggested that you should press "Skip" in the warning.
     
    satolas likes this.
  24. satolas

    satolas

    Joined:
    Feb 13, 2017
    Posts:
    21
    I think I found out !
    At least no errors now :
    If I press skip and then I import everything even the ProjectSettings it seems to work.
    upload_2020-10-22_18-50-9.png

    Thanks again for your fast answers.
    PS: I wrote beore your answer, you are totally right :)
     
    thelghome likes this.
  25. satolas

    satolas

    Joined:
    Feb 13, 2017
    Posts:
    21
    Another little question. :)
    On Demo_FMNetworkBasic
    When I do a Server on the Mac or on Windows (in Unity Editor) nobody can connect to it.
    But when I do a server on the Iphone then Mac can connect to it. (Not Windows)

    Summary :
    - Mac client can connect to Win server
    - Mac client can connect to Iphone server

    - Iphone client can connect to Mac server
    >>> Iphone client cannot connect to Win server <<<

    - Win client can connect to Mac server
    - Win client can connect to Iphone server

    I guess there is some Networking permissions that I have tweak somewhere ?
    - I did rules for unity editor in the firewall to make incoming and outgoing communication free.
    - I accepted the local network permission on the Iphone :
    upload_2020-10-22_19-30-56.png
     
    Last edited: Oct 22, 2020
  26. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    597
    Have your ever tried disabling firewall completely, instead of setting rules?
    It seems most likely the UDP broadcast message is blocked, which is used for auto IP detection purpose.

    In the worst case, you will need to type your server IP manually in FMNetworkManager's Client Settings.
     
    satolas likes this.
  27. satolas

    satolas

    Joined:
    Feb 13, 2017
    Posts:
    21
    Ok thanks good tip :)
    I just noticed that the scenes are very basic do you provide scenes like the FMDemo_AR visible on the documentation ?

    [EDIT]
    I found scenes like ''Demo_FMNetworkStreaming'', I guess this is a similar base to then do the AR app on top with AR Foundation ?
    Can you provide more "extended demo scenes" as well ?

    I guess this is to keep the Asset flexible and not dependent from other third-party libraries that would be updated and could break your demo scenes etc...
    Smart :-D
     
    Last edited: Oct 22, 2020
  28. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    597
    For AR scene with best performance, we suggested this:
    1) create a RenderCam with GameViewEncoder
    2) added TargetProjectionMatrix and referring to AR Cam

    and the rest setup should be almost as same as our FMNetwork demo scenes.

    We have Vuforia AR related tutorial:


    The setup can actually apply for any VR or AR and HoloLens headsets.
     
    satolas likes this.
  29. olemidt1

    olemidt1

    Joined:
    Mar 22, 2018
    Posts:
    11
    Hi @thelghome, related to my previous question, we found the root of the issue. While trying to use UDP, we were still sending the byte arrays as single packets, which were way bigger than the recommended 1000-1500 packet size recommended for UDP. We implemented our own function to split the packages into pieces, and I looked at some of your scripts and saw you had something similar. I'll attribute this one to our lack of knowledge of network coding and standards, we're still figuring things out :)
     
  30. olemidt1

    olemidt1

    Joined:
    Mar 22, 2018
    Posts:
    11
    Now the issue I have currently, is related to audio streaming (Desktop streaming works flawlessly now with UDP working properly). Because we have a setup with multiple different audio sources, I wanted to use just one single AudioDecoder and send data from that to all the audio sources.

    At the moment I'm doing it by invoking an Action<AudioClip> loaded with the audioClip that's created in the decoder. However this is not working as I would expect, and from what I can tell that's because the AudioDecoder only runs correctly once, after which it stops. It seems like it's stopping in the CreateClip function, at the very first line;
    Code (CSharp):
    1. if (samplerate != (int)SourceSampleRate || channel != SourceChannels)
    and when printing these values they are always 48 000/48 000 and 2/2. Would you have any clue what could be the problem?

    EDIT: I see I've misunderstood the purpose of CreateClip, let me debug the callback functions instead and see what I can find!
     
    Last edited: Oct 23, 2020
  31. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    597
    there is a chunksize variable in our encoder, which you can adjust the size of each packet.
     
  32. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    597
    You can reduce the audio samplerate in your sender app, which can be adjusted via MicEncoder, or AudioSettings(under Project settings) for AudioEncoder.

    If you have multiple audio streams, please be reminded that label ID should be different. Otherwise, Decoder may try decoding all streams from wrong sources as they shared same label ID.
     
  33. olemidt1

    olemidt1

    Joined:
    Mar 22, 2018
    Posts:
    11
    Thank you, that's good to know! I suppose the same goes for GameDecoder/Encoder?
     
  34. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    597
    Yes, all encoder and decoder work as a pair of couple, with different label ID.
     
  35. olemidt1

    olemidt1

    Joined:
    Mar 22, 2018
    Posts:
    11
    Alright, so I see the issue. What I want to do is to stream a windows desktop, with audio. But the AudioEncoder only encodes audio from within Unity. So if I play a video on my desktop, that sound is not encoded. Is there something I'm missing or this something you don't support?
     
  36. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    597
    Windows’ Audio is not supported yet, unfortunately.
     
  37. olemidt1

    olemidt1

    Joined:
    Mar 22, 2018
    Posts:
    11
    Thank you for confirmation, I'll try to find an alternative
     
  38. Lordmin

    Lordmin

    Joined:
    Mar 9, 2017
    Posts:
    62
    Hello, I want to implement the following functions with the Stream function of this asset. Please review whether it is possible.

    1. We plan to record gameplay video from Android (Galaxy Tab s7 / Client) to PC (Windows10 / Server) and transmit the microphone of the video and the sound of the game in real time.

    2. Android games must use ARCore. Could this asset and ARCore work together?

    3-1. The PC will capture the video received in real time and save the video file.

    3-2. Can I transfer the original video recorded on Android to the Server PC after streaming is finished?

    There must be a video file. Which method is safer, 3-1 or 3-2? Or is there a better way to save the streamed video by sending it to the server?
     
  39. Lordmin

    Lordmin

    Joined:
    Mar 9, 2017
    Posts:
    62
    When the <Capture Mode> is set to <Full Screen> from Android to Windows 10, the sound quality of the microphone streamed from Windows 10 is severely noise.

    How can I fix it?
     
  40. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    597
    1) opt1: You can either setup MicEncoder and AudioEncoder(for game sound) with different Label ID. You will need two decoders on the other side.
    opt2: You can playback the Mic in your Game using Unity's example, and capture everything including In-Game sound using AudioEncoder. This workflow can reduce the network traffic.

    2) ARCore should work, we recommended that you should setup a separated render cam and clone the projection matrix of your main cam. However, you can also simply capture the result using MainCam Mode or Full Screen Mode in GameViewEncoder.

    3) FMETP STREAM doesn't include the function of saving video file, thus, I suggested that you should use other asset to save your video as mp4 or mjpeg on PC instead of phone.
     
  41. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    597
    In general, the performance in these Modes(Render Cam mode is recommended):
    Render Cam > Main Cam > FULL SCREEN

    The audio may or may not related to GameViewEncoder, but you may compare the audio result in different render cam mode.
    In case that you have quality issue on audio, please try to reduce the sample rate of Mic, or Audio Quality Settings in Unity. You may email us for technical support if necessary: thelghome@gmail.com
     
  42. Tushar_kmphasis

    Tushar_kmphasis

    Joined:
    Oct 5, 2019
    Posts:
    7
    working with Unity WebGL??
     
  43. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    597
    Yes, it supports Unity WebGL.
    You may refer to this old demo:
     
  44. virtualjay

    virtualjay

    Joined:
    Aug 4, 2020
    Posts:
    68
    I came to the forum because I get the exact same errors importing the asset into a new blank project in Unity 2019.4.9f1. I think you guys must have accidentally included some stuff in the plugin that is incompatible with it. I urge you to test it from a scratch Unity 2019.4.x version.
     
  45. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    597
    Sorry for any inconvenience of importing in this version. We would definitely submit different version of package to store in coming few days.

    As all assets are submitted via Asset Store plugin with Unity 2018 LTS, they might have changed some settings during our submission. We didn't intend adding any package dependencies. Thus, we will surely submit another version for Unity 2019.

    In current version, please Press "Skip" when there is any warning related to package dependencies, which suggested by @satolas.
     
  46. virtualjay

    virtualjay

    Joined:
    Aug 4, 2020
    Posts:
    68
    Thanks for the reply. Yes, it did work when I chose skip and unchecked all the project settings folder. I just wanted to let you know that it wasn't just one persons screwed up settings but more likely a wider problem with your asset with 2019.4.

    One thing I'd request as a developer, though, is if you could clean up your warnings. There are 35 warnings, a lot of which are about inconsistent line ends but other things are stuff like variables declared/assigned but not used. Those are easy things to fix on your side, and they make it nicer for developers because they can concentrate on only their own warnings.
     
  47. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    597
    Thanks for your suggestion, we just submitted an update for these changes in v1.328.
    -Cleanup warnings, and exclude dependencies for the ease of import

    PS: We used to work on Mac OS, and we will keep checking those warning on Win10 once v1.328 available on store asap.
     
  48. adaylmz

    adaylmz

    Joined:
    May 6, 2015
    Posts:
    2
    Hi,

    I'm developing a VR application for Oculus Quest2, there will be a '''screen sharing'' feature in my app where the user is able to overlay their desktop screen to a e.g. tv screen in my VR app. Is this possible with this plugin?
     
  49. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    597
    Thanks for reaching out to us.
    You can stream your Win10 Desktop screen into your Quest 2. It's one of our experimental features.
     
  50. adaylmz

    adaylmz

    Joined:
    May 6, 2015
    Posts:
    2
    That's amazing to hear, thank you!
     
    thelghome likes this.