Search Unity

  1. Welcome to the Unity Forums! Please take the time to read our Code of Conduct here to familiarize yourself with the rules and how to post constructively.

  2. Unity 2022.1 is now available as the latest Tech release.
    Dismiss Notice
  3. Improve your project's performance with our new guide on profiling in Unity.
    Dismiss Notice

[Release] FMETP STREAM: All-in-One GameView+Audio Stream (UDP/TCP/WebSockets/HTML)

Discussion in 'Assets and Asset Store' started by thelghome, Apr 30, 2019.

  1. david-mal

    david-mal

    Joined:
    Nov 27, 2017
    Posts:
    6
    Hello,

    sorry if this is a duplicate post, I would like to extend the previous question of Lordmin.

    How can I achieve to send a message to a specific client, but not the ip but maybe an clientSocketID? This would be more flexible as we could test on the localhost (all clients have the same locale IP address)

    For example, after client A connects, A would send a broadcast to others, that he/she is now connected including the user's socketID. After that every other client would need to send back a short byte[] to the client A directly to the socketID. Is there an easy solution in FMNetworkManager for this kind of demand or is it more related to the server side? Any suggestions on how to do this?

    We are just about to figure things out with socket.io and your plugin. Thank you! David
     
    Last edited: Apr 30, 2021
  2. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    494
    FMNetworkManager doesn't have socketID, which is simply a pure UDP solution.
    For your purpose, I think you need to modify our FMWebSocket's node.js server script. Meanwhile, you have to add some meta data like socketID for the stream on socket.io server side.
     
    david-mal likes this.
  3. GordonHo

    GordonHo

    Joined:
    Feb 1, 2021
    Posts:
    1
    Hi,

    is there an example of sending a WebcamTexture directly?
     
  4. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    494
    This will be available in coming update v1.375.
    TextureEncoder will support WebcamTexture input in this version.
     
  5. tefrt

    tefrt

    Joined:
    Sep 10, 2019
    Posts:
    3
    Hello,

    Thank you! By following your instruction I succeed in streaming real-world camera's view on Hololens2. I use an additional camera to capture both the holograms and webcam texture rendered on raw image in canvas.
    However, when moving the hololens around the holograms will not stay in the same relative position according to the real-world view. There is a offset between the real-world view and the holograms. Is the TargetProjectionMatrix.cs can fix this issue? how should I use it.
     
  6. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    494
    In general, you can assign this TargetProjectMatrix script to your render cam. There is an option of referring to your MainCam(AR). Then, you should be able to correct the offset issue for AR. However, according to our schedule, we haven't done further test on HL2 for a while. If you have any screenshots or other concerns, please feel free to write us an email for tech sup: thelghome@gmail.com
     
  7. titoasty

    titoasty

    Joined:
    Dec 2, 2018
    Posts:
    21
    Hi,

    I have a question regarding FM WebSocket : can you make a server from Unity?
    Or is it only a client?
    Thanks
     
  8. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    494
    Our FMWebsocket demo is based on node.js server hosting.
    You may consider embedding it into Unity build. However, we didn't try this solution yet, just an idea for you.
     
  9. titoasty

    titoasty

    Joined:
    Dec 2, 2018
    Posts:
    21
    Ok I may consider it then :)
    And how is the support for socket.io? Is it ok with 4.0?
     
  10. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    494
    Our demo server is tested with socket.io 4.0.
    It works, but there is no guarantee for advanced specific 4.0 features. Overall, the basic should work fine.
     
  11. titoasty

    titoasty

    Joined:
    Dec 2, 2018
    Posts:
    21
    Is there any version it's guaranteed to work with ? I don't care the version.
    Otherwise I'll go with 4.0, I'll only use the basic features (probably wss though)
     
  12. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    494
    It’s developed based on socket.io v2.0.
    v2.3 is the most stable version, and our demo scene is compatible with v3.1 and above and v4.0.

    The only unsupported version is v3.0.
     
  13. titoasty

    titoasty

    Joined:
    Dec 2, 2018
    Posts:
    21
    Thanks a lot for your support, I'll get it then :)
    Can't wait to try it out!
     
    thelghome likes this.
  14. pdace

    pdace

    Joined:
    Mar 8, 2021
    Posts:
    3
    Hello, thanks for your Asset!
    I am using the SocketIOManager, the LargeFileEncoder/Decoder and the node js server to send files of around 20MB between multiple clients. I have the problem that the file doesn't always arrive at the client that shall receive. It works in ca. 50% of the attempts, but there are packages arriving at the client every attempt. (Currently testing on localhost) So for me it seems that the decoder can't handle the amount of packages that are arriving. I also noticed that when the file doesn't arrive successfully, the client reconnects to the server after some time.
    I already tried to adjust the "Add delay every package" parameter but it didnt seem to help.
    Do you have any advice?
    Thank you
    Philipp
     
  15. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    494
    It's a bit weird, as SocketIO is based on TCP protocol.
    Do you have sample project, for us to reproduce the bug you found?
    For the tech support, please feel free to write us email: thelghome@gmail.com
     
  16. titoasty

    titoasty

    Joined:
    Dec 2, 2018
    Posts:
    21
    Hi,

    I noticed that in some cases, OnReceiveData was skipped if messages were sent from server directly after connection.
    After some investigations, I found the problem with the function WaitForSocketIOConnected.
    It's sometimes called *after* OnReceiveData event is received and so is skipped.

    For now, I just replaced
    StartCoroutine(WaitForSocketIOConnected());

    (FMSocketIOManager line 350)
    with
    On("OnReceiveData", OnReceivedData);


    It seems to be ok now.
    Now I'm wondering if it's clean and why the WaitForSocketIOConnected exists at all.
     
  17. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    494
    For safety, we prefer registering the event listener after connection secured.
    As we found some bugs in Unity Editor mode, if we register the event listener immediately without checking the connection.
     
  18. Stamson

    Stamson

    Joined:
    Oct 31, 2020
    Posts:
    2
    For WebSocket usage, is there anything specific to javascript / socket.io library?

    For example, if I have a Golang server that follows the WebSocket protocol (upgrades and hijacks HTTP request, uses TCP), would that work as straightforward as it would be to set up with a socket.io server?
     
  19. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    494
    FMWebSocket supports raw websocket and socket.io server.
    However, we cannot guarantee other server types. You might need to test it out if you have specific setups.
     
  20. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    494
    Got M1 iMac few days ago, and here is our Live Streaming Latency test.
     
  21. devbit

    devbit

    Joined:
    Jan 9, 2019
    Posts:
    3
    Hi, we're having an issue with the screen share while running on notebooks, it is working nice and clean on PC desktop but when i try to share the screen on a notebook i just received a blank screen, i was worried if it was a connection issue then i tried to get the texture and put it on a Quad just to test the capture, again, working on PC Destktop but Blank Screen on Notebook.

    Can you help me?
     
  22. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    494
    This should be a known issue. It may not work well on discrete GPU.
    For your target monitor/screen, the temporary solution is switching between discrete gpu and integrated gpu in your notebook. This option should be under your GPU settings, or Win10's screen settings.
     
  23. devbit

    devbit

    Joined:
    Jan 9, 2019
    Posts:
    3
    So, the solution should be set manually, i just cant set it from within the app?
     
  24. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    494
    I'm afraid that there is no auto solution via Unity directly. It requires admin permission in general.
     
  25. devbit

    devbit

    Joined:
    Jan 9, 2019
    Posts:
    3
    Well, we tried all the options, to run with the nvidia and also with the intel HD graphics, also tried to set the option to let the windows decide but no luck. Neither options made it work
     
  26. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    494
    @devbit please refer to this old post, see if you have similar situations? The major issue is due to unsupported gpu, but it could be a bit tricky due to monitor setup. If you have any external monitor, please give it a try too.
     
  27. pdace

    pdace

    Joined:
    Mar 8, 2021
    Posts:
    3
    While trying to reproduce the error in another project I believe I discovered the problem. I am also using the OnReceivedRawMessageEvent in the FM Socket IO Messenger and was using too expensive operations for every package that arrived from the LargeFileEncoder so that the program was stuck for some seconds and the instance lost the connection.
    Now im checking if an arriving message starts with "42[On" to skip the Large File Packages when handeling RawMessageEvents and it seems to work well.

    While testing my program with three instances i encountered another error:
    Im sending a large byte[] from instance A by using SendToOthers. The first byte[] arrives at the other two instances (B and C). Then when sending a large byte[] from the next instance (B), the array only arrives at the instance that hasn't received a byte[] yet (A). The instance that already received a byte[] (C) received the individual small byte[] but it seems that the decoder can't assemble them.
    In a case where I first send from A two times and then from B one time the bytearrays all arrive as they should.

    In the project where I originally encountered this bug i also got an error message for each package in the case where the decoder could not assemble them:

    "System.ArgumentException: Offset and length were out of bounds for the array or count is greater than the number of elements from index to the end of the source collection.\r\n at System.Buffer.BlockCopy (System.Array src, System.Int32 srcOffset, System.Array dst, System.Int32 dstOffset, System.Int32 count) [0x00097] in <eae584ce26bc40229c1b1aa476bfa589>:0 \r\n at LargeFileDecoder.Action_ProcessData (System.Byte[] _byteData) [0x000a3] in C:\\Users\\phili\\Desktop\\Uni\\Bachelorarbeit\\gitlab-repo\\2021-ba-schaupp\\prototype\\AvatarEncryption\\Assets\\FM_ExhibitionToolPack\\FMNetwork\\Scripts\\LargeFileDecoder.cs:45 \r\n at UnityEngine.Events.InvokableCall`1[T1].Invoke (T1 args0) [0x00010] in <a5949084587241189a974c7df1839e1c>:0 \r\n at UnityEngine.Events.UnityEvent`1[T0].Invoke (T0 arg0) [0x00025] in <a5949084587241189a974c7df1839e1c>:0 \r\n at FMSocketIOManager.OnReceivedData (FMSocketIO.SocketIOEvent e) [0x00040] in C:\\Users\\phili\\Desktop\\Uni\\Bachelorarbeit\\gitlab-repo\\2021-ba-schaupp\\prototype\\AvatarEncryption\\Assets\\FM_ExhibitionToolPack\\FMWebSocket\\Scripts\\SocketIO\\FM_IO\\FMSocketIOManager.cs:154 \r\n at FMSocketIO.SocketIOComponent.EmitEvent (FMSocketIO.SocketIOEvent ev) [0x0003f] in C:\\Users\\phili\\Desktop\\Uni\\Bachelorarbeit\\gitlab-repo\\2021-ba-schaupp\\prototype\\AvatarEncryption\\Assets\\FM_ExhibitionToolPack\\FMWebSocket\\Scripts\\SocketIO\\FM_IO\\SocketIOComponent.cs:475"

    In the project where i reproduced the bug the program behaves the same but i don't get the exception.
    I will send you the project where i reproduced the bug via the tech support email.
    I tested the application with a Windows Standalone x86_64 built made with Unity version 2019.4 LTS and Api Compatibility Level .NET 4.x

    Thanks for your support in advance!
    Philipp
     
    Last edited: Jun 4, 2021
  28. pdace

    pdace

    Joined:
    Mar 8, 2021
    Posts:
    3
    Edit: It seems that the receivedLength in LargeFileDecoder is not set to 0 when different instances send files with the same dataID what causes that ProcessData is not called.
     
  29. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    494
    We just opened your example project, and we noticed that your decoders are same Label ID.
    As we split the big data into small chunks, it needs to assemble them for decode.

    However, if other clients are sending large byte[] at the same time to others, the decode will collect wrong data due to the same label ID from other clients.

    I suggested that you should assign different label ID for each pair of encoder/decoder.
     
  30. Lordmin

    Lordmin

    Joined:
    Mar 9, 2017
    Posts:
    62
    Hello,

    FMNetwork Server: PC (Windows 10, Standalone)
    FMNetwork Client: HL2 (UWP)
    The current UDP deployment environment is as above.

    Previously, I inquired because the UDP connection between the two devices did not work.
    It was solved in 'Place A (Wi-fi 5)' by the method you answered below. The USB connection for testing was a problem.
    -Do not connect your USB cable to your PC when running the app

    However, in addition, UDP connection is not working in other 'Place B (Wi-fi 6)'.
    -Try switching on/off "UseMainThreadSender" on all platform
    -Try remove all "AudioEncoder/MicEncoder", see if it's related to network traffic..etc

    Can you provide a little more detail about the above two methods you suggested?

    Currently applied "UseMainThreadSender" is Server(Off) / Client(On) status. How can I change it?

    The peculiar thing is that UDP connection is not working only in Unity Editor, and when running PC Build, UDP connection is working well.

    We are in a situation where we must make a UDP connection in the Unity Editor as well.

    Any problems you can guess?
     
  31. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    494
    The auto network discovery feature depends on UDP broadcasting. It could be some settings issue on Wi-fi 6 router.
    In v1.382, we fixed the found bug limit on iOS 14.5, due to the latest Apple UDP broadcasting Policy.
    Instead of broadcasting from client side, now both server and client will try broadcasting themselves in local network.
    This changes should improve other platforms too.

    Meanwhile, Could you please check if you have the latest version of FMETP STREAM?

    If your PC build works, it must be some hidden UDP blocked between Windows and Unity.
    On our side, we have no problem on editor connection. Of course, we turn off all firewall related stuff.
     
  32. Lordmin

    Lordmin

    Joined:
    Mar 9, 2017
    Posts:
    62

    Thank you, solved!

    Of course, we turn off all firewall related stuff.
    => Windows 10 => Network Profile Settings => Change Firewall from Public to Private
     
    thelghome likes this.
  33. naive_magic

    naive_magic

    Joined:
    Jun 5, 2020
    Posts:
    7
    Is it possible to let a user to control a FPS character with keyboard and mouse input over distance when running FMETP on Unity on a remote computer?
     
  34. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    494
    We have FMWebSocket demo scene for Internet Streaming. Both Server and Clients can communicate via string or byte[]. Thus, you can send commands remotely,
     
  35. Lordmin

    Lordmin

    Joined:
    Mar 9, 2017
    Posts:
    62
    The current device connection environment is as follows.

    1. Server (Windows10 Unity Editor)
    2. Client1 (HoloLens2 UWP Build)
    3. Client2 (HoloLens2 UWP Build)

    In this case, is it ok to run the program only with Unity editor in Windows 10 environment like in No. 1? (Does not build Windows 10)

    If I run the program itself with the Unity editor, will all functions work normally?
     
  36. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    494
    it should be as same as build app via Unity Editor.
     
  37. AntonioSierra

    AntonioSierra

    Joined:
    Jul 5, 2017
    Posts:
    1
    Congratulations on the asset, we are testing the demo scene with Win10 and hololens2, we see that in the Hololens console when the transmission is executed we get a continuous error that we cannot see at all but it says "Sync Gpu ...." what is the problem?
     
  38. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    494
    Not 100% sure, but you may try to disable the "Async GPU Readback" feature. It might be related, according to your error message.
    Screenshot 2021-06-30 at 10.20.20 PM.png
     
  39. Hiromichi-Yamada

    Hiromichi-Yamada

    Joined:
    Feb 16, 2013
    Posts:
    3
    Hi,

    FMETP is a great asset!
    But, it does not work with my URP Project.

    Attached GameViewEncoder to Main Camera GameObject, like a tutorial.

    I know it works with RenderTexture, or another camera.
    But, my project is for Oculus Quest 2,
    So, I don't like to use RenderTexture, another camera..

    Please tell me how to use with my URP Project (Oculus Quest 2).

    Unity 2020.1.14f1
    FMETP 1.383.0
     
  40. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    494
    It's compatible with URP main cam mode in most of cases.
    However, URP main cam mode is still incompatible with Oculus Quest 2 in the current version.

    In our testing, it seems some hardware specific issue on Quest 2, which couldn't return the frame buffer correctly from main camera. We are still investigating this problem.

    According to our testing result, RenderCam mode should perform better in frame rate as it doesn't need to wait for the main camera render.
     
    Hiromichi-Yamada likes this.
  41. Hiromichi-Yamada

    Hiromichi-Yamada

    Joined:
    Feb 16, 2013
    Posts:
    3
    Thank you!
    I will use RenderCam mode.
     
  42. BrawiX

    BrawiX

    Joined:
    Feb 2, 2018
    Posts:
    1
    Hello, does your (FTP STREAM CORE 60 $) work with callbacks from Node js server? Do you have any documentation or a diagram on listeners, I'm testing your asset, it is very different from all the others and this raises a lot of questions, in the rest you call socket.io create an instance and go and you have a bunch of managers, I need to modify or copy them and it is very inconvenient to rewrite it for yourself. Please answer to email iashakhov@gmail.com
     
  43. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    494
    FYI, the Emit() and On() function are all included in FMSocketIOManager.cs.
    You may call them from other script in this way:
    Code (CSharp):
    1. FMSocketIOManager.instance.Emit();
    2. FMSocketIOManager.instance.On();
    In case that you require technical support, please write us email: thelghome@gmail.com

    Screenshot 2021-07-06 at 3.37.45 PM.png
     
  44. kkloud

    kkloud

    Joined:
    Apr 22, 2021
    Posts:
    2
    Hi, I am running into an issue when trying to use FM Network UDP with a non-FM server. Are there certain pieces of information that the FM Network client needs in order to receive and decode bytes properly?

    Another team member has written a server in C++ that is broadcasting UDP messages. I need to pick them up on the Unity side and display them in an app. I have everything running perfect if I build a Server and Client from Unity with the FM Network, but if I just try to build a FM Network Client and use his server I get all 0's as the data coming in. We have the same message structure and offsets in place.

    I am just wondering if some of the network discovery settings or something in the FM Network side may only receive messages from FM Network Servers, and how I can modify that.

    Thanks,
    Kyle
     
  45. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    494
    You may refer to our FMServer.cs script for your C++ wrapper.
    In the first part of Sender(), which you will see that it requires a regular check of connection status from clients.

    As server is idle, it will try to send an empty byte to clients.
    For data structure, you have to match FMFPacket.

    In FMNetworkUDP, we only use broadcasting for network discovery.
    Once they found each others IPs, they will only send via IP.

    Overall, it's still pure UDP solution and we didn't limit the usage. You should be able to make it compatible with C++ referring to our C# codes.
     
  46. kkloud

    kkloud

    Joined:
    Apr 22, 2021
    Posts:
    2
    Thank you! Yes I am new to the networking world and wanted this tool to help with UDP. It is fantastic and your advice was correct. I just needed to actually look at how the FMPacket was being created. Then I was able to add the _meta of [0,0] to the front of the server data I was receiving so that the FM Client would read it properly as an FMPacket byte array and for all clients.

    Now I am communicating correctly and can use this with any server we produce.

    I really appreciate the help, I will leave a fantastic review.

    Thanks,
    Kyle
     
    thelghome likes this.
  47. einslyu1997

    einslyu1997

    Joined:
    Jun 7, 2021
    Posts:
    2
    hi,
    i have also use your packeage
    I can transmit real-time video to the PC, and can I control the mobile application on the PC?
    The server ("your PC") should send data to your AR app on your mobile phone.
    Programming of network interfaces which enable the HTML interface to interact with the augmented data. (Setting markers, showing and hiding text fields and components, marking components and displaying the corresponding PDF.
     
  48. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    494
    By default, both server and clients can communicate with each others via string or byte[].
    We have minimal javascript example for the html demo, but please be reminded that our main development is on Unity side.

    If you would like to add more web buttons via HTML, you may refer to our html demo. There are examples of how you can send string and byte[] via javascript functions. Other alternative would be WebGL build, which you can create buttons and info fully in Unity side without involving javascript.
     
  49. einslyu1997

    einslyu1997

    Joined:
    Jun 7, 2021
    Posts:
    2
    Thanks for your answer, could you give me the html demo?
     
  50. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    494
    It’s included in our TestServer zip file. It’s the same html demo when you run our FMWebsocket demo via web.
     
unityunity