Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

[Release] FMETP STREAM: All-in-One GameView + Audio Stream + Remote Desktop Control(UDP/TCP/Web)

Discussion in 'Assets and Asset Store' started by thelghome, Apr 30, 2019.

  1. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    737
    FMWebSocketManager is focusing on purely byte[] stream for the best performance, which json style events are excluded unfortunately. We prefer using "label ID" style for identify different data type.

    In theory, you may create your own data structure for it.
    For example, create your own encoder/decoder for "events(with data) in json string -> metadata bytes for identify your data structure + json data byte[]".

    We might consider adding this feature into our To-Do list, but probably not in high priority at the moment.
     
  2. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    737
    Could you remind us your use case? via MainCam stream, or RenderTexture Stream?
    We will see if we can push the priority of this feature, and embedding it into the GameViewEncoder option.

    You may also reach us via email for any urgent feature request: thelghome@gmail.com
     
  3. Airmouse

    Airmouse

    Joined:
    Jan 12, 2019
    Posts:
    107
    I'm using a GameView encoder with a custom stereo render texture.
     
    Last edited: Jan 21, 2023
  4. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    737
    It might be challenging to get accurate alpha result(0%~100% etc) without increasing streaming size.
    However, it might be possible if only boolean value needed, like reserving either 100% solid or 100% transparency.
     
  5. Airmouse

    Airmouse

    Joined:
    Jan 12, 2019
    Posts:
    107
    hey, I just updated my project from 2.0 to 3.0 and everything is working, however when I create a new script, edit a script or save or open my project I see a empty popup window that momentarily blocks my ability to click on the editor. Happens in all editors I tested. I import just the plugin into a new project and the blank window appears after each compile (any script). The popup appears to add extra time my compiler and prevents me from clicking in the project until the popup window closes, any ideas?
     
  6. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    737
    This shouldn't be an expected behaviour, could you please write us another email related to this bug?
     
  7. CodePoKE

    CodePoKE

    Joined:
    May 6, 2015
    Posts:
    23
    @thelghome

    Having delved into the code for performance, most performance issues in FMETP come from encoding-loop memory allocation and that SocketIO uses json based transmission in FMETP for binary (With the entire byte array being sent as JSON instead of a binary stream).

    However, SocketIO fully supports binary streaming and actually the same logic that you use in the WebSocketManager implementation can be used in the SocketIO manager. (Socket IO just wraps around the same library)

    I implore you to reconsider your approach because it forces us (and I assume others) to recode the entire backend and communication protocols making it very hard to onboard to FMETP 3.0 - to the point that I am reconsidering a refund for my organization because there is little point in it for me.

    As in - I am looking at the FMETP code now, and it would take me less work to recode the FMSocketIOManager to use binary streaming, than to reimplement our backend service to use WS instead of SocketIO.

    The list of work would be:
    - Ensure that `SocketIOComponent.EmitPacket()` uses `ws.send(byte[])` when it encounters binary data - which is already used in `FMWebSocket`
    - Ensure that `FMSocketIOManager.SenderCOR()` doesn't convert to JSON for binary data and forwards it.
    - Ideally, you forward the message encoding as well, because you are double allocating every string message - once with the `JSONUtility`, and once with your encoder in `SocketIOComponent`)
    - Remove the allocations in general on the overall sending thread. (E.g.: lambda in while loop is allocating, the byte[] in the encoder for every run is allocating)

    The above would solve 90% of the performance issues with the SocketIO implementation.

    (I would highly recommend using this plugin if you want to an optimization pass over your allocations: https://plugins.jetbrains.com/plugin/9223-heap-allocations-viewer )
     
  8. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    737
    We will try our best to investigate your suggested methods, though the "On(event name, data)" function may still likely require json parse in C# part for customised events in socket.io server.

    Otherwise, they are simply raw byte[] stream like basic WebSocket, and there is no benefit of using socket.io.

    We will do our best to cover all the use cases for everyone, but please be reminded that you should take your own responsibility for any customised backend server.
     
  9. natoy7000

    natoy7000

    Joined:
    Oct 6, 2020
    Posts:
    3
    Hello. Is it possible to make communication between two devices through your asset (audio)?
    I tried to do something, but the problem is in the echo.
    I tried to solve it through unity.
    upload_2022-11-18_10-10-58.png
    in two ways
    1) Mute
    2) AudiMixer = -80
    But the steam audio stops going.

    In the documentation you have only about the components (AudioEncoder, AudioDecoder)
    But I also tried to do something
    upload_2022-11-18_10-15-13.png
    the echo is gone, the steam audio is going well.
    but it no longer receives sound from another device (
     

    Attached Files:

  10. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    737
    Let me quote my email reply here too:
    Please refer to our example scene, which you may enable mic stream between two devices as voice call like.

    If you are in the same room with two devices, it might have echo due to sound physics.

    If you enable audio encoder, it might have echo if you playback others’ mic input via AudioDecoder.

    The scene name is “Demo_FMNetworkStreamingMainCam”
    Ref: https://frozenmist.com/docs/apis/fmetp-stream/
     
  11. Airmouse

    Airmouse

    Joined:
    Jan 12, 2019
    Posts:
    107
    Hello, I entered my purchase invoice in the license thing, but I still see the FMETP advertisement popup every time I compile any script even unrelated scripts. Can you please tell me how to remove your popup or disable it from constantly appearing? thank you
     
  12. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    737
    We've just released a quick fix v3.250 for the Windows 10 Editor bug, please give it a try and see if it's been solved.
    Sorry for any inconvenience caused.
     
  13. Airmouse

    Airmouse

    Joined:
    Jan 12, 2019
    Posts:
    107
    Hello, I already did a lot of work upgrading from v2 to v3 and if I try to update to v3.250 all of my custom scripts will be overwritten. Do I need to replace all scripts to stop the popup?
     
    Last edited: Nov 21, 2022
  14. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    737
    You may skip the encoders, if you've been already customised them.
    Meanwhile, we also noticed that there is a Unity package manager, which it doesn't import the latest version though you've been downloaded them. (for example, in some version of Unity Editor, you may still importing the older version of FMETP STREAM 3.xx, though it shows the latest 3.250 in package manager)

    In case that your issue still exists, please reach us via email and I will send you the latest package.
    and Hopefully Unity Asset Store team should fix it on the end soon.
     
  15. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    737
    [v3.320] Updates: HTML5 Mic Capture and Stream Example
    -added "TestServer_v2.2.2.zip", with HTML Mic Stream(Experiment Feature)

    It's been requested by many users for a long while, we've finally included html5 mic stream solution with minimal javascript. Hopefully it may help others.
     
  16. moliminous

    moliminous

    Joined:
    Oct 16, 2015
    Posts:
    70
    Hey there, I want to buy this, I think it will do what I need. I need my unity scene to be played in OBS on a mother computer. I don't need any overlays or anything just as close to one to one as possible. I got the cheaper 2.0 just to try but I'm having trouble getting it to obs. That index page has a bunch of overlays, and connecting to obs via a web adress doesn't work since it shows the tool bar, but also needs to be connected by clicking connect. Obs will display a web page, but you can't interact with said webpage. Is there a way I can get just the raw video on the adress so anything that goes there will just see the video, with no need to press connect?
     
  17. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    737
    Those html5 buttons for connection are just for demo purpose, you can actually remove them by modifying our example "index.html" under "public" folder.

    Those example buttons are just triggering our example javascript functions inside the html file.
    You may set them auto connect and remove those unnecessary UIs. It won't be too difficult as long as you have basic html5 experience.
     
  18. moliminous

    moliminous

    Joined:
    Oct 16, 2015
    Posts:
    70
    Is there a pre done just blank example? I'm familiar with html but I've never really touched 5, I just need a quick proof of concept so I can see if obs will even take it. It just needs the video and sound and nothing else
     
  19. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    737
    Do you mean something like this? capturing web browser via OBS? Then the answer is "YES" for both audio and video.
    Screenshot 2022-12-05 at 4.07.38 AM.png

    We've just pushed an update(v3.322), with optimised HTML part. You should be able to remove UIs with ease by following below steps.

    1) remove the highlighted html5 scripts
    Code (CSharp):
    1. //<!-- Region Begin: You may commented and remove below script, if you don't need any Demo UI buttons, input Text or any info -->
    2.     <h2 style="text-align:center"> FMETP STREAM: FM WebSocket 2.0 (Demo Receiver)</h2>
    3. ...
    4. ...
    5. //<!-- Region End: You may commented and remove below script, if you don't need any Demo UI buttons, input Text or any info -->
    6.  
    2) in javascript part, we added one more variable for your need, default was "false", please change it to "true" instead.
    Code (JavaScript):
    1. //You may set "canAutoConnectAsClient" to true, and it will automatically connect as client;
    2. var canAutoConnectAsClient = true;
     
    Last edited: Dec 4, 2022
  20. moliminous

    moliminous

    Joined:
    Oct 16, 2015
    Posts:
    70
    That looks like it does everything i need nice. I'll buy v3 and give it a shot. I appreciate your help!
     
  21. moliminous

    moliminous

    Joined:
    Oct 16, 2015
    Posts:
    70
    so isee the highlighted code
    Code (CSharp):
    1.  <th><button onclick=FMStartMicStream('SendToAll')><Strong id="BtnMicTextAll">Mic To All</Strong></button></th>
    2.           <th><button onclick=FMStartMicStream('SendToServer')><Strong id="BtnMicTextServer">Mic To Server</Strong></button></th>
    3.           <th><button onclick=FMStartMicStream('SendToOthers')><Strong id="BtnMicTextOthers">Mic To Others</Strong></button></th>
    but I don't see the var?
    var canAutoConnectAsClient
    also the notes say it's file 2.2.2 but the one we have is 2.2.3 I'm guessing that's not a big deal though
     
  22. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    737
    Sorry about this, we forgot adding that variable in the zip after our rush test.
    We've now pushed another small update V3.323 for your need.

    Meanwhile, that variable is just for the if-else condition of calling "ConnectAsClient();", fyi.
    Code (JavaScript):
    1. if(canAutoConnectAsClient) ConnectAsClient();
     
  23. dawnr23

    dawnr23

    Joined:
    Dec 5, 2016
    Posts:
    41
    When performing UDP communication on the Android platform, we confirmed that the connection is disconnected when the app is minimized or the screen is locked (unfocused).

    Therefore, I would like to add the following code to reconnect to the FMNetworkManager.cs script. Please review the code and reply if it's okay.

    (+ The code I wrote works on my Galaxy tablet, but not on my Oculus Quest 2. Could there be a cause?)

    FMNetworkManager.cs
    Code (CSharp):
    1.   private void OnApplicationFocus(bool focus)
    2.         {
    3.             if (focus)
    4.             {
    5.                 Debug.Log("focus true");
    6.                 if (Server != null)
    7.                 {
    8.                     Action_InitAsServer();
    9.                     Debug.Log("Init As Server");
    10.                 }
    11.  
    12.                 if (Client != null)
    13.                 {
    14.                     Action_InitAsClient();
    15.                     Debug.Log("Init As Client");
    16.                 }
    17.             }
    18.         }
     
    Last edited: Dec 5, 2022
  24. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    737
    Thanks for your suggestion, we've modified it a bit and it works on Quest 2 and it's now working fine in either pause/resume/locked.

    Code (CSharp):
    1.  
    2.         private void OnApplicationFocus(bool focus)
    3.         {
    4.             if (focus)
    5.             {
    6.                 if (Server != null)
    7.                 {
    8.                     Action_Close();
    9.                     Invoke(nameof(Action_InitAsServer), 1f);
    10.                 }
    11.                 else if (Client != null)
    12.                 {
    13.                     Action_Close();
    14.                     Invoke(nameof(Action_InitAsClient), 1f);
    15.                 }
    16.                 else if (DataStream != null)
    17.                 {
    18.                     Action_Close();
    19.                     Invoke(nameof(Action_InitDataStream), 1f);
    20.                 }
    21.             }
    22.         }
    If someone wants to keep the connection continuously even the app is running in background, I suggested the most related topic of "Activity vs Service".
    https://forum.unity.com/threads/application-runinbackground-is-not-working-on-android.117723/
     
    Last edited: Dec 5, 2022
  25. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    737
    [V3.331] updates related to FMNetwork re-connection issue on mobile device

    Thanks for your contribution again.
    We've tested on Quest 2 with simple modification mentioned in my last post, hopefully this can solve your problem too.
     
  26. dawnr23

    dawnr23

    Joined:
    Dec 5, 2016
    Posts:
    41

    Thank you for responding quickly.

    However, even with that update, it still doesn't work on Quest2.

    After minimizing the app, press 'resume' to reconnect. This part seems well reflected.

    upload_2022-12-6_14-56-34.png
    However, if you click the red button in the attached image once to lock screen and then click again to release the lock screen, the app will not attempt to reconnect.


    Could you please check again?
     
  27. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    737
    We also tested the Quest 2 power button for jumping out/in many times without issue yesterday, with the latest Oculus SDK & Unity 2022.1.22f1, between Quest 2 and Unity Editor(Mac). The behaviour seems fine, the network component(client) will be deleted and re-initialised once Quest 2 resumed.

    You may try similar setup, by downloading our github template, and update it to the latest SDK & Unity version.
    https://github.com/frozenmistadventure/fmetp_tutorial_questvr_stream

    Edit 2022/12/07: We've also updated the github template to the latest Oculus SDK, and Unity 2022.1.22f1
     
    Last edited: Dec 6, 2022
  28. dawnr23

    dawnr23

    Joined:
    Dec 5, 2016
    Posts:
    41
    I figured out the cause.

    When using Unity version 2019.4.34f1 on Oculus Quest2
    OnApplicationPause is called.

    Existing Android apps have OnApplicationFocus called.

    Step 1 PowerButton Touch (lock screen and disable apps)
    => OnApplicationPause(true)

    Step 2 PowerButton ReTouch (Unlock and Activate App)
    => OnApplicationPause(false)

    Therefore, depending on the Unity version, we request exception handling again with OnApplicationPause value. I'd appreciate it if you could pass along the code or update the assets!
     
  29. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    737
    Do you mean only "OnApplicationPause" triggered on Quest 2 with Unity 2019, and "OnApplicationFocus" wasn't triggered at all?

    The latest v3.331 should included my verified solution with "OnApplicationFocus".
    Please give it a try? If necessary, we may include "OnApplicationPause" if it helps, but probably after your testing first.
     
  30. moliminous

    moliminous

    Joined:
    Oct 16, 2015
    Posts:
    70
    I'm still not seeing it anywhere. the rar file is what I'm using. testserver_v2.2.3 is its name. neither var exist? is there something else I'm supposed to be using?
     
  31. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    737
    Please make sure that you have the latest FMETP STREAM V3.331, the zip file should be the same name. In case that you have trouble in updating package via package manager, you could also reach us directly via email: thelghome@gmail.com
    Screenshot 2022-12-07 at 1.01.25 AM.png

    In case that you still can't find this line in your html, you can actually add a line "ConnectAsClient();" before the "ResetButtons()".
    Screenshot 2022-12-07 at 1.04.52 AM.png
     
  32. moliminous

    moliminous

    Joined:
    Oct 16, 2015
    Posts:
    70
    Turns out it was a unity issue. It was not pulling the latest version and said it was up to date. Had to delete it in app data and re-download it. So if anyone else is having that issue that would be the fix.
     
    thelghome likes this.
  33. youtosan0

    youtosan0

    Joined:
    Nov 28, 2022
    Posts:
    12
    Just a quick question, what should i do to display the things i get from game view decoder if there is no testingprop under the gameviewdecoder
     
  34. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    737
    Please set the preview type to either Raw Image or Mesh, and you will see the input field for preview.
    Screenshot 2022-12-08 at 7.09.56 PM.png
     
  35. youtosan0

    youtosan0

    Joined:
    Nov 28, 2022
    Posts:
    12
    Is there anyway i can use to make two clients for example, two android phones exchange their data to each other. Like phone A stream to B and B stream to A?
     
  36. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    737
    Our default main cam stream demo scene “Demo_FMNetworkStreamingMainCam” includes this example use case. Streaming video/mic network two devices.
     
  37. dawnr23

    dawnr23

    Joined:
    Dec 5, 2016
    Posts:
    41
    1. As you said, OnApplicationFocus is not called when pressing the 'Power Button' of Quest2 in Unity 2019.4.34f1. Only OnApplicationPause is called.

    2. In the same environment, OnApplicationFocus occurs only when the app is minimized.

    3. Other Android devices OnApplicationFocus is called just fine.
     
  38. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    737
    Thanks for clarify the situations on Quest 2, we will try to release another update soon by this weekend.
     
  39. youtosan0

    youtosan0

    Joined:
    Nov 28, 2022
    Posts:
    12
    Question is how can I made that within the same applications
     
  40. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    737
    It's very straight forward..
    The setup is the same on both server and client, the only different is network type.

    Device 1:
    FMNetworkManager(Server)
    GameViewEncoder -> byte[] -> FMNetworkManager(SendToOthers) -> byte[] -> GameViewDecoder

    Device 2:
    FMNetworkManager(Client)
    GameViewEncoder -> byte[] -> FMNetworkManager(SendToOthers) -> byte[] -> GameViewDecoder

    For step by step tutorial:
     
  41. dawnr23

    dawnr23

    Joined:
    Dec 5, 2016
    Posts:
    41
    Hello,

    Is there a way to remotely run a specific App on Android via FMNetwork (UDP)?
     
  42. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    737
    In theory, you can send a string command and trigger an Android App link to open another app.
     
  43. jongnam

    jongnam

    Joined:
    Nov 12, 2012
    Posts:
    2
    hi. Can I create multiple clients? I want to connect to multiple servers in the same scene.
    For example, creating several FMNetworkManager components and trying to connect to them with different server IPs. However, it seems that only one instance of FMNetworkManager can be created.

    Also, is there a way to check the client's IP in the server's receive event handler? There are several clients connecting to the server, and I want to know who is sending them. Like Action_ProcessStringData(string _string, string _ip).
     
  44. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    737
    You may create multiple FMNetworkManager with different Server ports and Client ports, as it requires two unique ports for the whole networking communication.

    If you have multiple FMNetworkManager, you may need to assign the Send() function with selected manager, as there is only one global instance.

    You can get IPs in this way:
    FMNetworkManager.instance.Server.ConnectedClients.IP

    The event handler for client IP is only invoked when they have new connection/disconnection.
    The data receiver event doesn't include IP info, you might need to add your own metadata when you send the string.
     
  45. dawnr23

    dawnr23

    Joined:
    Dec 5, 2016
    Posts:
    41
    hello,

    [Device (Platform)]
    - Server is using Galaxy Tab S8 (Android). It is 1 device.
    - Client is using VR Oculus Quest2 (Android). 20 devices.

    [Scene structure]
    - Server is managing FMnetworkManager.cs in only one scene.
    - The Client has FMnetworkManager.cs in Scene A and Scene B, respectively, and the scene is loaded asynchronously and moved once every 5 minutes. Naturally, when moving, the server and client reconnect.

    In the meantime, if you move about 10 times on average, the connection between the server and the client is maintained, but there are times when the String communication (Server Send) stops working.


    Any solution?
    Or is there a reason you can guess?

    The version of the asset is the latest version, and it is not that the string Send does not work immediately after an error occurs, but the string Send continues and there comes a point when it suddenly does not work. (Average 5-10 Scene moves)


    [(Server) When I take a debug, I get this error.]
    InvalidOperationException: Collection was modified; enumeration operation may not execute.
    at System.ThrowHelper.ThrowInvalidOperationException (System.ExceptionResource resource) [0x0000b] in <44afb4564e9347cf99a1865351ea8f4a>:0
    at System.Collections.Generic.List`1+Enumerator[T].MoveNextRare () [0x00013] in <44afb4564e9347cf99a1865351ea8f4a>:0
    at System.Collections.Generic.List`1+Enumerator[T].MoveNext () [0x0004a] in <44afb4564e9347cf99a1865351ea8f4a>:0
    at FMETP.FMNetworkManager.UpdateDebugText (System.Boolean onNetworkManagerDisabled) [0x001c7] in <cfe54ec18ead474498a8efa916f845e0>:0
    at FMETP.FMNetworkManager.Update () [0x00265] in <cfe54ec18ead474498a8efa916f845e0>:0
    (Filename: <44afb4564e9347cf99a1865351ea8f4a> Line: 0)
    upload_2022-12-20_13-45-16.png
     

    Attached Files:

    Last edited: Dec 20, 2022
  46. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    737
    Seems likely the bug was related to the debug info, you may please disable the "Debug Status" for comparison.
    And please let us know if the debug status mode is related on your testing.

    Meanwhile, we are investigating the related errors now and hopefully a quick fix will be ready soon.
    Edited: We've released a quick fix, please check it out.
     
    Last edited: Dec 20, 2022
  47. eagleeyez

    eagleeyez

    Joined:
    Mar 21, 2013
    Posts:
    406
    Is a demo for my use case in this package? I want to stream the camera of a Samsung S8 to my desktop pc unity project.
     
  48. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    737
    Yes, webcam stream is included in our demo scene.
     
  49. dawnr23

    dawnr23

    Joined:
    Dec 5, 2016
    Posts:
    41
    What can I add to enable this option?
    - added reliable send option(experiment)

    Also, what does this option mean?

    We implemented UDP communication through the asset, but we are desperate for this option because we confirmed that communication does not occur once or twice in 100 String Send attempts.
     
  50. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    737
    This is one of extra options, besides "SendToAll/Server/Others".
    Instead, you can choose "SendToAllReliable()", "SendToServerReliable()" and "SendToOthersReliable()". Screenshot 2022-12-21 at 8.13.11 PM.png

    This option contains a sequence id for confirmation between Server and Clients via UDP.
    We suggested that you should only use it for important data, because if you have many packet loss in your network, the queued for re-sent will be accumulated heavily.