Search Unity

[Release] FM ExhibitionToolPack # In-Game Audio+Video Streaming (UDP/TCP/WebSocket) Cross Platform!

Discussion in 'Assets and Asset Store' started by thelghome, Apr 30, 2019.

  1. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    49
    Main001-100.jpg

    battery drain? overheat? live/game view streaming? gigapixel panorama? custom tcp/udp network action solution?

    Webpage | Asset Store
    FM Exhibition Tool Pack
    provides you a simple solution for popular troubles in public events or exhibitions. All source codes are optimised for exhibition purpose, and used in many exhibition over past few years.

    Some new features are in Beta version. If you found any bug, please report to us via email and we would like to provide support.
    Support::thelghome@gmail.com

    supported platform:
    iOS, Android, Mac, PC

    [v1.093]
    -Added Streaming Demo for FM Network UDP
    -Added Server-Client Networking structure for WebSocket Demo

    [New!] FM Network UDP
    - Simple Server-Client Networking Solution with UDP
    - Added Game View Streaming Demo

    [New!] WebSocket Demo (cross platform)
    -added WebGL support, tested WebGL(mobile)
    -tested on native build with iOS/Android/Mac/PC/Oculus Go


    In-Game Audio+Video Streaming Demo
    - Capture & Streaming In-Game Audio and Video
    - Tested: Streaming Game View between Mac/PC/iOS/Android/Oculus Go

    Network Action (Beta)
    - Custom RPC networking solution with TCP and UDP
    - Sync / Remote trigger for multiple clients
    - Simple Example Scene provided

    Battery Saving Mode
    provide a sleep mode for your app, when there is no one playing. it's extremely useful with AR app when there is no power supply in your exhibition gallery.

    Live Streaming
    Simple local streaming solution, used for live streaming game view, live webcam, live reflection map for AR

    Panorama 360 Viewer
    Gyro Demo & Editor Mode Simulation

    Gigapixel Example
    Custom Shader for high resolution panorama
    Mobile Devices: 4 x 4 x 4096 (W) x 2 x 4 x 4096 (H)
    on 8k supported platforms: 4 x 4 x 8192 (W) x 2 x 4 x 8192 (H)

    *All Source Code in C#
    *All Core parts are used in many exhibitions in past few years
    *Will Keep adding new features
    Tested devices in our public projects:
    iPhone 5
    iPad Air 2
    iPad pro(2017)
    iPad pro(2018)
    iPhone 6s
    iPhone XS
    Macbook(2017)
    Macbook Pro(2018)
    Windows 10
    Oculus Go
    Samsung S9+

    Main003-100.jpg Main002-100.jpg Main004-100.jpg Main005-100.jpg
     
    Last edited: Sep 2, 2019
  2. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    49
    We just submitted an update to Asset Store and the new feature will be available in v1.02 very soon.
    [NEW!] Network Action (Beta)
    - Custom RPC networking solution with TCP and UDP
    - Sync / Remote trigger for multiple clients
    - Simple Example Scene provided

    Artboard 6-100.jpg
     
  3. hrjunker

    hrjunker

    Joined:
    Nov 15, 2011
    Posts:
    24
    Hi, do you have any experience streaming from a mobile VR headset. We are looking for a solution that can do this without compromising the framerate on the headset, currently using a Oculus Quest.
     
  4. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    49
    Thanks for reaching us!
    I assumed that you want to mirror the headset view to other screens(pc/tablet). We mainly do testing between iOS/Android/Mac/Win10, and even iphone5 can stream the webcam/game view without problem.

    for the performance, it depends on few things:
    1. hardware side: cpu / network card / wifi router
    2. image quality: resolution / compression(file size)

    We tried GearVR with Samsung S9+ and it works fine. Unfortunately, we don't have Oculus Quest in our lab now, but we will order it soon.
    For your concern, we will do some test with Oculus Go in coming few days.
     
  5. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    49
    We did a game view streaming on Oculus GO with resolution of 1024x576(streaming fps: 20).
    It costs 5ms in headset performance. I believe Oculus Quest should have a better processor than Oculus Go, thus, the performance will be better too.

    The headset view is wirelessly streaming to iphone XS / Macbook Pro almost in real time without noticeable delay.
    the latency with my plugin is much much lower than "Oculus Go Wired mirroring 2~4 second delay" said on their official website.
    https://developer.oculus.com/blog/oculus-go-wired-mirroring-how-to/

    I will update a GameViewStreaming demo very soon, which are tested with Oculus Go.
     
  6. Tibor0991

    Tibor0991

    Joined:
    Dec 11, 2016
    Posts:
    18
    I have a similar question regarding VR streaming: in my case, I want to stream the content of a desktop Unity app to an instance of another Unity app running on a mobile VR headset (Go or Quest); will your plugin satisfy this use case? More specifically, does your plugin provide a way to render the stream on an arbitrary surface?
     
  7. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    49
    We just submitted version1.03 update to Asset Store, hopefully it will be available soon.
    -Added Game View Streaming, Tested VR Headset: mirror Oculus Go Game View to Macbook Pro & iPhone XS
    -Optimised Connection(Server)
    -Available to Send Raw Byte[] in Network Action(custom RPC networking solution)

    Any questions from you guys could help us understood your needs! We would like to test and take action immediately if possible.
     
  8. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    49
    It should work. for your use case, I need to test the performance on Oculus Go in office next week.
    what is your target streaming resolution & streaming fps, which will satisfy you?

    PS: Once your server & clients are on Mac/PC/iOS/Android, they should work. but the image quality and performance rely on your hardware. Usually, decoding streaming on clients side demands more resources.
     
  9. Tibor0991

    Tibor0991

    Joined:
    Dec 11, 2016
    Posts:
    18
    I am looking to stream content on an Oculus Go/Quest at 640x720, 60 fps at least; I have written a contrived setup that allows me to stream still frames over the net and I get around 20-ish frames per second, but it wastes a lot of bandwith.
     
  10. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    49
    I personally don't believe that Oculus Go can achieve 60fps game view streaming, as it's a very cheap VR headset. Maybe Quest can do it, but I didn't own one at the moment.

    Would you mind to share your current bandwidth/data size for each frame?
     
  11. agokoglu

    agokoglu

    Joined:
    Sep 23, 2014
    Posts:
    3
    Hi, I wonder if the network solution can also be used for AR applications. Is the RTMP protocol supported during Live Stream? For example, can I use a mobile broadcast to transfer to Wowza or Dacast servers? Or what else do I have to do for this? I would be very happy if you could help me with this subject. Will you also prepare a video sample for the new network feature?
     
  12. Tibor0991

    Tibor0991

    Joined:
    Dec 11, 2016
    Posts:
    18
    I own them both, and as far as I can say there's no huge improvement, it's also true that I have to test my system on the Quest yet.


    Each frame is 160000 bytes long tops, packed as JPEGs with a 80% compression rate at 640x720, theoretically speaking a second should be around 9 MB (60 frames per sec) of data and a full 30 minutes session should be around 15 GB; of course this is a ridiculous amount of data sent that should (could?) be compressed further with a proper codec. The biggest bottleneck so far has been the Texture2D.Apply() function, which is obviously needed to switch from CPU space to GPU space.
     
  13. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    49
    we have two separated demos for "live webcam/game view streaming" and "custom rpc networking(experiment feature)".
    However, those features are aiming for local network during our development.
    Technically speaking, they are all developed based on UDP+TCP. As all our codes are writing in C# and can be modified freely, they might be a good starting point for your subject.
    We would like to do some research about RTMP protocol and It's kind of interesting topic.

    PS: please feel free to discuss and hopefully we both can learn something new.
     
  14. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    49
    I agreed that it needs a proper codec for compression. In general, encoding and decoding heavily will affect the performance, which it's another challenge for all-in-one headsets.

    I suggested that you should target 30fps for oculus quest, with balanced image quality.
    In order to maximise your device's hardware limit, your streaming fps should be same as your game fps. It means, your streaming fps should not be higher than your game fps.

    In your case, I can imagine your biggest bottleneck should be VR headset, as you can still upgrade your computer with insane cpu for "Texture2D.Apply()" if budget allows.
     
  15. Tibor0991

    Tibor0991

    Joined:
    Dec 11, 2016
    Posts:
    18
    (I've bought your plugin in the meanwhile, if it doesn' fit this project I have still another use planned.)

    Yes, my biggest bottleneck is the VR headset, because Texture2D.Apply() takes too much time to update the screen; it should be possible to stream a hi-def video though, since the guys from Cubicle Ninjas (Guided Meditation VR) are able to play a 360 video in 5k/Strips format in Unity without any stuttering on Oculus Go (okay, the source is different since they stream from persistentDataPath, but still the texture has to be updated fast enough somehow).
    I will take a look at your code in the meanwhile, maybe there's something I'm missing.
    Thank you for your replies!
     
  16. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    49
    loading video from persistentDataPath is totally another story, as it doesn't need to render image and transfer any data via wifi in real time.

    share a little bit of my experience:
    I had a project using AVPro Video with Oculus Go for normal 5k video playback in 60~72fps in last year. if you do not need to render texture in real time wirelessly, this might be your option.

    I am just curious: Does Oculus Quest have sleep mode after 5mins? I gave up Oculus Go as its annoying sleep mode.

    Recently, there is a project for an immersive 360 cylindrical projection system, which requires a video player solution to sync 6~11 computers. I was using UNet for networking solution but it will be depreciated soon. Thus, I started creating a custom RPC networking method to upgrade the player. I am not sure if someone will be interested in a multi-video sync player solution demo?
     
  17. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    49
    and please be reminded that the latest version 1.03 is just submitted in few hours ago, it will take few days to get approval on Asset Store. In case you are urgent, you could send me an email with invoice number.
     
  18. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    49
    [Good News!]
    As requested by some customers, we planned to add Audio Streaming demo in future updates, which allows you to capture in-Game Audio, and stream it between any platforms (Mac/PC/iOS/Android).

    PS: now we are in debugging stage & trying to optimise it, hopefully it can be included in coming updates asap.
     
  19. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    49
    [v1.04] updated:
    basic audio streaming demo is available on Asset Store

    [v1.05] submitted & will be available soon:
    optimised audio streaming demo for different Sample Rate & Channels Count between server & clients
     
  20. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    49
    [v1.061] submitted
    -optimise for Windows 7
    -optimise for old cpu, which are less than 4 threads
    -testing async solution on devices, in order to reduce thread usage
    -streaming experiment on VR headsets
     
    Last edited: Jul 23, 2019
  21. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    49
    [NEW: work in progress]
    Web Streaming demos
    -support: Web Browser, Mobile browser, Html, WebGL

    current status:
    tested on devices & doing documentation & organising files.
    we will release in coming updates when ready.
     
    Last edited: Jul 28, 2019
  22. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    49
    [v1.07] New feature: Streaming Game Camera to web browser
    Supported: iOS/Android/Mac/PC/Oculus Go
     
    Last edited: Jul 29, 2019
  23. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    49
    [v1.08]
    [New!] Native Build + WebGL support for WebSocket Streaming Demo (cross platform)
     
    Last edited: Aug 19, 2019
  24. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    49
    [v1.09] available for download
    [New!] FM Network UDP
    Instead of TCP solution, we released a UDP Server-Client Networking solution, which will be included in our full pack.
    FMNetwork1-100.jpg
     
    Last edited: Aug 27, 2019
  25. muellg

    muellg

    Joined:
    Jul 23, 2019
    Posts:
    1
    Hi! I'm trying to create an application where two devices can see each other's webcams... similar to a chat application. I've been trying to combine the Demo_LiveStreamingClient and Demo_LiveStreamingServer scenes from the kit, but I haven't had much luck with it. How would I go about making this happen?
     
  26. tomas-vymazal-upp

    tomas-vymazal-upp

    Joined:
    Aug 14, 2019
    Posts:
    7
    Hi,
    I am considering to buy this plugin for my work project. As I have seen the video WebSocket Streaming demo, it seems that is is suitable for our use, but I want to ask directly:
    I want to use it to stream the image/video taken from the camera in scene running in editor (in play mode) to the iOS device. I already have the code which does this (without any plugin), but by using the raw textures only with simple compression and it is not usable with higher resolutions due to the high latency caused by transfering large images. So my question is: can I stream the camera viewport from scene running in desktop Unity (Mac/PC) to the iOS device, in realtime, with this plugin? Is the compression good enough even for FullHD (or possible higher) resolutions?

    Thanks.
     
  27. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    49
    This feature is in our latest beta version(v1.091beta). Instead of old TCP solution, we added working demo with FM Network UDP, which allows you to stream between server and client at the same time.
    please send us an email with your invoice number, we could send you the beta version if you are urgent.
    In current version, you probably need to modify the codes a bit, as the demo combines image processing and streaming.
     
    Last edited: Aug 23, 2019
  28. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    49
    I just did a quick testing between iPad Pro(Web Browser) and Macbook Pro with the demo dancing boxes. It's capturing the game view in realtime for live streaming.
    1080HD: still can achieve realtime streaming, but streaming FPS is not as high as 720P.
    720P: smooth during my testing.

    In theory, my plugin won't create huge latency as it's not buffering all frames. the streaming fps might be affected in higher resolution.
    however, you can always adjust the image quality for streaming data size.
     
    Last edited: Aug 23, 2019
  29. tomas-vymazal-upp

    tomas-vymazal-upp

    Joined:
    Aug 14, 2019
    Posts:
    7
    Hi! This is awesome and thanks for quick response!
    Another question: is it possible to stream it (to the iOS) not to the browser but to another Unity app? So there is Unity on both sides (desktop PC/Mac and iOS)?
     
  30. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    49
    Yes, it works for iOS build too!
     
    tomas-vymazal-upp likes this.
  31. tomas-vymazal-upp

    tomas-vymazal-upp

    Joined:
    Aug 14, 2019
    Posts:
    7
    Hi, what video codec are you using for the video streaming? So for example if I would stream my desktop app unity camera content to the iOS device with unity app running, with which video codec it would be encoded? Or is it encoded as video at all? Or is it using only the one image compression at a time? Thanks!
     
  32. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    49
    Technically speaking, the streaming method is sending image sequences. It's the most common solution for low-end/high-end devices, and the best and cleanest way to support almost all platforms, without importing tones of plugins.
     
  33. tomas-vymazal-upp

    tomas-vymazal-upp

    Joined:
    Aug 14, 2019
    Posts:
    7
    But is it actually diffing the images in the sequences and sending only the difference between the two consecutive images?
    Thanks.
     
  34. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    49
    In current version, we just stream the full image data. We focus on the performance in general devices on local network, as our first priority.
    PS: We are still doing research on reducing data size. The encoding method you mentioned is one of solutions, but the diffing process is a heavy task in general. It is still not an ideal way for general devices, and it will cause heavy drop-frames on low-end devices.
     
  35. tomas-vymazal-upp

    tomas-vymazal-upp

    Joined:
    Aug 14, 2019
    Posts:
    7
    I believe for example Unreal 4 does actually use "video streaming" (in terms of diffing the consecutive images) with their pixel streaming technology:
    "Pixel Streaming Plugin - This Plugin runs inside the Unreal Engine. It encodes the final results of every rendered frame using H.264 video compression, packs those video frames along with the game audio into a media stream, and sends that stream to the WebRTC Proxy Server."

    I thought that you are using something similar and that this is the way how you are able to stream even Full HD stream. Because according to my tests in my prototype project (without your plugin) i was not able to stream bigger than 512x512 images without significant slowdown (while using generic non image based compression).

    So are you just actually using some type of image based compression (PNG/JPEG)? I have not yet bought your plugin but I am considering buying it, but I need some technical details :)
     
  36. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    49
    Thanks for your interest, we simply use jpeg compression.
    There is no big technical secret in our plugin, but just the effort of testings, and trying to make it stable and smooth. As we mentioned, all core scripts are written in C#, based on all possible solutions within Unity.

    However, could you specify the challenge in your current project? are you suffering in streaming data size, or just bottleneck of hardware compression?
     
  37. tomas-vymazal-upp

    tomas-vymazal-upp

    Joined:
    Aug 14, 2019
    Posts:
    7
    The problem that I am trying to solve is that of capturing the scene on the desktop (with Unity camera) and transferring the image to the iOS unity app.
    This works already, but only for small resolutions. I am just using Texture2D which is than serialized and compressed (only using C#'s deflate) and decompressed on the iOS device.
    For the communication, I am not using sockets, but Unity's messaging (for example ARKit sample is built on this). This maybe in fact another bottleneck as it may have additional overhead in comparison with normal socket communication.

    The problem that I have with my solution is that transferring any bigger resolution than 512x512 over the local network introduces a lot of lag manifesting as low framerate of image updates on the iOS device. I believe that your plugin and JPEG compression over the raw network socket IO may mitigate this problem and allow me to transfer at least FullHD resolution with reasonable framerate (at least 30fps). Am I right?

    There will also be input sent back from the iOS device to the desktop, possibly moving with the camera, so I hope the percieved latency loop from the control on the iOS device up until updated image on the iOS device won't be that high.

    Thanks!
     
  38. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    49
    I think 720P is realistic but 1080P(30fps) on iOS will be overhead. Unity's messaging is slow for sure, in comparison to any raw socket.

    If your project only needs local streaming solution, we have demo with "FM Network UDP" in our beta version, doing exact things you are asking. It's much faster than WebSocket demo, which is for online web streaming instead local streaming.
     
  39. tomas-vymazal-upp

    tomas-vymazal-upp

    Joined:
    Aug 14, 2019
    Posts:
    7
    Awesome! But I guess UDP streaming only improves latency, not anything related to "bandwidth", so the 1080p option is still out of reach. Am I right? But even if this option improves latency in 720p, it would still be great. We will definitely buy the plugin then, we just need to setup few things first.
     
  40. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    49
    If you were sending Unity Messages based on string data, the bandwidth is very huge.
    Our UDP or TCP solutions will display the image immediately once received data. I think the latency you mentioned, is related to the speed of data processing. The data could be accumulated very quick if you didn't handle them correctly.

    PS: Please don’t be hesitated for requesting us the latest beta via email if you purchased.
     
    tomas-vymazal-upp likes this.
  41. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    49
    [v1.093] Available on Asset Store now!
    -Added Streaming Demo for FM Network UDP
    -Improved and Simplified Demos

    [Lite Edition] FM Network UDP is also available on Asset Store, for someone who just need UDP solution without Live Streaming Demos. You can upgrade to FM Exhibition Tool Pack if needed.
     
    Last edited: Sep 2, 2019
  42. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    49
    [v1.094] Available on Asset Store now.
    -Optimised Playback Quality of Audio Streaming
    -Simplified Demo Scenes
     
  43. Dawar

    Dawar

    Joined:
    Dec 7, 2014
    Posts:
    122
    can i connect this with andriod tv and change the volume and other othings with this ?
     
  44. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    49
    Could you specify your case?
    Do you mean deploy an app on an Android TV device and control your app remotely?
     
  45. Dawar

    Dawar

    Joined:
    Dec 7, 2014
    Posts:
    122
    thanks for your reply .
    i want to make tv wifi remote means control channel volume and other things through mobile wifi. is this possible with this.
     
  46. thelghome

    thelghome

    Joined:
    Jul 23, 2016
    Posts:
    49
    I think that you need some Native SDK, which allows you to pair and get access of Android TV directly via wifi.

    The networking solution in our plugin is for communication between two apps. Theoretically you can have some solution mentioned below.
    # found native sdk for devices control, install the app on android tv and run in background
    # use FM Network UDP for communication between apps
     
    Dawar likes this.
  47. Dawar

    Dawar

    Joined:
    Dec 7, 2014
    Posts:
    122
    thats great thank you so much