Search Unity

Unity Render Streaming Introduction & FAQ

Discussion in 'Unity Render Streaming' started by kazuki_unity729, Sep 10, 2019.

  1. kazuki_unity729

    kazuki_unity729

    Unity Technologies

    Joined:
    Aug 2, 2018
    Posts:
    11
    Hi everyone,

    Welcome to the Unity Render Streaming discussion forum.

    What is Unity Render Streaming?

    This is a solution that provides Unity's high definition rendering abilities via a browser. It's designed to meet the needs of tasks like viewing car configurators or architectural models on mobile devices.
    This solution's streaming technology takes advantage of WebRTC, and through customization, developers can create their own unique solutions.



    Github repository is here.
    These links should help you to understand this software.
    Also, the WebRTC package is developed in other Github repository which is under the Apache 2.0 license. I'd be grateful if you contribute to the open-source project.

    We encourage all interested users to download the Unity Render Streaming and try it out for themselves. We hope that this package will inspire users to create their own new solutions.
     
    Last edited: Sep 13, 2019
    Mauri, biemann, jdcampbe and 3 others like this.
  2. Cascho01

    Cascho01

    Joined:
    Mar 19, 2010
    Posts:
    1,110
    This means there´s a server PC (with a NVidiaCard) running an Executable as a server that streams the video and reacts to the input coming from the mobile devices, right?
    Is this comparable to Stadia or other Streaming-Services?
     
  3. Sam77777

    Sam77777

    Joined:
    Jun 15, 2019
    Posts:
    5
    Is possible to stream on browser of the VR device and get input from HMD?
     
    Last edited: Sep 18, 2019
    ina and markjanzen88 like this.
  4. NeoSerge

    NeoSerge

    Joined:
    Aug 19, 2014
    Posts:
    1
    [WebRTC] The version of the hardware codec driver does not support API
    UnityEngine.Debug:LogError(Object)
    Unity.WebRTC.WebRTC:Initialize() (at Packages/com.unity.webrtc/Runtime/Srcipts/WebRTC.cs:236)
    Unity.RenderStreaming.RenderStreaming:Awake() (at Assets/Scripts/RenderStreaming.cs:59)

    How to solve this?
     
  5. kazuki_unity729

    kazuki_unity729

    Unity Technologies

    Joined:
    Aug 2, 2018
    Posts:
    11
    This error caused by the NVIDIA graphics driver version.
    See `Dependencies ` section on this page.
    https://developer.nvidia.com/nvidia-video-codec-sdk
     
    NeoSerge likes this.
  6. kazuki_unity729

    kazuki_unity729

    Unity Technologies

    Joined:
    Aug 2, 2018
    Posts:
    11
    > This means there´s a server PC (with a NVidiaCard) running an Executable as a server that streams the video and reacts to the input coming from the mobile devices, right?

    Yes.

    > Is this comparable to Stadia or other Streaming-Services?

    I am not sure about the technology stack of these products.
     
  7. kazuki_unity729

    kazuki_unity729

    Unity Technologies

    Joined:
    Aug 2, 2018
    Posts:
    11
    I am not sure the webrtc technology works on VR devices.
     
  8. Cascho01

    Cascho01

    Joined:
    Mar 19, 2010
    Posts:
    1,110
  9. jk314

    jk314

    Joined:
    Jun 23, 2011
    Posts:
    48
    Do touches and clicks on gameobject work or do you have to set up separate UI because it is only a render and not the actual scene? Thanks.
     
  10. shubhamshetkar25

    shubhamshetkar25

    Joined:
    Feb 4, 2017
    Posts:
    17
    where's the demo, I can play in browser?
     
  11. sindharta_at_unity

    sindharta_at_unity

    Unity Technologies

    Joined:
    Jul 4, 2019
    Posts:
    12
  12. milanwijnmaalen

    milanwijnmaalen

    Joined:
    Dec 14, 2016
    Posts:
    1
    WebRTC support for unity is very great news!

    The tutorial says you have to import com.unity.webrtc.tgz, however this file is not included in the 1.1.1 github release?
    https://github.com/Unity-Technologies/UnityRenderStreaming/releases

    When importing com.unity.renderstreaming-1.1.1-preview.tgz I get the error that com.unity.webrtc@1.0.1-preview cannot be found.

    Where can I find this file?
    Thanks!
     
  13. biemann

    biemann

    Joined:
    Jun 28, 2019
    Posts:
    3
    milanwijnmaalen likes this.
  14. shubhamshetkar25

    shubhamshetkar25

    Joined:
    Feb 4, 2017
    Posts:
    17
    Can we create build with this and upload on any webhost and stream on android devices? (like stadia)
    (suppose I'm building unity APK, can I host my very high quality games on webhost, call them through app and play(stream), so users can play high quality collection of my games just with one app, or on my website in browser)
     
  15. kazuki_unity729

    kazuki_unity729

    Unity Technologies

    Joined:
    Aug 2, 2018
    Posts:
    11
    here.
    https://github.com/Unity-Technologies/com.unity.webrtc/releases/tag/1.0.1-preview
    I am sorry for your confusion, this package should be registered in the PackageManager but it works in progress.

    Thank you @biemann,
    This video is the best helpful tutorial ever!
    It's so amazing!
    https://www.gamefromscratch.com/post/2019/09/18/Unity-Release-Render-Streaming-Over-WebRTC.aspx

    > I have also a question, is it (or will it be) possible to stream from an Android app to the browser?

    Currently, we have no plan to support android natively.
     
    Last edited: Sep 19, 2019
    biemann and milanwijnmaalen like this.
  16. Weidz_

    Weidz_

    Joined:
    Feb 15, 2018
    Posts:
    5
    Does it support multiple, different outputs ? Let say I want to map one camera to a stream and a second camera on another stream ?
     
  17. Blarp

    Blarp

    Joined:
    May 13, 2014
    Posts:
    193
    Looking to stream from an ios Arfoundation app to a desktop browser.

    Loaded up the sample app and looks like this is HDRP only? ALOT of people are going to want this to stream their AR sessions. Can we get a URP priority bump on this one? @kazuki_unity729

    For example, I could theoretically stream AR FPS matches from my users with this and output to twitch. Or a repo of my user's gameplay videos on my website.

    webrtc.png

     
    Last edited: Sep 19, 2019
    newguy123 likes this.
  18. kazuki_unity729

    kazuki_unity729

    Unity Technologies

    Joined:
    Aug 2, 2018
    Posts:
    11
    > Can we create build with this and upload on any webhost and stream on android devices?

    The current version doesn't have native Android support.

    > users can play high quality collection of my games just with one app, or on my website in browser

    Generally, streaming technologies have some latency which may be acceptable for some games but not for others.
    You should experiment directly to see whether it is suitable for your needs or not.
     
  19. sindharta_at_unity

    sindharta_at_unity

    Unity Technologies

    Joined:
    Jul 4, 2019
    Posts:
    12
    Not at the moment, but it's on our roadmap. The ETA hasn't been decided though.


    This is not only for HDRP. We can use any graphics pipeline that we want.
    Please note that the current version of RenderStreaming requires DirectX 11.
     
  20. jk314

    jk314

    Joined:
    Jun 23, 2011
    Posts:
    48
    I'm trying to use RemoteInput.RemoteMouse.position.ReadValue() to get the remote mouse position but I always get (0.0, 0.0)

    Is this not how I should get the remote mouse position? Thanks.
     
  21. sindharta_at_unity

    sindharta_at_unity

    Unity Technologies

    Joined:
    Jul 4, 2019
    Posts:
    12
    No. In our sample, the position received from the network is stored inside RemoteInput class, but it is not being sent to the input system. We only store the delta when queuing the mouse event.
    See
    RemoteInput::ProcessMouseMoveEvent()
    .
     
  22. jdcampbe

    jdcampbe

    Joined:
    Jun 10, 2015
    Posts:
    8
    I'm trying to test the scenario of having the server on a remote cloud computer with the application locally (eventually the application will also be moved to the cloud computer). Sometimes when I connect it works; other times it fails. It looks like the JSON packet received by unity is too long and gets cut off before it is fully received:

    This is the error I'm getting from the demo application.

    {"offers":[{"connectionId":"dfff74b9-3c94-4ac1-9082-a3fe8e793abe","sdp":"v=0\r\no=- 7171538904994622442 2 IN IP4 127.0.0.1\r\ns=-\r\nt=0 0\r\na=group:...Long string of connection information
    UnityEngine.Debug:LogError(Object)
    Unity.RenderStreaming.DownloadHandlerJson`1:ReceiveData(Byte[], Int32) (at Assets/Scripts/Signaling.cs:34)​

    subsequently that causes this error:

    ArgumentException: JSON parse error: Missing a closing quotation mark in string.
    Unity.RenderStreaming.DownloadHandlerJson`1[T].ReceiveData (System.Byte[] data, System.Int32 dataLength) (at Assets/Scripts/Signaling.cs:35)​

    any advice or ideas would be great! Thanks.
     
  23. jdcampbe

    jdcampbe

    Joined:
    Jun 10, 2015
    Posts:
    8
    it looks like this is already flagged on the git repo so I'll watch for a solution there.

    Git repo issue: https://github.com/Unity-Technologies/UnityRenderStreaming/issues/137
     
  24. jdcampbe

    jdcampbe

    Joined:
    Jun 10, 2015
    Posts:
    8
    Is the package employing anything specific to Windows 10? Or could it possibly work on some windows server versions?
     
  25. Aiursrage2k

    Aiursrage2k

    Joined:
    Nov 1, 2009
    Posts:
    4,807
    1) Are you guys planning to add mac ?
    2) Does this work outside of the local network? tried on my labtop and it didnt work (because the graphics card was too old), however on my windows server computer it worked. However when I tried to connect to it, it didnt work.
    video-player.js:155 Uncaught (in promise) DOMException: Failed to execute 'addIceCandidate' on 'RTCPeerConnection': Error processing ICE candidate
     
    Last edited: Sep 23, 2019
  26. Aiursrage2k

    Aiursrage2k

    Joined:
    Nov 1, 2009
    Posts:
    4,807
    I was able to work on my windows server, or at least work locally. You need to make sure you have a windows 64, with radeon card capable of running the codec (it has to be at least 1050 I think)

    I am not able to stream the video to my phone -- the server is in a different city (newyork to toronto, so its possible the latency is too long -- not really sure how the encoder/decoder works.

    I have an older labtop with a radeon 1050 I think so I can test that tomorrow.
    ##############
    As an aside
    I was messing around the JSMegplayer, sending the video to a server on AWS and forwarding that data to the clients (using node). The biggest problem with that is data cost, but piggybacking off the "hosts" network would certainly cut the cost
    https://github.com/phoboslab/jsmpeg
     
    Last edited: Sep 23, 2019
  27. jdcampbe

    jdcampbe

    Joined:
    Jun 10, 2015
    Posts:
    8
    Excellent. Thanks! I'll give it a try sometime this week. (you mean Nvidia GTX 1050, not radeon x1050, right? ;))

    I suspect that's due to the same issue that I had (https://github.com/Unity-Technologies/UnityRenderStreaming/issues/137)
    The offer is too long so it never completes the connection. If you look at the debug log on the application it will show an overflow error (if it's the same) or a missing quotation mark error.

    It sounds like we're working on similar problems on opposite sides of the country. Are you using Amazon's EC2s for your remote computers and running application/server on the same computer?
     
  28. sindharta_at_unity

    sindharta_at_unity

    Unity Technologies

    Joined:
    Jul 4, 2019
    Posts:
    12
    We haven't tried using other versions of Windows, but I believe it should work as long as it has a graphics card that supports NVidia's NVCodec.

    Mac doesn't support NVidia hardware (as far as we know anyway), so we don't have mac support on our roadmap.

    Yes, it should work provided that your server can be accessed from outside, although we do have the json overflow issue as mentioned by @jdcampbe
     
    jdcampbe likes this.
  29. Aiursrage2k

    Aiursrage2k

    Joined:
    Nov 1, 2009
    Posts:
    4,807
    I thought you were going to add other encoders/decoders. I think its important to hit as many computers as possible.

    Okay I tried my other labtop and it works, I am able to stream from computer. But when I try to stream to my iphone it didnt work
     
    Last edited: Sep 24, 2019
  30. sindharta_at_unity

    sindharta_at_unity

    Unity Technologies

    Joined:
    Jul 4, 2019
    Posts:
    12
    Did you create your own SSL certificate and connect using https ?
    This is required to connect using iPhone.
     
  31. sindharta_at_unity

    sindharta_at_unity

    Unity Technologies

    Joined:
    Jul 4, 2019
    Posts:
    12
    To clarify, I was referring to our current roadmap.
    We are indeed planning to support other GPUs in the future, but this is still at a very early stage in our discussions.

    Thank you for your feedback. We welcome any feedback/contributions to drive these discussions.
     
    Last edited: Sep 24, 2019
  32. Aiursrage2k

    Aiursrage2k

    Joined:
    Nov 1, 2009
    Posts:
    4,807
    Okay when I try to connect to my computer from a work computer it doenst work (I assume I have to open port 80 in order to do that).

    Is it possible to use https without requiring a domain name? If we want to have many different clients (IE every user that acts as a host)
     
  33. sindharta_at_unity

    sindharta_at_unity

    Unity Technologies

    Joined:
    Jul 4, 2019
    Posts:
    12
    How far is the distance between your computers ? It might be caused by the json overflow issue. Perhaps you can confirm it by looking at the debug log ?

    For testing locally, yes, it is possible to create your own certificate and use https without a public IP or a domain name.
    We have a tutorial about it on Github
     
    Last edited: Sep 25, 2019
    yosun likes this.
  34. Aiursrage2k

    Aiursrage2k

    Joined:
    Nov 1, 2009
    Posts:
    4,807
    Have you guys seen the new AWS, too bad gpus are still so expensive -- still stuck at 4 gpus to 48 cpus . Still too expensive to really take gaming to the next level where the average person no longer has a computer they can use there phone, tablet, tv. Maybe g5-g6's
    https://aws.amazon.com/ec2/instance-types/g4/
     
    Last edited: Sep 25, 2019
  35. kognito1

    kognito1

    Joined:
    Apr 7, 2015
    Posts:
    309
    So I made a small prototype with this plugin, but is there a reason why when it streams it seems to change the shader settings or something (for us it seems to make the shaders transparent)? Within the camera preview in the unity editor it's fine, but the result in the browser is obviously very wrong (see attached).
     

    Attached Files:

  36. Tobs-

    Tobs-

    Joined:
    Feb 12, 2016
    Posts:
    9
    Hi folks. Thanks for the great news, a Unity webRTC plugin :D Actually I was building the same stuff myself the last couple of month, but this code seems more suffisticated. I've refactored my project and incoporated the Unity webRTC plugin and finally got it running, but I am experiencing some weird video fragmentation issues. I am running on a NVidia 1080 with 1280x960@30fps. Turing the camera will cause some elements in the frame go wild and resharpen after the cam holds still. You guys have any suggestions to boost this? In Unity i simply use Graphics.Blit(source, destination) in the OnRenderImage method to render to the target texture.

    If you guys require an alternative solution for decoding the frame I could forward you some code. It simply uses an YUV I420 shader and the Unity C++ render backend reads the render texures holding th YUV frame. This solution will work on almost any graphics card, I easily can stream from my onboard intel 620 graphics card. Let me know if you need some code :)

    One more thing, how long you reckon until multi-camera targets are supported?

    Keep up the good work.

    Cheers,
    Tobs.
     
  37. sindharta_at_unity

    sindharta_at_unity

    Unity Technologies

    Joined:
    Jul 4, 2019
    Posts:
    12
    Renderstreaming does not modify shaders in any way.
    It just renders the objects into a RenderTexture and send it to the network.
    How does it look like in the Unity Editor itself (not the preview camera), and the RenderTexture itself in the editor ?
     
  38. sindharta_at_unity

    sindharta_at_unity

    Unity Technologies

    Joined:
    Jul 4, 2019
    Posts:
    12
    This might be caused by the decoding process. Can you let us know how far is the distance between the client and the server that you are using ? We'd also like to see how bad it is, so it would be great if you can attach a link to a video showing it.

    We welcome any contribution, so go ahead and share it here (or by creating a pull request directly). I am sure a lot of people would benefit from it before we start supporting other GPUs.

    Having support for multi-camera targets is one of our goals for version 2.0, so expect it to be launched in the next version.
     
  39. kognito1

    kognito1

    Joined:
    Apr 7, 2015
    Posts:
    309
    Forgot to mention that, it's completely black in the editor game window when the RenderStreaming script is enabled (but not the camera preview...which is why I attached the camera preview).
     
  40. Tobs-

    Tobs-

    Joined:
    Feb 12, 2016
    Posts:
    9
    Hi.
    So my setup is completly localhost. I am running the signaler and the unity app on my machine and connecting via the local installed chrome browser. So there is basically no distance in between.

    The resulting stream looks something like that:


    Cheers.
     
  41. Aiursrage2k

    Aiursrage2k

    Joined:
    Nov 1, 2009
    Posts:
    4,807
  42. gavanpb

    gavanpb

    Joined:
    Sep 27, 2019
    Posts:
    3
    I cannot seem to find how to edit the buttons / text labels for them. Can you point me in the right direction?
     
  43. kazuki_unity729

    kazuki_unity729

    Unity Technologies

    Joined:
    Aug 2, 2018
    Posts:
    11
  44. gavanpb

    gavanpb

    Joined:
    Sep 27, 2019
    Posts:
    3
    Thanks Kazuki!

    One more issue. The first time I ran this, it seemed to work fine. Now I am continuously getting:


    Network Error: Cannot connect to destination host
    UnityEngine.Debug:LogError(Object)
    Unity.RenderStreaming.<Start>d__15:MoveNext() (at Assets/Scripts/RenderStreaming.cs:83)
    UnityEngine.SetupCoroutine:InvokeMoveNext(IEnumerator, IntPtr)

    I've tried playing with different URLs for Url Signaling (localhost, local IP, different ports) and disabling Windows firewall, but I haven't gotten anything to work.
     
  45. kazuki_unity729

    kazuki_unity729

    Unity Technologies

    Joined:
    Aug 2, 2018
    Posts:
    11
    It seems the signaling server might not be running, please confirm whether you can access the URL from your browsers or not.
     
  46. gavanpb

    gavanpb

    Joined:
    Sep 27, 2019
    Posts:
    3
    When this happens, I cannot. It looks like if I run the server manually, it shows up, but without a video stream.
     
  47. sindharta_at_unity

    sindharta_at_unity

    Unity Technologies

    Joined:
    Jul 4, 2019
    Posts:
    12
    Please confirm how the RenderTexture looks like first.
    In our template, we provide a script that blits the render texture directly to screen, so it'll be great if this script works for you to confirm the result on Unity side.
     
  48. sindharta_at_unity

    sindharta_at_unity

    Unity Technologies

    Joined:
    Jul 4, 2019
    Posts:
    12
    That is the first time we have ever seen such a bad result.
    Did you make sure that you updated the NVidia drivers, browsers, etc ?
     
  49. Tobs-

    Tobs-

    Joined:
    Feb 12, 2016
    Posts:
    9
    Yeah, updated the graphics card driver to the latest version. The stream looks similar in Firefox and Chrome both are up to date. The only thing thats different though is the frontend page. I am not using the Unity provided web-app, but using my own, which simply is a video element receiving the stream. Do I need to do some buffering or extra settings in the frontend? From the resources I read this bad stream results seem to come from missed iframes? Also the default webRTC stream encoding is V9 ? But here we using H264 right? IS there some special configuration required when setting up webRTC?
     
  50. kazuki_unity729

    kazuki_unity729

    Unity Technologies

    Joined:
    Aug 2, 2018
    Posts:
    11
    It is strange.
    You can show up the web page from your browser, but you cannot access the server from Unity editor.
    Is this error still out?

    Network Error: Cannot connect to destination host
    UnityEngine.Debug:LogError(Object)
    Unity.RenderStreaming.<Start>d__15:MoveNext() (at Assets/Scripts/RenderStreaming.cs:83)
    UnityEngine.SetupCoroutine:InvokeMoveNext(IEnumerator, IntPtr)