Search Unity

  1. Good news ✨ We have more Unite Now videos available for you to watch on-demand! Come check them out and ask our experts any questions!
    Dismiss Notice
  2. Enter the 2020.2 Beta Sweepstakes for a chance to win an Oculus Quest 2.
    Dismiss Notice

Unity Render Streaming Introduction & FAQ

Discussion in 'Unity Render Streaming' started by kazuki_unity729, Sep 10, 2019.

  1. RocketPixel

    RocketPixel

    Joined:
    Jun 23, 2020
    Posts:
    5
    Thank you for your reply. Would this require changes from the unity engine or do you think it’s possible to integrate directly?
     
  2. princemioRocker

    princemioRocker

    Joined:
    Apr 1, 2020
    Posts:
    3
    Hey guys,

    i wanted to ask about the general plans of WebRTC @Unity. Since a couple of month there is an unstable version with varying quality - new feautures that then get reverted (SDP) and input stutter appearing. Unreal on the other hand is stable and production ready since a year. On Github i can see Kazuki doing an amazing job working very hard on new and old issues. But it looks like this Dev is alone with EVERY Bug of quite a complex technology. Is there anyone else at Unity who can support? So my general question is - what are the future plans of unity to invest into this technology.
     
  3. salvolannister

    salvolannister

    Joined:
    Jan 3, 2019
    Posts:
    19
    I'm still stacked with this issue, I did give up but then I saw this video and I am starting to ask myself If I did something wrong. I noticed that also in the example for URP the camera is rendering blue, but we don't notice because the stream is made from the other two cameras.

    Any idea or suggestion on how to fix this?
     
  4. Lukas_Kastern

    Lukas_Kastern

    Joined:
    Aug 31, 2018
    Posts:
    36
    Is the scaleResolutionDownBy parameter inside RTCRtpEncodingParameters already supported?
    When I set this parameter below one I get an invalid range error.
    Setting it above one results in the image being frozen, the input does still get transmitted.
     
  5. Maktech

    Maktech

    Joined:
    Jan 15, 2015
    Posts:
    19
    I have a situation where I am dynamically loading cameras when a client connects to the session. Typically the RTCPeerConnection has video / audio tracks that already are registered added to it during the OnOffer in RenderStreaming.cs. Now I want to add a new video track to an existing RTCPeerConnection. I have tried a number of variations but, the documentation (link below) says to add the track and then CreateAnswer. Do I need to then use the m_signaling to SendAnswer?

    https://docs.unity3d.com/Packages/com.unity.webrtc@2.1/manual/videostreaming.html

    Thank you
     
  6. kazuki_unity729

    kazuki_unity729

    Unity Technologies

    Joined:
    Aug 2, 2018
    Posts:
    148
    Could you tell me more detail your issue?
    - OS
    - Graphics API
    - Unity version
    - Package version (latest webrtc package version is 2.1.3-preview)
     
  7. kazuki_unity729

    kazuki_unity729

    Unity Technologies

    Joined:
    Aug 2, 2018
    Posts:
    148
    I am afraid the "scaleResolutionDownBy" parameter is not work when using Hardware encoder.
     
    Lukas_Kastern likes this.
  8. kazuki_unity729

    kazuki_unity729

    Unity Technologies

    Joined:
    Aug 2, 2018
    Posts:
    148
    Generally you need to renegotiate between two peers after the other track added.
    You can read an example code which is the pattern provided by W3C.
    https://w3c.github.io/webrtc-pc/#perfect-negotiation-example

    This pull request is for implementing renegotiation process. This feature will be released next version.
    https://github.com/Unity-Technologies/UnityRenderStreaming/pull/334/files
     
  9. pandaizzle

    pandaizzle

    Joined:
    Sep 1, 2020
    Posts:
    1
    What is the state of WebRTC @Unity for mac? I would like to use the pattern from the MediaStream sample but I noticed in the documentation that Unity WebRTC requires a NVDIA graphics card. Is there currently a way to make this work and if not, is it on the roadmap?
     
  10. Maktech

    Maktech

    Joined:
    Jan 15, 2015
    Posts:
    19
    We have developers on our team developing on the Mac. Use the latest Unity WebRTC 2.1.3 and make sure you use Software encoding. If you do have a Nvidia card in your Mac you can use the Hardware encoding. Good luck and happy streaming.
     
  11. fourbb

    fourbb

    Joined:
    Jul 22, 2020
    Posts:
    6
    Hello, I want to know how to set the encoding frame rate?
     
  12. Maktech

    Maktech

    Joined:
    Jan 15, 2015
    Posts:
    19
    In the latest version 2.1.3 (could be in an earlier version but, I have not checked). When using AddTrack on RTCPeerConnection it will return a RTCRtpSender. Using 'GetParameters' 'SetParameters' you can get access to the RTCRtpEncodingParameters.

    It gives you access to encoding settings where you can set min/max bitrate and framerate.

    https://docs.unity3d.com/Packages/com.unity.webrtc@2.1/api/Unity.WebRTC.RTCRtpSender.html
    https://docs.unity3d.com/Packages/c...pi/Unity.WebRTC.RTCRtpEncodingParameters.html
     
    kayy likes this.
  13. fourbb

    fourbb

    Joined:
    Jul 22, 2020
    Posts:
    6
    Thank you for your reply. It was very useful to me.
     
  14. salvolannister

    salvolannister

    Joined:
    Jan 3, 2019
    Posts:
    19
    - OS =Windows 10
    - Graphics API = Auto Graphics API for WIndows
    - Unity Version = 2019.4.10f
    - Package version = 2.1.1 preview

    Repro steps:
    --> Make a project with unity render streaming and import the URP package
    --> Add XR Interaction Toolkit and XR Interaction Toolkit management
    --> In Project Settings under XR Plugin Management add Oculus
    --> Open the sample scene and add an XR Rig room scale then press play

    This will enable you to use the controllers and the headset as inputs that can be seen also in the web, but nothing from the local server through the hmd that will show you a blue screen.
    Playing with the Rendering Priority in the Camera of the Rendering Streaming Camera I was able to see something also in the hmd camera ( from the hmd), but I see myself from a third point of view like in the picture below.

    My problem is that I don't want just to be able to see correctly from the server ( through the hmd) but to use also the hmd from the web, do you think I can achieve this in some way with this package?


    View from the game window in the server unity editor : render_result.png
     
  15. Maktech

    Maktech

    Joined:
    Jan 15, 2015
    Posts:
    19
    We have been successfully using Unity Render Streaming to setup our own platform (front end, database, session management, link automation, regions, cost management ...) for streaming our applications. We currently use AWS instances with Tesla T4 cards, with differing memory configurations. (see Nvidia support grid below)

    https://developer.nvidia.com/video-encode-decode-gpu-support-matrix

    The next big issue we are attempting to address is quality of our streams. We are tweaking the following data values:
    • Render Texture size (720p to 1080p). we are not looking to do 4K at this point.
    • Bitrates (x-google-start-bitrate and x-google-max-bitrate) using SDP munging.
    I am currently updating to the latest version (com.unity.webrtc 2.1.3 and integrating UnityRenderStreaming latest changes). We are removing the SDP munging and going to use the RTCRtpSender Get/SetParameters logic to set:
    • RTCRtpEncodingParameters - Frame Rate
    • RTCRtpEncodingParameters - Min Bit Rate
    • RTCRtpEncodingParameters - Max Bit Rate
    Considering we are attempting to maximize the quality and number of streams:
    • Are there other settings we can tweak to manage encoding settings, quality and performance?
    • Do we have more low level control of encoding hardware?
    • Besides chrome://webrtc-internals/ are there other tools to measure quality?
    • We have situations where the quality is fine and we get periods of corruption, do you have suggestions on how to handle dynamic changes in output?
    Thank you for all your work, we realize there is not a large team behind this effort.
     
    kayy likes this.
  16. kazuki_unity729

    kazuki_unity729

    Unity Technologies

    Joined:
    Aug 2, 2018
    Posts:
    148
    I am afraid we have never tested streaming feature on VR yet. This is new attempt for ours.
    I guess this issue is occuerred by browser. Which browser do you use on Oculus?
     
  17. kazuki_unity729

    kazuki_unity729

    Unity Technologies

    Joined:
    Aug 2, 2018
    Posts:
    148
    We recognize there are many issue about video quality and stability.
    Now, `RTCRtpEncodingParameters` that is all we can manage settings. In the near future we will add APIs to select codec and profile.

    No, we don't provide the features to control settings of hardware encoder.

    We developed Stats C# API and statistics tool.
    https://docs.unity3d.com/Packages/com.unity.webrtc@2.1/manual/statistics.html
    https://docs.unity3d.com/Packages/com.unity.webrtc@2.1/manual/webrtcstats.html
    Basically, libwebrtc tunes video quality automatically following bitrate.
    I guess it is related this issue. we are focusing on fixing it.
    https://github.com/Unity-Technologies/com.unity.webrtc/issues/205
     
  18. Maktech

    Maktech

    Joined:
    Jan 15, 2015
    Posts:
    19
    Thank you for the response. I am going down the path of making a branched version of the native plugin, giving us control of the NVENC settings, I am sure it is one of the efforts that are happening on your side. Looking forward to any future improvements you make. If I make strides with this effort I will share my findings with the forum.

    If you have any recommendations or suggestions, my first approach will be exposing the nvEncInitializeParams setup in NvEncoder.cpp. Want to be able to flip between the various encoders (H264 and HEVC) and available presets.

    Thank you for all your efforts and help.
     
  19. LazyOnion

    LazyOnion

    Joined:
    Mar 6, 2018
    Posts:
    22
    Is it possible to adaptively change the resolution or the bitrate?
     
  20. jalemanyf

    jalemanyf

    Joined:
    Jun 16, 2013
    Posts:
    18
  21. bubble8773

    bubble8773

    Joined:
    Feb 4, 2017
    Posts:
    2
    My Unity is crashing after playing the Render streaming project.
     
unityunity