Official Unity Render Streaming Status Update (Q4 2021)

Hello everyone. This thread is for sharing with you about the development status of Unity Render Streaming, and the plans for 2022. If you have any questions about the 2022 project initiatives, please send them to this thread.

Many users have asked us about our product roadmap. I will try to answer in this thread as much as possible. If you have individual questions, please post another thread on the Unity Forum. If you have a bug report, please send it to GitHub Issues.

Looking back on 2021
In 2021, we mainly dealt with the following contents.

• Mobile platform support (iOS, Android)
• Integration with Input System
• Audio stream support

We have released 2.4.0-exp.4 for the WebRTC package, and 3.1.0-exp.2 for the Unity Render Streaming package. For details on the update, see the changelog of each package.

• Changelog - Unity Render Streaming
• Changelog - WebRTC

Contributors
We have received various feedback from many developers. We would like to thank all the developers who have a proven track record of pull requests being brought upstream in 2021.

• ZzXxZzKk
• rollersteaam
• juju812
• kant
• DavidMohrhardt
• leejeonghun
• timgates42
• tmiv
• Ziriax
• belveder79
• MFatihMAR
• akermen

WebGL platform support
At this moment, WebRTC package does not officially support the WebGL platform, but developers of the community have been developed.

• Experimental WebGL implementation by nvanofwegen · Pull Request # 478 · Unity-Technologies / com.unity.webrtc

We pay tribute to all the developers who have helped with WebGL support.

• nvanofwegen
• antoined73
• mukaschultze
• Gtk2k

Many other people have sent us detailed bug reports and checked for fixes. Thank you again.

Plan for 2022
In 2022, we plan to focus on improving stability and performance. In addition, there are two additional functions: hardware acceleration support for video codecs and metadata support.

If you have any other requests, please reply to this thread.

Stability
We have received many bug reports from developers and are working on them. Regarding the behavior of Unity2021.2 on Apple Silicon, we have confirmed that it is currently unstable. We will deal with it as a priority.

• [BUG] The Apple Silicon support is rarely showing picture when running in editor: never on Apple Silicon and extremely poorly on Intel Rosetta 2 · Issue # 559 · Unity-Technologies / com.unity.webrtc

Also, using hardware acceleration encoder issues haven't been addressed since they were first reported in 2020. We will raise the priority and respond. The related tickets are listed below.

• [BUG] Visual artifacts when using hardware encoding · Issue # 205 · Unity-Technologies / com.unity.webrtc
• Poor Video Quality (using Nvidia T4) · Issue # 446 · Unity-Technologies / UnityRenderStreaming
• Poor quality over good connection! · Issue # 450 · Unity-Technologies / UnityRenderStreaming

Performance measurement and improvement
We have received reports of delays in video streams compared to Google Chrome. We will continue to investigate and respond to problems.

• [BUG] Worse performance when using the Receiver scene compared to a web browser · Issue # 551 · Unity-Technologies / UnityRenderStreaming
• [BUG] I cannot get the Unity host to send more than about 2.5mbps to Chrome with ANY encoder · Issue # 552 · Unity-Technologies / com.unity.webrtc

We continue to improve audio delays with user feedback.

• [BUG] high latency for inbound audio track · Issue # 525 · Unity-Technologies / com.unity.webrtc

Hardware Acceleration Support
Many users have requested hardware acceleration video codecs.

• AMD (AMF) hardware encoding support? · Issue # 456 · Unity-Technologies / com.unity.webrtc
• [REQUEST] Use Media Foundation SDK for hardware accelerated video encode and decode on Windows · Issue # 558 · Unity-Technologies / com.unity .webrtc

We have also received requests for hardware decoder support.

• [REQUEST] h.264 decode support · Issue # 272 · Unity-Technologies / com.unity.webrtc

In addition, we have received requests for APIs for users to create their own codecs.

• [REQUEST] Expose incoming video stream within Unity (receiver) · Issue # 551 · Unity-Technologies / com.unity.webrtc

We discussed about the topic. I hope we can share our plan in the near future.

Metadata Support
Encoded Transforms API is to add metadata to Video Frame. The draft is published by W3C.

• WebRTC Encoded Transform Editor's Draft

Content related to metadata is also discussed in the Unity Forum. It is used to send information that needs to be synchronized with the video frame. For example, the timestamp on the stream sender and the location information of the camera.

• [Help] Render Streaming ARForum
• [REQUEST] Support Encoded Transform API · Issue # 305 · Unity-Technologies / com.unity.webrtc

Other Features
We plan to add the following in Unity Render Streaming.

Zero Configuration
To stream a Unity scene which the developer created, component settings are required. Simply add a package and you'll be able to stream instantly without any configuration.

Command line options
After creating the executable file, it was necessary to rebuild to change the settings such as signaling server URL and port. Providing command line options simplifies configuration when deploying the server.

Controlling the quality and resolution of the video stream
The video streaming couldn't freely choose the codec and bandwidth. Prepare an API that allows fine control.

Frequently Asked Question
We have received the following questions from users.

• Hololens2 support
• WebGL support
• Validated package
• Various hardware acceleration of the codec
• VR / AR platform support

Hololens2 support
Inquiry about the support of the Hololens2 has been asked from many users. It is discussed in this Issue.

• Windows UWP ARM64 support · Issue # 15 · Unity-Technologies / com.unity.webrtc

Mentioned in this issue, we are currently lowering our priority due to the high cost of supporting Windows ARM64.

WebGL support
Community-based development. You can check the operation by checking out the following pull request source branch.

• Experimental WebGL implementation by nvanofwegen · Pull Request # 478 · Unity-Technologies / com.unity.webrtc

Landing on the upstream is still difficult at the moment, but we will continue to work towards future support.

Verified Package
We have received questions about when to.

• Unity Render Streaming Introduction & FAQ - Unity Forum

Unity package is promoted to a validated package, it guarantees a 3-year long term support (LTS). Please see the following materials for the package life cycle.

• Package states and lifecycle -The Unity User Manual

The two packages are based on the package development cycle and are currently in the experimental stage. Development team aims to move to a validated package, but needs to clear challenges for long-term support and is currently working on it.

Support for various hardware codecs
The following hardware codecs are candidates, but the support time is not decided.

• VAAPI (Support for AMD GPU) Support
• Windows Media Foundation Codecs
• Intel Media SDK

VR / AR platform
VR/AR platform is not supported yet, but developers have already reported it. Latency issues have been reported with Oculus Quest 2.

• Improving render streaming latency from PC > The Quest 2 - Unity Forum

WebRTC package includes an AR Foundation sample, so please give it a try and give us feedback.

• AR Foundation Sample --Unity Render Streaming

In Closing
We hope this allows you to get a better understanding of how we are approaching development. Also, We wait for your feedback in this thread.

Last, best wishes for the holidays and New Year 2022!

 

Thank you for your feedback.
I would like to discuss about the priority of supporting HWA codecs.