Official Unity Render Streaming Introduction & FAQ

Thank you for your reply. Would this require changes from the unity engine or do you think it’s possible to integrate directly?

 

Hey guys,

i wanted to ask about the general plans of WebRTC @Unity. Since a couple of month there is an unstable version with varying quality - new feautures that then get reverted (SDP) and input stutter appearing. Unreal on the other hand is stable and production ready since a year. On Github i can see Kazuki doing an amazing job working very hard on new and old issues. But it looks like this Dev is alone with EVERY Bug of quite a complex technology. Is there anyone else at Unity who can support? So my general question is - what are the future plans of unity to invest into this technology.

 

I'm still stacked with this issue, I did give up but then I saw this video and I am starting to ask myself If I did something wrong. I noticed that also in the example for URP the camera is rendering blue, but we don't notice because the stream is made from the other two cameras.

Any idea or suggestion on how to fix this?

 

Is the scaleResolutionDownBy parameter inside RTCRtpEncodingParameters already supported?
When I set this parameter below one I get an invalid range error.
Setting it above one results in the image being frozen, the input does still get transmitted.

 

I have a situation where I am dynamically loading cameras when a client connects to the session. Typically the RTCPeerConnection has video / audio tracks that already are registered added to it during the OnOffer in RenderStreaming.cs. Now I want to add a new video track to an existing RTCPeerConnection. I have tried a number of variations but, the documentation (link below) says to add the track and then CreateAnswer. Do I need to then use the m_signaling to SendAnswer?

https://docs.unity3d.com/Packages/com.unity.webrtc@2.1/manual/videostreaming.html

Thank you

 

What is the state of WebRTC @Unity for mac? I would like to use the pattern from the MediaStream sample but I noticed in the documentation that Unity WebRTC requires a NVDIA graphics card. Is there currently a way to make this work and if not, is it on the roadmap?

 

We have developers on our team developing on the Mac. Use the latest Unity WebRTC 2.1.3 and make sure you use Software encoding. If you do have a Nvidia card in your Mac you can use the Hardware encoding. Good luck and happy streaming.

 

Hello, I want to know how to set the encoding frame rate?

 

In the latest version 2.1.3 (could be in an earlier version but, I have not checked). When using AddTrack on RTCPeerConnection it will return a RTCRtpSender. Using 'GetParameters' 'SetParameters' you can get access to the RTCRtpEncodingParameters.

It gives you access to encoding settings where you can set min/max bitrate and framerate.

https://docs.unity3d.com/Packages/com.unity.webrtc@2.1/api/Unity.WebRTC.RTCRtpSender.html
https://docs.unity3d.com/Packages/c...pi/Unity.WebRTC.RTCRtpEncodingParameters.html

 

Thank you for your reply. It was very useful to me.

 

- OS =Windows 10
- Graphics API = Auto Graphics API for WIndows
- Unity Version = 2019.4.10f
- Package version = 2.1.1 preview

Repro steps:
--> Make a project with unity render streaming and import the URP package
--> Add XR Interaction Toolkit and XR Interaction Toolkit management
--> In Project Settings under XR Plugin Management add Oculus
--> Open the sample scene and add an XR Rig room scale then press play

This will enable you to use the controllers and the headset as inputs that can be seen also in the web, but nothing from the local server through the hmd that will show you a blue screen.
Playing with the Rendering Priority in the Camera of the Rendering Streaming Camera I was able to see something also in the hmd camera ( from the hmd), but I see myself from a third point of view like in the picture below.

My problem is that I don't want just to be able to see correctly from the server ( through the hmd) but to use also the hmd from the web, do you think I can achieve this in some way with this package?


View from the game window in the server unity editor :

 

We have been successfully using Unity Render Streaming to setup our own platform (front end, database, session management, link automation, regions, cost management ...) for streaming our applications. We currently use AWS instances with Tesla T4 cards, with differing memory configurations. (see Nvidia support grid below)

https://developer.nvidia.com/video-encode-decode-gpu-support-matrix

The next big issue we are attempting to address is quality of our streams. We are tweaking the following data values:

• Render Texture size (720p to 1080p). we are not looking to do 4K at this point.
• Bitrates (x-google-start-bitrate and x-google-max-bitrate) using SDP munging.

I am currently updating to the latest version (com.unity.webrtc 2.1.3 and integrating UnityRenderStreaming latest changes). We are removing the SDP munging and going to use the RTCRtpSender Get/SetParameters logic to set:

• RTCRtpEncodingParameters - Frame Rate
• RTCRtpEncodingParameters - Min Bit Rate
  
• RTCRtpEncodingParameters - Max Bit Rate

Considering we are attempting to maximize the quality and number of streams:

• Are there other settings we can tweak to manage encoding settings, quality and performance?
• Do we have more low level control of encoding hardware?
• Besides chrome://webrtc-internals/ are there other tools to measure quality?
• We have situations where the quality is fine and we get periods of corruption, do you have suggestions on how to handle dynamic changes in output?

Thank you for all your work, we realize there is not a large team behind this effort.

 

Thank you for the response. I am going down the path of making a branched version of the native plugin, giving us control of the NVENC settings, I am sure it is one of the efforts that are happening on your side. Looking forward to any future improvements you make. If I make strides with this effort I will share my findings with the forum.

If you have any recommendations or suggestions, my first approach will be exposing the nvEncInitializeParams setup in NvEncoder.cpp. Want to be able to flip between the various encoders (H264 and HEVC) and available presets.

Thank you for all your efforts and help.

 

Is it possible to adaptively change the resolution or the bitrate?

 

Hi all,

I'm interested to use webRTC for a project but, where can I find a Functional Project like the one that appears on the blog? https://blogs.unity3d.com/2019/09/1...988.127638938.1603090313-333205416.1594282419

The samples that come on the github are very spartans. I need something more "spectacular" to show at my boss

Thanks!

 

My Unity is crashing after playing the Render streaming project.

 

Hi kazuki,

I tried but i can't follow well the instructions:

I have installed de WebRTC 2.1.3 preview packege on Unity 2019.4.12f1 without problems but...:

-I can't find the 2 templates to copy on the directory...

-I don't have the menú option "Render Streaming / Download web App".

Which step I have missed?

Thanks in advance,

 

We have been more successful using VP8 (software encoding) as of late. The new issue we get with it is the start bitrate is extremely low and the ramp up time to something reasonable can be a minute or more. I have read the default start bit rate for VP8 is 20 kbps.

Do we have access to modifying the start / min / max on software encoding?
Would SDP munging be the better route? Has anyone had success with tweaking software encoding settings?

Looking at the native plugin I see that function on the software encoder has no implementation.

Thanks again for all your work.

 

Hi kazuki,

Thanks for your quick response.

I could complete correctly the next step! Yeaaaah!!! but..:

1.-The menu option "Edit / Render Streaming / Download web app" never could be completed. Always give me an "Input string was not in a correct format". With a new folder or an old folder the system don't recognise it. I'm doing in windows 10.

2.-The option "Edit / Render Streaming / Show web app documentation" returns an 404 error. It seems the link is broken.

( Now, I'm going to download the web server.exe from the same place of the templates and run the node project from the shell prompt. )

Sorry for the inconvenience. Thanks a lot for your help! I think your project is really interesting

King Regards

 

Hi Kazuki,

I done it! Works like a charm!!!

I downloaded the server from the same place of the templates and just jumped the menus steps of the tuttorial, directly run the server and changing the IP address on the script!

If you want, I can translate the documentation to spanish ( if it helps you to get more public ).

Now, I have to see how to implement in my project

Thanks for all and have a nice weekend.

 

all in all... but how do you run the

webserver_mac

? ...on Mac obviously :|

 

you have to make it executable before you can run it, to do so change directory to where the file is and:

chmod +x ./webserver_mac
./webserver_mac

that opens the webserver for port 80, if you want to use another port (on linux for an example, i cant use 80, so i need to set 8000 or similar), you have to set the port in the rendersettings in the project, and also add -p <port-number> when running the webserver

 

thank you @subjectdenied - all good

 

Hi,

i installed the Template
com.unity.template.renderstreaming-rtx-2.1.1-preview.tgz
to
<Unity Editor Root>/Data/Resources/PackageManager/ProjectTemplates

Downloaded manually the webserver to a desktop location and started the webserver.exe.

because Edit/Render Streaming/Download web app is broken and end in an file browser window you cannot close anymore.

Then i started the example scene but there is no streaming in the browser windows.

The only error i see is
Assertion failed on expression: 'sceneRanges.empty()'

Do you have an idea what´s wrong here?
I tried to catch all needed info in the screenshot.

 

I aslo downladed manualy the web server.exe.

Can you explain this step you did ?
"run the node project from the shell prompt"

 

There are pack_webapp.cmd or .sh files in the root of the UnityRenderStreaming github

https://github.com/Unity-Technologies/UnityRenderStreaming

Also, if you look in the WebApp directory look at the run.bat, you can use that to run the webserver locally from the command line.

 

Hi, is there a way to set custom headers for the socket connection? I want to send an authentication token with it which I can't seem to find a way to do

 

Thanks a lot, that did the job.
I leave this here because it s not self explainig.)

I installed
nmp from
https://www.npmjs.com/get-npm

and startet
UnityRenderStreaming/test_webapp

The compiled webserver.exe did not work for me.

 

Thanks for the response. I checked the WebSocketSignaling class and apparently WebSocket class only lets me set the cookie. There is no option to add headers, would I possibly need to try other socket library?

 

Hi! I'm trying to figure out how to use the WebRTC Plugin (without streaming) to receive audio and video from a URL stream using the software encoder.

I'm trying to use the MediaStream sample from 2.2.0-preview, but when I switch it to use the software encoder, I get exceptions such as the following:

Error Detail Type: Failed to set remote offer sdp: Called in wrong state: kHaveLocalOffer
UnityEngine.Debug:LogError(Object)
MediaStreamSample:OnSetSessionDescriptionError(RTCError&) (at Assets/Samples/WebRTC/2.2.0-preview/Example/MediaStreamSample.cs:284)
<OnCreateOfferSuccess>d__39:MoveNext() (at Assets/Samples/WebRTC/2.2.0-preview/Example/MediaStreamSample.cs:253)
UnityEngine.SetupCoroutine:InvokeMoveNext(IEnumerator, IntPtr)


Error Detail Type: PeerConnection cannot create an answer in a state other than have-remote-offer or have-local-pranswer.
UnityEngine.Debug:LogError(Object)
MediaStreamSample:OnCreateSessionDescriptionError(RTCError) (at Assets/Samples/WebRTC/2.2.0-preview/Example/MediaStreamSample.cs:327)
<OnCreateOfferSuccess>d__39:MoveNext() (at Assets/Samples/WebRTC/2.2.0-preview/Example/MediaStreamSample.cs:268)
UnityEngine.SetupCoroutine:InvokeMoveNext(IEnumerator, IntPtr)

Is this scenario currently supported with the software encoder right now? I already have a WebRTC Server set up (with a url) that I'm trying to receive a stream from. I would really appreciate guidance on what I need to use and implement to properly receive the stream using the WebRTC plugin in its current state.

 

I'm using Windows 10 Enterprise with Unity Version: 2019.4.2f1.

 

Hello everyone, could please anyone advise on the following error?
Error Detail Type: Failed to set remote answer sdp: Failed to set remote video description send parameters for m-section with mid='1'.

 

Also, is there example of how to play an audio with AudioStreamTrack?

 

Hi,
Very pleased to see Unity implementing open platforms, like WebRTC. Is there any estimated time for iOS support?
Thanks in advance,

 

Hi,
Is it possible to use Unity WebRTC as client on a unity app and built for WebGL, iOS and Android?