Feedback WebRTC / RenderStreaming code improvements

I recently started using Unity Render Streaming and I want to share some thoughts and findings because on the Github issue page and here in the forum a lot of issues/questions are about similar topics and are posted again and again. Maybe my code changes (see next post) will help in one case or another.

First off, I'm really happy about Render Streaming. Having this functionality is really cool and opens up new use cases for us. It's already working pretty well even though some key features for real world use are still missing.
Especially improved and more stable hardware encoding because using software encoding for a high-resolution application running with 60fps is not really feasible. And not so much of a feature but a guide/sample on how to integrate an SFU/media server would also be necessary. Since we don't have experience with that it would probably take us a lot of time to achieve that.

What also took quite some time (more than feels necessary) was to find out about issues and workarounds for common problems. After reading the docs of the WebRTC and Render Streaming packages I had to read through all the Github issues and the forum as well to look for certain information. What was especially troublesome was to find the right information on how to improve the quality of the video stream. What is a bit annoying since video streaming is what it's all about. And there are quite a few Github issues about bad video quality. There is a short explanation about bitrate in the WebRTC docs but this should definitely be shown somewhere in the samples of the Render Streaming package. Because out of the box the hardware encoded stream looks pretty bad compared to the software encoded one.
So it would be good to know what the default parameters for the encoders are and which one can be changed and how. The fact that information and issues are split up between the WebRTC and Render Streaming packages makes it even a bit more difficult to find information. Maybe they could link to each other more often in certain topics.

 

So I want to share some changes/improvements I made to the WebRTC and Render Streaming code that worked for us. I used the source code of Render Streaming 3.1.0-exp.1 and WebRTC 2.4.0-exp.3 and the WebBrowserInputSample with the built-in render pipeline in Unity 2020.3.10f1:

• Expose VideoStreamBase.StreamingSize to set RenderTexture resolution via code
• Use VideoStreamBase.ChangeVideoParameters() to set encoding bitrate (and also downscale resolution when using software encoding, see WebBrowserInputSample.HandleStreamStarted())
• Free resources of video/audio tracks when a receiver/web browser is disconnected (see Github issue, Broadcast.Disconnect())
• Fallback to software encoding when hardware encoding is not possible on startup (see WebBrowserInputSample.StartRenderStreaming())
• Free all resources (render textures) when no one is receiving the stream anymore (see WebBrowserInputSample.HandleStreamStopped(), made CameraStreamer/StreamSourceBase disposable)
• Specify encoding resolution different from render texture resolution (hardware and software encoding, see EncodingResolution in WebBrowserInputSample and CameraStreamer, creates destination RenderTexture in VideoStreamTrack with different resolution than the camera's RenderTexture)

Feel free to use the files as they are or copy the code you need. Be aware that if you replace a file that you might get compilation errors because of possible API changes I made. Also I can only give you limited (if any) support if my changes don't work for you/your scenario.

 

Since I can only attach 5 files to a post, here is the last one:

 

thank you it's useful

 

Everything was ok in the unity editor but when I release PC ， I start to connect it and the picture quality is very poor and after 30 seconds later everything is fine 3.1.0-exp.1 and WebRTC 2.4.0-exp.3 @three10 @
kazuki_unity729

 

Everything was ok in the unity editor but when I release PC ， I start to connect it and the picture quality is very poor and after 30 seconds later everything is fine 3.1.0-exp.1 and WebRTC 2.4.0-exp.3 @three10 @
kazuki_unity729

 

like stream is not stable when starting connect

 

@kazuki_unity729 Sorry, I was away for a few days. Yes of course, you can use the code as you wish!

 

@chrisming999 Yes, I can confirm that when first receiving the stream it takes a few seconds before the quality of the stream improves. but for me it is really only about 3 seconds and not 30.

 

Thank you for your reply
Could you give me a small demo.my email address

 

@three10 Thanks for your sharing. I am very new to Unity and this Unity Render Streaming. I downloaded the same versions of these two packages. Could you please give me some instructions on how to use your files? Should I add these files to the RenderStreaming gameobject to replace the original files and keep the renderstreaming.cs? What about the videostreambase.cs, videostreamtrack.cs and StreamSourceBase.cs? I did not find them .

Thanks a lot.

 

Maybe the author has a different way to use those files, but this is how I did it:

1. Copy the com.unity.renderstreaming@3.1.0-exp.1 and the com.unity.webrtc@2.4.0-exp.3 folders from <Your Project Folder> / Library / PackageCache, then paste them into another directory
2. Replace some of the files inside those cloned packages with the ones the author provided
3. Put the cloned packages into <Your Project Folder> / Packages
4. In your project's Package Manager, click the (+) plus button on the top left corner and select Add Package from Disk
5. The resolution and the data rate will be exposed in the Inspector, through the altered WebBrowserInputSample.cs script, in the WebBrowserInputSample scene

We can set the resolution, but the frame rate is still not good. I have a heavy scene with around 30 fps on a desktop PC and it will drop to around 12 fps when broadcasting it with Unity Render Streaming. I hope the performance gets better in the next version.

 

Thank you so much for your kind reply! @liquify I really appreciate it.

I followed your instructions and copied the two packages in a different directory. Then I used the first 5 files provided by the author to replace the files in the com.unity.renderstreaming@3.1.0-exp.1/runtime/scripts and the last 1 for the com.unity.webrtc@2.4.0 exp.3/ runtime/scripts.

Then I created a new project. When I did the step3, I got an error in the WebBrowserInputSample.cs file:

CS0246: The type or namespace name 'CopyTransform' could not be found (are you missing a using directive or an assembly reference?).

I tried to skip step3 to do step4. I still got the same result.

Could you please tell me did I replace the files in a right way? Is step3 necessary?

 

Try to replace the WebBrowserInputSample.cs in the cloned package with the original one and replace the one in <Your Project folder> / Assets / Unity Render Streaming / 3.1.0-exp.1 / Example / WebBrowserInput folder instead. If the customized packages are shown in the Package Manager (see the screenshot below), step 3 and 4 won't be necessary:



If the errors persist:
1. Close the project
2. Delete the Library and the obj folders from the project
3. Delete all folders from from C:\Users\<Your Username>\AppData\Local\Unity\cache\packages\packages.unity.com
4. Reopen the project

 

Thanks a lot! It finally worked. Though when I tried to change the bitrate in the play mode, nothing got changed (observed from Chrome webrtc-internals). I guess there is still a lot of bugs and problems needed to be solved.

 

how to broadcast multi-camera with one Signaling server?

i have 2-3 camera to broadcast from unity to several browsers, now i need to use multi webserver.exe, with different ip-port pair.

Can i use one webserver.exe ? (maybe use some config connection-id or some i don't know)

Thank you!

 

Sorry @xiaolongyin, I haven't looked in here for a while. Thanks @liquify for helping out!
The way I imported the packages was by downloading the source code directly from the Github release pages, e.g. https://github.com/Unity-Technologies/UnityRenderStreaming/releases, and copying the necessary folders into the Unity project. So I didn't use the Package Manager at all for this.
The 'CopyTransform' script is part of the Render Streaming Samples and not in the main packages, so make sure to import/copy them as well. E.g. from the Zip from the repository you could just copy the whole 'UnityRenderStreaming-3.1.0-exp.1\com.unity.renderstreaming' folder.
But as you already got it to work this info comes a bit late.
Changing the bitrate of the stream works for me. Make sure to set it before starting the application and not afterwards. Also try debugging and setting a breakpoint at the position where the bitrate is set to make sure it is actually called. As @liquify already mentioned setting the frame rate doesn't work because it is not implemented by the package. Frame rate is basically just the same as the application's FPS. To change the streaming FPS probably a separate update loop for the stream independent of the application's update loop would have to be implemented.

 

@chopin1998 Probably not the right thread for this question But your requirement sounds pretty much the same as the WebBrowserInputSample from the Render Streaming package: https://docs.unity3d.com/Packages/com.unity.renderstreaming@3.1/manual/sample-browserinput.html
If you get this to work then you should have a good starting point. You only need to run the Unity application and one webserver.exe on the same machine then you can connect with as many browsers as you want as long as the performance is good enough. In our case we only use it in the LAN with a powerful host computer an can connect with up to 10 other machines to the stream with acceptable performance.

 

how to use those scripts?

 

Please have a look at the previous posts in this thread. @liquify already gave a good explanation on how to import the source code of the packages into your project to then be able to replace the necessary files.
Be aware that the scripts were only tested with the version mentioned in the post. I haven't had the time to test the newest versions of the WebRTC and Render Streaming packages. In the newest versions some things have been fixed and added already. So it might be a good idea to just use the latest versions of the packages. As soon as I have the time to test them I will check if my scripts are still necessary or if they need to be adapted for the newer versions.

 

still low bitrate aka poor image qulity?

 

@three10 You're my hero!

RE: 3.1.0-exp.2 poor video quality
https://github.com/Unity-Technologies/UnityRenderStreaming/issues/453#issuecomment-884873236

Specifically the HandleStreamStarted coroutine solved all my issues with poor quality. The only change I made to that was waiting a second instead of for end of frame because the encoding params were still empty at end of frame for me.

The munging of sdp to set the google bitrate values at the offer stage mentioned by others in a different thread did nothing for me.

Thankyou!

 

@three10 Thank you so much for your kind reply. I really appreciate it.

What I want to do is to change the bitrate whenever the network is not good. So is it possible to change the bitrate after starting the application anytime I want to do that? like some functions I could use in the scripts?

 

@PHARTGAMES Glad to hear that I could help

 

@xiaolongyin That's an interesting idea. I haven't tried it yet but will give it a go. Did you try and it didn't work for you?
I'm currently in the process of updating to the newest package versions of WebRTC and RenderStreaming. Maybe there's already something different now. If necessary I will post updated versions of my changes if they are still necessary.

 

Yes, I tried. The bitrate couldn't be changed after starting the application.
If you make it, please let me know. Thanks a lot.

 

Hello, I am experimenting with Render Streaming. After many hours of testing I see a stable behaviour: sometimes the Audio crashes and the resolution goes to bad quality. In all cases where the Audio stops working, bad image quality (artifacts like too much compressed or bad bandwidth). This bug happens stable. Is there any solution for us to put this on Production?

 

I have managed to make it work following the documentation and researching many hours about new concepts to me like WebRTC, hardware encoders, software encoders, NVENC and etc. Thank you so much for your work! !

 

Maybe it is a little confusing the documentation about Broadcast scene or Receiver scene at first , because I was jumping around Git or packages samples, so a scene like: "This is prepared for Streaming the game one shot" will be awesome or something like that. On the Broadcast scene (or it was bidirectional) I was testing, sometimes it connects and sometimes it doesn't connect. I didn't had too many time to check WebApp logs yet. Finally I decided to try Receiver Scene so I can connect, but I don't have the Remote capabilities of Broadcast sample. But it connects and I can stream. However I am not able to make the quality a little little higher, it is close but a little like Furioos will be cool. I know there can be limitations with Consumer GPU maybe and the Adaptive Bitrate Furioos is using or something like that? I can be wrong in everything because I don't have experience with WebGL WebRTC streaming (or any streaming) at all.