Search Unity

  1. Welcome to the Unity Forums! Please take the time to read our Code of Conduct to familiarize yourself with the forum rules and how to post constructively.
  2. The 2023.1 beta is now available for testing. To find out what's new, have a look at our 2023.1 feature highlights.
    Dismiss Notice

Feedback WebRTC / RenderStreaming code improvements

Discussion in 'Unity Render Streaming' started by three10, Jul 22, 2021.

  1. three10

    three10

    Joined:
    Jan 3, 2014
    Posts:
    21
    I recently started using Unity Render Streaming and I want to share some thoughts and findings because on the Github issue page and here in the forum a lot of issues/questions are about similar topics and are posted again and again. Maybe my code changes (see next post) will help in one case or another.

    First off, I'm really happy about Render Streaming. Having this functionality is really cool and opens up new use cases for us. It's already working pretty well even though some key features for real world use are still missing.
    Especially improved and more stable hardware encoding because using software encoding for a high-resolution application running with 60fps is not really feasible. And not so much of a feature but a guide/sample on how to integrate an SFU/media server would also be necessary. Since we don't have experience with that it would probably take us a lot of time to achieve that.

    What also took quite some time (more than feels necessary) was to find out about issues and workarounds for common problems. After reading the docs of the WebRTC and Render Streaming packages I had to read through all the Github issues and the forum as well to look for certain information. What was especially troublesome was to find the right information on how to improve the quality of the video stream. What is a bit annoying since video streaming is what it's all about. And there are quite a few Github issues about bad video quality. There is a short explanation about bitrate in the WebRTC docs but this should definitely be shown somewhere in the samples of the Render Streaming package. Because out of the box the hardware encoded stream looks pretty bad compared to the software encoded one.
    So it would be good to know what the default parameters for the encoders are and which one can be changed and how. The fact that information and issues are split up between the WebRTC and Render Streaming packages makes it even a bit more difficult to find information. Maybe they could link to each other more often in certain topics.
     
    liquify and CloudyVR like this.
  2. three10

    three10

    Joined:
    Jan 3, 2014
    Posts:
    21
    So I want to share some changes/improvements I made to the WebRTC and Render Streaming code that worked for us. I used the source code of Render Streaming 3.1.0-exp.1 and WebRTC 2.4.0-exp.3 and the WebBrowserInputSample with the built-in render pipeline in Unity 2020.3.10f1:
    • Expose VideoStreamBase.StreamingSize to set RenderTexture resolution via code
    • Use VideoStreamBase.ChangeVideoParameters() to set encoding bitrate (and also downscale resolution when using software encoding, see WebBrowserInputSample.HandleStreamStarted())
    • Free resources of video/audio tracks when a receiver/web browser is disconnected (see Github issue, Broadcast.Disconnect())
    • Fallback to software encoding when hardware encoding is not possible on startup (see WebBrowserInputSample.StartRenderStreaming())
    • Free all resources (render textures) when no one is receiving the stream anymore (see WebBrowserInputSample.HandleStreamStopped(), made CameraStreamer/StreamSourceBase disposable)
    • Specify encoding resolution different from render texture resolution (hardware and software encoding, see EncodingResolution in WebBrowserInputSample and CameraStreamer, creates destination RenderTexture in VideoStreamTrack with different resolution than the camera's RenderTexture)
    Feel free to use the files as they are or copy the code you need. Be aware that if you replace a file that you might get compilation errors because of possible API changes I made. Also I can only give you limited (if any) support if my changes don't work for you/your scenario.
     

    Attached Files:

    liquify likes this.
  3. three10

    three10

    Joined:
    Jan 3, 2014
    Posts:
    21
    Since I can only attach 5 files to a post, here is the last one:
     

    Attached Files:

  4. chrisming999

    chrisming999

    Joined:
    Jan 15, 2021
    Posts:
    10
    thank you it's useful
     
  5. kazuki_unity729

    kazuki_unity729

    Unity Technologies

    Joined:
    Aug 2, 2018
    Posts:
    581
    @three10
    Thanks for your feedback and we will refer to your code and comments for the next release.
     
  6. kazuki_unity729

    kazuki_unity729

    Unity Technologies

    Joined:
    Aug 2, 2018
    Posts:
    581
    @three10
    Can I make a branch that committing your changes?
     
  7. chrisming999

    chrisming999

    Joined:
    Jan 15, 2021
    Posts:
    10
    Everything was ok in the unity editor but when I release PC , I start to connect it and the picture quality is very poor and after 30 seconds later everything is fine 3.1.0-exp.1 and WebRTC 2.4.0-exp.3 @three10 @
    kazuki_unity729
     
  8. chrisming999

    chrisming999

    Joined:
    Jan 15, 2021
    Posts:
    10
    Everything was ok in the unity editor but when I release PC , I start to connect it and the picture quality is very poor and after 30 seconds later everything is fine 3.1.0-exp.1 and WebRTC 2.4.0-exp.3 @three10 @
    kazuki_unity729
     
  9. chrisming999

    chrisming999

    Joined:
    Jan 15, 2021
    Posts:
    10
    like stream is not stable when starting connect
     
  10. three10

    three10

    Joined:
    Jan 3, 2014
    Posts:
    21
    @kazuki_unity729 Sorry, I was away for a few days. Yes of course, you can use the code as you wish!
     
  11. three10

    three10

    Joined:
    Jan 3, 2014
    Posts:
    21
    @chrisming999 Yes, I can confirm that when first receiving the stream it takes a few seconds before the quality of the stream improves. but for me it is really only about 3 seconds and not 30.
     
  12. kazuki_unity729

    kazuki_unity729

    Unity Technologies

    Joined:
    Aug 2, 2018
    Posts:
    581
  13. chrisming999

    chrisming999

    Joined:
    Jan 15, 2021
    Posts:
    10
    Thank you for your reply
    Could you give me a small demo.my email address
     
    Last edited: Dec 30, 2021
  14. xiaolongyin

    xiaolongyin

    Joined:
    Jan 18, 2022
    Posts:
    5
    @three10 Thanks for your sharing. I am very new to Unity and this Unity Render Streaming. I downloaded the same versions of these two packages. Could you please give me some instructions on how to use your files? Should I add these files to the RenderStreaming gameobject to replace the original files and keep the renderstreaming.cs? What about the videostreambase.cs, videostreamtrack.cs and StreamSourceBase.cs? I did not find them :(.

    Thanks a lot.
     
  15. liquify

    liquify

    Joined:
    Dec 9, 2014
    Posts:
    139
    Maybe the author has a different way to use those files, but this is how I did it:

    1. Copy the com.unity.renderstreaming@3.1.0-exp.1 and the com.unity.webrtc@2.4.0-exp.3 folders from <Your Project Folder> / Library / PackageCache, then paste them into another directory
    2. Replace some of the files inside those cloned packages with the ones the author provided
    3. Put the cloned packages into <Your Project Folder> / Packages
    4. In your project's Package Manager, click the (+) plus button on the top left corner and select Add Package from Disk
    5. The resolution and the data rate will be exposed in the Inspector, through the altered WebBrowserInputSample.cs script, in the WebBrowserInputSample scene

    We can set the resolution, but the frame rate is still not good. I have a heavy scene with around 30 fps on a desktop PC and it will drop to around 12 fps when broadcasting it with Unity Render Streaming. I hope the performance gets better in the next version.
     
  16. xiaolongyin

    xiaolongyin

    Joined:
    Jan 18, 2022
    Posts:
    5
    Thank you so much for your kind reply! @liquify I really appreciate it.

    I followed your instructions and copied the two packages in a different directory. Then I used the first 5 files provided by the author to replace the files in the com.unity.renderstreaming@3.1.0-exp.1/runtime/scripts and the last 1 for the com.unity.webrtc@2.4.0 exp.3/ runtime/scripts.

    Then I created a new project. When I did the step3, I got an error in the WebBrowserInputSample.cs file:

    CS0246: The type or namespace name 'CopyTransform' could not be found (are you missing a using directive or an assembly reference?).

    I tried to skip step3 to do step4. I still got the same result.

    Could you please tell me did I replace the files in a right way? Is step3 necessary?
     
  17. liquify

    liquify

    Joined:
    Dec 9, 2014
    Posts:
    139
    Try to replace the WebBrowserInputSample.cs in the cloned package with the original one and replace the one in <Your Project folder> / Assets / Unity Render Streaming / 3.1.0-exp.1 / Example / WebBrowserInput folder instead. If the customized packages are shown in the Package Manager (see the screenshot below), step 3 and 4 won't be necessary:

    upload_2022-1-20_7-46-58.png

    If the errors persist:
    1. Close the project
    2. Delete the Library and the obj folders from the project
    3. Delete all folders from from C:\Users\<Your Username>\AppData\Local\Unity\cache\packages\packages.unity.com
    4. Reopen the project
     
  18. xiaolongyin

    xiaolongyin

    Joined:
    Jan 18, 2022
    Posts:
    5
    Thanks a lot! It finally worked. Though when I tried to change the bitrate in the play mode, nothing got changed :( (observed from Chrome webrtc-internals). I guess there is still a lot of bugs and problems needed to be solved.
     
  19. chopin1998

    chopin1998

    Joined:
    Jan 29, 2019
    Posts:
    20
    how to broadcast multi-camera with one Signaling server?

    i have 2-3 camera to broadcast from unity to several browsers, now i need to use multi webserver.exe, with different ip-port pair.

    Can i use one webserver.exe ? (maybe use some config connection-id or some i don't know)

    Thank you!
     
  20. three10

    three10

    Joined:
    Jan 3, 2014
    Posts:
    21
    Sorry @xiaolongyin, I haven't looked in here for a while. Thanks @liquify for helping out!
    The way I imported the packages was by downloading the source code directly from the Github release pages, e.g. https://github.com/Unity-Technologies/UnityRenderStreaming/releases, and copying the necessary folders into the Unity project. So I didn't use the Package Manager at all for this.
    The 'CopyTransform' script is part of the Render Streaming Samples and not in the main packages, so make sure to import/copy them as well. E.g. from the Zip from the repository you could just copy the whole 'UnityRenderStreaming-3.1.0-exp.1\com.unity.renderstreaming' folder.
    But as you already got it to work this info comes a bit late.
    Changing the bitrate of the stream works for me. Make sure to set it before starting the application and not afterwards. Also try debugging and setting a breakpoint at the position where the bitrate is set to make sure it is actually called. As @liquify already mentioned setting the frame rate doesn't work because it is not implemented by the package. Frame rate is basically just the same as the application's FPS. To change the streaming FPS probably a separate update loop for the stream independent of the application's update loop would have to be implemented.
     
  21. three10

    three10

    Joined:
    Jan 3, 2014
    Posts:
    21
    @chopin1998 Probably not the right thread for this question :) But your requirement sounds pretty much the same as the WebBrowserInputSample from the Render Streaming package: https://docs.unity3d.com/Packages/com.unity.renderstreaming@3.1/manual/sample-browserinput.html
    If you get this to work then you should have a good starting point. You only need to run the Unity application and one webserver.exe on the same machine then you can connect with as many browsers as you want as long as the performance is good enough. In our case we only use it in the LAN with a powerful host computer an can connect with up to 10 other machines to the stream with acceptable performance.
     
  22. chopin1998

    chopin1998

    Joined:
    Jan 29, 2019
    Posts:
    20

    how to use those scripts?
     
  23. three10

    three10

    Joined:
    Jan 3, 2014
    Posts:
    21
    Please have a look at the previous posts in this thread. @liquify already gave a good explanation on how to import the source code of the packages into your project to then be able to replace the necessary files.
    Be aware that the scripts were only tested with the version mentioned in the post. I haven't had the time to test the newest versions of the WebRTC and Render Streaming packages. In the newest versions some things have been fixed and added already. So it might be a good idea to just use the latest versions of the packages. As soon as I have the time to test them I will check if my scripts are still necessary or if they need to be adapted for the newer versions.
     
  24. chopin1998

    chopin1998

    Joined:
    Jan 29, 2019
    Posts:
    20
    still low bitrate aka poor image qulity?
     
  25. PHARTGAMES

    PHARTGAMES

    Joined:
    Jan 30, 2013
    Posts:
    5
    @three10 You're my hero!

    RE: 3.1.0-exp.2 poor video quality
    https://github.com/Unity-Technologies/UnityRenderStreaming/issues/453#issuecomment-884873236

    Specifically the HandleStreamStarted coroutine solved all my issues with poor quality. The only change I made to that was waiting a second instead of for end of frame because the encoding params were still empty at end of frame for me.

    The munging of sdp to set the google bitrate values at the offer stage mentioned by others in a different thread did nothing for me.

    Thankyou!
     
    three10 likes this.
  26. xiaolongyin

    xiaolongyin

    Joined:
    Jan 18, 2022
    Posts:
    5
    @three10 Thank you so much for your kind reply. I really appreciate it.

    What I want to do is to change the bitrate whenever the network is not good. So is it possible to change the bitrate after starting the application anytime I want to do that? like some functions I could use in the scripts?
     
  27. three10

    three10

    Joined:
    Jan 3, 2014
    Posts:
    21
    @PHARTGAMES Glad to hear that I could help :)
     
  28. three10

    three10

    Joined:
    Jan 3, 2014
    Posts:
    21
    @xiaolongyin That's an interesting idea. I haven't tried it yet but will give it a go. Did you try and it didn't work for you?
    I'm currently in the process of updating to the newest package versions of WebRTC and RenderStreaming. Maybe there's already something different now. If necessary I will post updated versions of my changes if they are still necessary.
     
  29. xiaolongyin

    xiaolongyin

    Joined:
    Jan 18, 2022
    Posts:
    5
    Yes, I tried. The bitrate couldn't be changed after starting the application.
    If you make it, please let me know. Thanks a lot.