Search Unity

  1. Welcome to the Unity Forums! Please take the time to read our Code of Conduct here to familiarize yourself with the rules and how to post constructively.

  2. Unity 2022.1 is now available as the latest Tech release.
    Dismiss Notice

Help Wanted Resize VideoStreamTrack and ReplaceTrack, framerate drops

Discussion in 'Unity Render Streaming' started by ypvypv, Jul 22, 2021.

  1. ypvypv

    ypvypv

    Joined:
    Apr 16, 2019
    Posts:
    7
    We have a requirement where we need to resize canvas on the server and in the browser in order to maximize quality and framerate of the video stream. In other words, resizing just browser video element is not useful because the resolution of the canvas on the server remains the same.

    The idea is to create a differently sized UnityEngine.RenderTexture and create a VideoStreamTrack, then RTCRtpSender.ReplaceTrack(track). But there is framerate drop from 30 fps to 1 fps, regardless of size of texture. The framerate drop is only in webrtc transmission and unity backend renders at the same framerate.

    We are using Render Streaming package 3.0.1 as well as webrtc 2.3.3-preview. An interesting detail - when a webrtc client connects to Unity and negotiates a new connection, the framerate is back to normal 30 fps at the new, modified resolution.

    Here is the relevant code:
    Code (CSharp):
    1.  
    2.  
    3. //in RenderStreamingInternal.cs
    4. public bool ReplaceAllSenderTracks(MediaStreamTrack track)
    5. {
    6.       foreach(var entry in _mapConnectionIdAndPeer)
    7.      {
    8.           var each_senders = entry.Value.GetSenders();
    9.           foreach (var sender in each_senders)
    10.           {
    11.                if(sender.Track.Kind == track.Kind)
    12.                     sender.ReplaceTrack(track);
    13.           }
    14.      }
    15.      return true;
    16. }
    17.  
    Code (CSharp):
    1.  
    2. //in CameraStreamer.cs
    3. public void setResolution( int x, int y ){
    4.     streamingSize.x = x;
    5.     streamingSize.y = y;
    6.     Track = CreateTrack();
    7. }
    8.  
    9. protected override MediaStreamTrack CreateTrack()
    10. {
    11.     RenderTexture rt;
    12.     if (m_camera.targetTexture != null && m_camera.targetTexture.width == streamingSize.x && m_camera.targetTexture.height == streamingSize.y )
    13.     {
    14.         rt = m_camera.targetTexture;
    15.         RenderTextureFormat supportFormat = WebRTC.WebRTC.GetSupportedRenderTextureFormat(SystemInfo.graphicsDeviceType);
    16.         GraphicsFormat graphicsFormat = GraphicsFormatUtility.GetGraphicsFormat(supportFormat, RenderTextureReadWrite.Default);
    17.         GraphicsFormat compatibleFormat = SystemInfo.GetCompatibleFormat(graphicsFormat, FormatUsage.Render);
    18.         GraphicsFormat format = graphicsFormat == compatibleFormat ? graphicsFormat : compatibleFormat;
    19.  
    20.         if (rt.graphicsFormat != format)
    21.         {
    22.             Debug.LogWarning(
    23.                 $"This color format:{rt.graphicsFormat} not support in unity.webrtc. Change to supported color format:{format}.");
    24.             rt.Release();
    25.             rt.graphicsFormat = format;
    26.             rt.Create();
    27.         }
    28.  
    29.         m_camera.targetTexture = rt;
    30.     }
    31.     else
    32.     {
    33.         var format = WebRTC.WebRTC.GetSupportedRenderTextureFormat(SystemInfo.graphicsDeviceType);
    34.         rt = new RenderTexture(streamingSize.x, streamingSize.y, depth, format)
    35.         {
    36.             antiAliasing = antiAliasing
    37.         };
    38.         rt.Create();
    39.  
    40.  
    41.         RenderTexture temp_target_texture = null;
    42.         if(m_camera.targetTexture!=null)
    43.             temp_target_texture = m_camera.targetTexture;
    44.  
    45.         m_camera.targetTexture = rt;
    46.  
    47.         if(temp_target_texture!=null)
    48.             temp_target_texture.Release();
    49.     }
    50.  
    51.     return new VideoStreamTrack(m_camera.name, rt);
    52. }
    53.  
    Code (CSharp):
    1. //invoking the code
    2.    cameraStreamer.setResolution(X, Y);
    3.    renderStreaming.ReplaceAllSenderTracks( cameraStreamer.Track );
    4.  
     
  2. teexiii

    teexiii

    Joined:
    Oct 7, 2015
    Posts:
    15
    Did you manage to figure out the solution @ypvypv
     
    ypvypv likes this.
  3. kazuki_unity729

    kazuki_unity729

    Unity Technologies

    Joined:
    Aug 2, 2018
    Posts:
    414
    Hi, Could you try the latest version of the package?
    I would like to know whether reproducing the issue for the latest package or not.
     
    ypvypv likes this.
  4. ypvypv

    ypvypv

    Joined:
    Apr 16, 2019
    Posts:
    7
    No, I have not.
     
  5. ypvypv

    ypvypv

    Joined:
    Apr 16, 2019
    Posts:
    7
    Yes, I have tried 3.1.0-exp.2 and the Replace Track still brings down the framerate. What change would fix this? libwebrtc package upgrade?
     
  6. kazuki_unity729

    kazuki_unity729

    Unity Technologies

    Joined:
    Aug 2, 2018
    Posts:
    414
    @ypvypv
    OK, thanks.
    Have you created the thread on the GitHub Issues?
    And I am glad if you have the simple project to reproduce this issue.
     
    ypvypv likes this.
unityunity