Question Resize VideoStreamTrack and ReplaceTrack, framerate drops

We have a requirement where we need to resize canvas on the server and in the browser in order to maximize quality and framerate of the video stream. In other words, resizing just browser video element is not useful because the resolution of the canvas on the server remains the same.

The idea is to create a differently sized UnityEngine.RenderTexture and create a VideoStreamTrack, then RTCRtpSender.ReplaceTrack(track). But there is framerate drop from 30 fps to 1 fps, regardless of size of texture. The framerate drop is only in webrtc transmission and unity backend renders at the same framerate.

We are using Render Streaming package 3.0.1 as well as webrtc 2.3.3-preview. An interesting detail - when a webrtc client connects to Unity and negotiates a new connection, the framerate is back to normal 30 fps at the new, modified resolution.

Here is the relevant code:

Code (CSharp):

1.  
2.  
3. //in RenderStreamingInternal.cs
4. public bool ReplaceAllSenderTracks(MediaStreamTrack track)
5. {
6.       foreach(var entry in _mapConnectionIdAndPeer)
7.      {
8.           var each_senders = entry.Value.GetSenders();
9.           foreach (var sender in each_senders)
10.           {
11.                if(sender.Track.Kind == track.Kind)
12.                     sender.ReplaceTrack(track);
13.           }
14.      }
15.      return true;
16. }
17.  

Code (CSharp):

1.  
2. //in CameraStreamer.cs
3. public void setResolution( int x, int y ){
4.     streamingSize.x = x;
5.     streamingSize.y = y;
6.     Track = CreateTrack();
7. }
8.  
9. protected override MediaStreamTrack CreateTrack()
10. {
11.     RenderTexture rt;
12.     if (m_camera.targetTexture != null && m_camera.targetTexture.width == streamingSize.x && m_camera.targetTexture.height == streamingSize.y )
13.     {
14.         rt = m_camera.targetTexture;
15.         RenderTextureFormat supportFormat = WebRTC.WebRTC.GetSupportedRenderTextureFormat(SystemInfo.graphicsDeviceType);
16.         GraphicsFormat graphicsFormat = GraphicsFormatUtility.GetGraphicsFormat(supportFormat, RenderTextureReadWrite.Default);
17.         GraphicsFormat compatibleFormat = SystemInfo.GetCompatibleFormat(graphicsFormat, FormatUsage.Render);
18.         GraphicsFormat format = graphicsFormat == compatibleFormat ? graphicsFormat : compatibleFormat;
19.  
20.         if (rt.graphicsFormat != format)
21.         {
22.             Debug.LogWarning(
23.                 $"This color format:{rt.graphicsFormat} not support in unity.webrtc. Change to supported color format:{format}.");
24.             rt.Release();
25.             rt.graphicsFormat = format;
26.             rt.Create();
27.         }
28.  
29.         m_camera.targetTexture = rt;
30.     }
31.     else
32.     {
33.         var format = WebRTC.WebRTC.GetSupportedRenderTextureFormat(SystemInfo.graphicsDeviceType);
34.         rt = new RenderTexture(streamingSize.x, streamingSize.y, depth, format)
35.         {
36.             antiAliasing = antiAliasing
37.         };
38.         rt.Create();
39.  
40.  
41.         RenderTexture temp_target_texture = null;
42.         if(m_camera.targetTexture!=null)
43.             temp_target_texture = m_camera.targetTexture;
44.  
45.         m_camera.targetTexture = rt;
46.  
47.         if(temp_target_texture!=null)
48.             temp_target_texture.Release();
49.     }
50.  
51.     return new VideoStreamTrack(m_camera.name, rt);
52. }
53.  

Code (CSharp):

1. //invoking the code
2.    cameraStreamer.setResolution(X, Y);
3.    renderStreaming.ReplaceAllSenderTracks( cameraStreamer.Track );
4.  

 

Did you manage to figure out the solution @ypvypv

 

No, I have not.

 

Yes, I have tried 3.1.0-exp.2 and the Replace Track still brings down the framerate. What change would fix this? libwebrtc package upgrade?