Question Using ESRGAN for superresolute Video

Hi,

I'm currently trying to use the ESRGAN, which I have converted to ONNX and works completely fine.
Now I'm trying to do the same with a video having the same resolution. The measured timings with around 40ms for each frame are okay, but its stuttering hardly. The console shows me that five frames are rendered
than it stutters and continues 100 frames later.

My example code looks like this:

Code (CSharp):

1. public class MLVideo : MonoBehaviour
2. {
3.     public MeshRenderer Renderer;
4.     public NNModel modelAsset;
5.     private Model m_RuntimeModel;
6.     private IWorker Worker;
7.  
8.     Tensor InputTensor;
9.     Tensor OutputTensor;
10.     RenderTexture OutputTexture;
11.  
12.     Queue<RenderTexture> ImageQueue = new Queue<RenderTexture>();
13.     void Start()
14.     {
15.         m_RuntimeModel = ModelLoader.Load(modelAsset);
16.         Worker = WorkerFactory.CreateWorker(WorkerFactory.Type.ComputePrecompiled, m_RuntimeModel);
17.         OutputTexture = new RenderTexture(2048, 1024, 3);
18.  
19.         var videoPlayer = GetComponent<VideoPlayer>();
20.         videoPlayer.renderMode = VideoRenderMode.APIOnly;
21.         videoPlayer.sendFrameReadyEvents = true;
22.         videoPlayer.frameReady += FrameReady;
23.         videoPlayer.Prepare();
24.         videoPlayer.Play();  
25.     }
26.  
27.     private void Update()
28.     {
29.         if (ImageQueue.Count > 0)
30.         {
31.             OutputTexture = ImageQueue.Dequeue();
32.             Renderer.material.mainTexture = OutputTexture;
33.         }
34.     }
35.  
36.  
37.     void FrameReady(VideoPlayer vp, long frameIndex)
38.     {
39.         Stopwatch stopwatch = new Stopwatch();
40.  
41.         stopwatch.Start();
42.  
43.         InputTensor = new Tensor(vp.texture, 3);
44.  
45.         Worker.Execute(InputTensor);
46.  
47.         OutputTensor = Worker.PeekOutput();
48.         var texture = new RenderTexture(2048, 1024, 3);
49.         OutputTensor.ToRenderTexture(texture);
50.         ImageQueue.Enqueue(texture);
51.  
52.         stopwatch.Stop();
53.  
54.         Debug.Log(string.Format("FrameReady {0}: Elapsed Time is {1} ms", frameIndex, stopwatch.ElapsedMilliseconds)); // around 40ms
55.  
56.         InputTensor.Dispose();
57.         OutputTensor.Dispose();
58.         texture.DiscardContents();
59.     }
60.  
61.     public void OnDestroy()
62.     {
63.         Worker?.Dispose();
64.         InputTensor?.Dispose();
65.         OutputTensor?.Dispose();
66.     }
67. }
68.  

Do you have any advice on how to improve the performance?
If it matters I'm using a 2080Ti.
Thanks in advance.

 

Also, another thought @XPhilipX - can you try feeding it images from a different source to isolate whether it is the video playback or Barracuda?