Search Unity

  1. Get the latest news, tutorials and offers directly to your inbox with our newsletters. Sign up now.
    Dismiss Notice

Help Wanted Using ESRGAN for superresolute Video

Discussion in 'Barracuda' started by XPhilipX, Dec 7, 2020.

  1. XPhilipX

    XPhilipX

    Joined:
    Jun 16, 2020
    Posts:
    1
    Hi,

    I'm currently trying to use the ESRGAN, which I have converted to ONNX and works completely fine.
    Now I'm trying to do the same with a video having the same resolution. The measured timings with around 40ms for each frame are okay, but its stuttering hardly. The console shows me that five frames are rendered
    than it stutters and continues 100 frames later.

    My example code looks like this:
    Code (CSharp):
    1. public class MLVideo : MonoBehaviour
    2. {
    3.     public MeshRenderer Renderer;
    4.     public NNModel modelAsset;
    5.     private Model m_RuntimeModel;
    6.     private IWorker Worker;
    7.  
    8.     Tensor InputTensor;
    9.     Tensor OutputTensor;
    10.     RenderTexture OutputTexture;
    11.  
    12.     Queue<RenderTexture> ImageQueue = new Queue<RenderTexture>();
    13.     void Start()
    14.     {
    15.         m_RuntimeModel = ModelLoader.Load(modelAsset);
    16.         Worker = WorkerFactory.CreateWorker(WorkerFactory.Type.ComputePrecompiled, m_RuntimeModel);
    17.         OutputTexture = new RenderTexture(2048, 1024, 3);
    18.  
    19.         var videoPlayer = GetComponent<VideoPlayer>();
    20.         videoPlayer.renderMode = VideoRenderMode.APIOnly;
    21.         videoPlayer.sendFrameReadyEvents = true;
    22.         videoPlayer.frameReady += FrameReady;
    23.         videoPlayer.Prepare();
    24.         videoPlayer.Play();  
    25.     }
    26.  
    27.     private void Update()
    28.     {
    29.         if (ImageQueue.Count > 0)
    30.         {
    31.             OutputTexture = ImageQueue.Dequeue();
    32.             Renderer.material.mainTexture = OutputTexture;
    33.         }
    34.     }
    35.  
    36.  
    37.     void FrameReady(VideoPlayer vp, long frameIndex)
    38.     {
    39.         Stopwatch stopwatch = new Stopwatch();
    40.  
    41.         stopwatch.Start();
    42.  
    43.         InputTensor = new Tensor(vp.texture, 3);
    44.  
    45.         Worker.Execute(InputTensor);
    46.  
    47.         OutputTensor = Worker.PeekOutput();
    48.         var texture = new RenderTexture(2048, 1024, 3);
    49.         OutputTensor.ToRenderTexture(texture);
    50.         ImageQueue.Enqueue(texture);
    51.  
    52.         stopwatch.Stop();
    53.  
    54.         Debug.Log(string.Format("FrameReady {0}: Elapsed Time is {1} ms", frameIndex, stopwatch.ElapsedMilliseconds)); // around 40ms
    55.  
    56.         InputTensor.Dispose();
    57.         OutputTensor.Dispose();
    58.         texture.DiscardContents();
    59.     }
    60.  
    61.     public void OnDestroy()
    62.     {
    63.         Worker?.Dispose();
    64.         InputTensor?.Dispose();
    65.         OutputTensor?.Dispose();
    66.     }
    67. }
    68.  
    Do you have any advice on how to improve the performance?
    If it matters I'm using a 2080Ti.
    Thanks in advance.
     
  2. fguinier

    fguinier

    Unity Technologies

    Joined:
    Sep 14, 2015
    Posts:
    121
    Have you tried ComputeInfo.channelsOrder = ComputeInfo.ChannelsOrder.NCHW; it is in experimental state for now however can dramatically boost performance on GPU in some case.
     
  3. amirebrahimi_unity

    amirebrahimi_unity

    Unity Technologies

    Joined:
    Aug 12, 2015
    Posts:
    378
    Also, another thought @XPhilipX - can you try feeding it images from a different source to isolate whether it is the video playback or Barracuda?
     
unityunity