Search Unity

  1. Welcome to the Unity Forums! Please take the time to read our Code of Conduct to familiarize yourself with the forum rules and how to post constructively.
  2. Unite 2023 Registrations are now LIVE! We are thrilled to announce that Unite 2023 Registration is now live and open for all!
    Dismiss Notice
New Forum User Notice Unite 2023 Registrations are now LIVE!

Video Video playback issue: setting VideoPlayer.time not working after update from 2020.1.16 to 2020.3.25

Discussion in 'Audio & Video' started by connor_carson, Jan 18, 2022.

  1. connor_carson

    connor_carson

    Joined:
    Jun 3, 2020
    Posts:
    5
    Currently video playback is handled this way:
    - the video is set via a url and prepared
    - when finished preparing we call VideoPlayer.Play()
    - when the frame is ready and the video is playing we call VideoPlayer.Pause()
    - then from fixed update we're calling SetTargetTime(), which updates the Video Player via VideoPlayer.time

    We're playing the video this way to be able to smoothly scrub in either direction and control playback with more precision.

    Originally in 2020.1.16 this was working like a charm, but since updating to 2020.3.25 the video playback is choppy and lagging. Upon further inspection it seems that VideoPlayer.time is not actually being updated when set, and only updates its value after several frames.

    I also noticed that the VideoPlayer values (e.g. frameRate, length, width, height) which used to be populated after VideoPlayer.prepareCompleted are all zero after prepareCompleted. I found a workaround by using a callback for onSeekCompleted, and this has worked for getting a reference to the correct video values, but I wasn't sure if this was at all related to the laggy playback.

    Any help is much appreciated. Code is below:

    Code (CSharp):
    1. using System;
    2. using UnityEngine;
    3. using UnityEngine.UI;
    4. using UnityEngine.Video;
    5. using TMPro;
    6.  
    7. public class VideoPlayback_Unity : MonoBehaviour, IVideoPlaybackSource
    8. {
    9.     private Action<IVideoPlaybackSource, VideoPlaybackState> _stateChangeCallback;
    10.     private VideoPlayer _player;
    11.     private Vector2 _size;
    12.     private double _time;
    13.     private double _startTime;
    14.     private double _frameTime;
    15.     private double _frameRate;
    16.     private float _length;
    17.     private float _masterVolume;
    18.     private long _frameCount;
    19.     private int _frame;
    20.     private bool _started;
    21.  
    22.     private const int MinFramesBeforeFinish = 20;
    23.  
    24.     public bool FrameReady { get; private set; }
    25.     public Texture Texture { get { return _player.texture; } }
    26.     public string Id { get { return _player != null ? System.IO.Path.GetFileNameWithoutExtension( _player.url ) : "None Loaded"; } }
    27.  
    28.     public void Create(Action<IVideoPlaybackSource, VideoPlaybackState> stateChangeCallback)
    29.     {
    30.         _stateChangeCallback = stateChangeCallback;
    31.  
    32.         _player = gameObject.AddComponent<VideoPlayer>();
    33.  
    34.         _player.isLooping = false;
    35.         _player.playOnAwake = false;
    36.         _player.sendFrameReadyEvents = true;
    37.  
    38.         _player.audioOutputMode = VideoAudioOutputMode.None;
    39.         _player.controlledAudioTrackCount = 0;
    40.  
    41.         _player.prepareCompleted += OnPlayerPrepareCompleted;
    42.         _player.frameReady += OnFrameReady;
    43.         _player.started += OnPlayerStarted;
    44.         _player.loopPointReached += OnPlayerLoopPointReached;
    45.         _player.errorReceived += OnPlayerErrorReceived;
    46.     }
    47.  
    48.     public void SetTargetTime(double time)
    49.     {
    50.         time = Math.Min(Math.Max(time, 0.0), _length);
    51.         _player.time = _time = time;
    52.         _frame = Mathf.Clamp((int)Math.Round((_frameRate * time) + 1), 1, (int)_frameCount);
    53.     }  
    54.  
    55.     public bool CheckForFinish()
    56.     {
    57.         bool finished = false;
    58.  
    59.         if (_started)
    60.         {
    61.             finished = _player.frame <= MinFramesBeforeFinish ||
    62.                 Mathf.Abs((long)_player.frameCount - _player.frame) <= MinFramesBeforeFinish;
    63.         }
    64.  
    65.         return finished;
    66.     }
    67.  
    68.     public void Load(string filename, double time = 0.0, float duration = 0.0f)
    69.     {
    70.         Debug.LogFormat("Looking for clip: {0}", filename);
    71.  
    72.         _startTime = time;
    73.         _length = duration;
    74.  
    75.         _player.source = VideoSource.Url;
    76.         _player.url = filename + ".mp4";
    77.  
    78.         _player.audioOutputMode = VideoAudioOutputMode.None;
    79.         _player.controlledAudioTrackCount = 0;
    80.  
    81.         _player.sendFrameReadyEvents = true;
    82.         FrameReady = false;
    83.         _started = false;
    84.  
    85.         _player.seekCompleted += OnSeekCompleted;
    86.         _player.Prepare();
    87.     }
    88.  
    89.     public Vector2 GetSize()
    90.     {
    91.         return _size;
    92.     }
    93.  
    94.     public int GetFrame()
    95.     {
    96.         return _frame;
    97.     }
    98.  
    99.     public double GetFrameTime()
    100.     {
    101.         return (_frame - 1) / _frameRate;
    102.     }
    103.  
    104.     public long GetTotalFrames()
    105.     {
    106.         return _frameCount;
    107.     }
    108.  
    109.     public bool IsPrepared()
    110.     {
    111.         return _player.isPrepared;
    112.     }
    113.  
    114.     public bool IsPlaying()
    115.     {
    116.         return _player.isPlaying;
    117.     }
    118.  
    119.     public void Play()
    120.     {
    121.         _player.Play();
    122.     }
    123.  
    124.     public void Pause()
    125.     {
    126.         _player.Pause();
    127.     }
    128.  
    129.     public void Stop()
    130.     {
    131.         _player.Stop();
    132.     }
    133.  
    134.     public void SetVolume(float volume)
    135.     {
    136.         _masterVolume = volume;
    137.         _player.SetDirectAudioVolume(0, _masterVolume);
    138.     }
    139.  
    140.     private void OnPlayerPrepareCompleted(VideoPlayer source)
    141.     {
    142.         _player.time = _time = _startTime;
    143.     }
    144.  
    145.     private void OnSeekCompleted(VideoPlayer source)
    146.     {
    147.         if (_length == 0.0f)
    148.         {
    149.             _length = (float)_player.length;
    150.         }
    151.  
    152.         _frameRate = _player.frameRate;
    153.         _frameCount = (long)Math.Round(_length * _frameRate);
    154.         _size = new Vector2(_player.texture.width, _player.texture.height);
    155.  
    156.         if (_player.pixelAspectRatioNumerator != 1.0f || _player.pixelAspectRatioDenominator != 1.0f)
    157.         {
    158.             Debug.LogWarningFormat("Warning: Loaded Video {0} does not use square pixels ({1}:{2})", _player.url, _player.pixelAspectRatioNumerator, _player.pixelAspectRatioDenominator);
    159.         }
    160.  
    161.         _stateChangeCallback?.Invoke(this, VideoPlaybackState.Prepared);
    162.  
    163.         source.seekCompleted -= OnSeekCompleted;
    164.     }
    165.  
    166.     private void OnPlayerStarted(VideoPlayer source)
    167.     {
    168.         _stateChangeCallback?.Invoke(this, VideoPlaybackState.Playing);
    169.  
    170.         _started = true;
    171.     }
    172.  
    173.     private void OnPlayerLoopPointReached(VideoPlayer source)
    174.     {
    175.         _stateChangeCallback?.Invoke(this, VideoPlaybackState.Finished);
    176.     }
    177.  
    178.     private void OnPlayerErrorReceived(VideoPlayer source, string message)
    179.     {
    180.         _stateChangeCallback?.Invoke(this, VideoPlaybackState.Error);
    181.     }
    182.  
    183.     private void OnFrameReady(VideoPlayer source, long frameIndex)
    184.     {
    185.         FrameReady = true;
    186.         _player.sendFrameReadyEvents = false;
    187.     }
    188. }
    189.  
     
    connor_carson, Jan 18, 2022
    #1

  2. The_Island

    The_Island

    Unity Technologies

    Joined:
    Jun 1, 2021
    Posts:
    502
    Are you able to record what you mean by "choppy and lagging"?
     
    The_Island, Jan 19, 2022
    #2

  3. connor_carson

    connor_carson

    Joined:
    Jun 3, 2020
    Posts:
    5
    Sure thing! Attached are two example videos. In the first I'm not debugging at all, in the second I'm debugging the requested time I'm setting player.time to and player.time after it's been set. You can see that the time will not update for several frames in the debug.



     
    Last edited: Jan 20, 2022
    connor_carson, Jan 20, 2022
    #3

  4. DominiqueLrx

    DominiqueLrx

    Unity Technologies

    Joined:
    Dec 14, 2016
    Posts:
    256
    Hi Connor!

    Thanks for reporting this problem and providing this description.

    As mentioned earlier via e-mail, there has been a bugfix that has introduced a regression, which has subsequently been repaired and what you are describing here may be a manifestation of this regression. Essentially, the buggy situation is that a lot of the VideoPlayer data becomes zero (e.g.: frameRate, length, width, height and so on) while a seek operation is ongoing.

    This is the original bugfix, and the regression that it introduced is repaired here.

    As you can see by inspecting the two issues, the first bugfix has already been backported to 2019.4, but the repair for the regression has not yet landed in 2019.4.

    One thing you could do is migrate this subpart of your project to the latest Unity 2022.1 beta and see if the misbehaviour is gone. Of course, I'm not suggesting that you start using 2022.1 in production, but this will let you know in advance if the fix I'm talking about is indeed solving the issue.

    It's still possible that what we're looking at in your scenario is something else, but it's hard to say without looking at the whole logic that you use to drive the VideoPlayer. For example, when you are saying "from fixed update we're calling SetTargetTime(), which updates the Video Player via VideoPlayer.time", this may be problematic. If you're constantly setting the VideoPlayer time while it's playing back, you may be causing constant flushing of its buffering, causing noticeable performance issues like we are seeing in your videos. Doing what you are mentioning when scrubbing (and only when scrubbing) does make sense, but should not be done while playback is happening. So while the code you provided does let us know a bit how you're using the VideoPlayer, we don't have visibility of how (and how frequently) SetTargetTime is invoked.

    So at this point, you can create a small test scene that can be used in 2022.1 for experimentation and see if the problem is still there. If it's gone, then it means the pending backport for the bugfix will address the issue (and we can try to accelerate this process). If it still is a problem in 2022.1, then you should file a new bug and report the bug id to this thread so that we can immediately start working with our QAs to reproduce the problem and make sure it lands in our backlog as quickly as possible.

    Another thing I've spotted in your code that may be useful in the future (but probably doesn't participate in the current issue) is what happens in OnPlayerPrepareCompleted. Triggering a seek operation immediately after preparation completes is a bit wasteful, since the buffering that's been performed before invoking the callback will have been in vain (since you're moving the play head elsewhere right away). You can set _player.time before calling Prepare(), in which case your callback will be invoked when preload has happened for the intended start time. It's not catastrophic and only impacts how quickly you get to see the initial frames, but I thought you should know.

    Finally, in the code you are sharing above, I see that you are reading data (length, frame rate, width, height, pixel aspect ratio) from OnSeekCompleted. This data will not change over time (unless you change the source), so I suggest reading it just once, in OnPlayerPrepareCompleted (before triggering a seek operation, should you choose to keep doing this). There is no harm in constantly reading this data - other than doing unneeded repetitive work - but it makes you more exposed to the (temporary) issues that I've pointed out.

    Hope this is helpful and that we soon get you back to the smooth scrubbing performance you had in 2020.1.16.

    Dominique Leroux
    A/V developer at Unity
     
    DominiqueLrx, Jan 21, 2022
    #4

  5. connor_carson

    connor_carson

    Joined:
    Jun 3, 2020
    Posts:
    5

    Hi Dominique! Thanks so much for your reply and input!

    I took your advice and migrated the project to 2022.1 Beta and the playback was perfect. Glad to have some confirmation of the underlying issue! Any idea how quickly the repair for the regression issue might hit 2020.3?

    To address some of your other concerns, we're never actually setting the time while the video is in playback, so hopefully no issues there down the line *fingers crossed*. Regarding the OnSeekComplete() call, I actually remove the callback at the bottom of the method itself so that it only is called once after initially loading the game and doing the first seek. :)

    line 63 in the above code snippet:
    source.seekCompleted -= OnSeekCompleted;


    This was just my workaround for getting the values that were zeroing out after the video finished preparing. For some reason, the values only populated for me after calling that initial seek. But that code will be entirely removed once the regression is fixed, as it was a added as a result.

    Thanks again!
     
    connor_carson, Jan 24, 2022
    #5