Search Unity

Question how to know that videoPlayer has rendered the first frame?

Discussion in 'Audio & Video' started by taowenbo1993, Jul 31, 2022.

  1. taowenbo1993

    taowenbo1993

    Joined:
    Feb 12, 2022
    Posts:
    12
    version: 2021.3.0f1 Apple Silicon

    I'm using AVProMovieCapture to record a VideoPlayer's playback. However, I can't figure out when the first frame has been rendered so that I can start the video capture.

    What I did was to include the following in the Update() method:
    Code (CSharp):
    1.  
    2. if (!hasStartedPlaying)
    3. {
    4.     videoPlayer.Play();
    5.     hasStartedPlaying = true;
    6. }
    7. if (videoPlayer.isPlaying) // this doesn't work, and will include several empty frames, forming a flash of the sky background.
    8. {
    9.      videoCapture.startCapture();
    10. }
    Apparently videoPlayer.isPlaying == true doesn't mean the first frame has arrived nor been rendered. The same goes for videoPlayer.frame == 0.

    Weirdly, if I keep printing videoPlayer.frame, it will be 0 for a long time, and the next value would be 2, after which it becomes normal (i.e. 3, 4, 5, ...). Why is that?

    In general, it's very hard to record a scene from the exact beginning a VideoPlayer. Please advice, thank you!
     
  2. The_Island

    The_Island

    Unity Technologies

    Joined:
    Jun 1, 2021
    Posts:
    502
    Yes, you are right. We change the frame when we send the frame to the GPU, but in the end, the GPU is the one deciding when blitting.
    Yeah, I agree with you. In the past, I tried doing the same things for debugging purposes and stopped before finding a good solution. The issue is a RenderTexture is on the GPU, not the CPU otherwise, you could check Texture.updateCount, but it doesn't work for RenderTexture. For that reason, we landed on 2022.2, a feature to synchronize the VideoPlayer with the Time.captureDeltaTime. If you are interested, beware it is freshly out of the oven, so expect bugs.
    It could be multiple things, but generally, it is because your video doesn't start at 0. You should have received a warning when importing the video telling you that. And if it is the case, frame 2 is, in reality, frame 1. They just have an offset at the start.
     
    Last edited: Aug 1, 2022
  3. taowenbo1993

    taowenbo1993

    Joined:
    Feb 12, 2022
    Posts:
    12
    Thank you for your reply @The_Island !
    I wonder how to use this new feature in code terms? Would something like this work? https://docs.unity3d.com/ScriptReference/Time-captureDeltaTime.html
    But even with that I'm not sure when I should start the capture... since the first frame would require time to prepare.
    I don't recall seeing the error when importing the video... And I've tried starting capture if videoPlayer.frame == 2, and the end result would be like 1-2 missing frames...
     
  4. The_Island

    The_Island

    Unity Technologies

    Joined:
    Jun 1, 2021
    Posts:
    502
    Rereading your question, I should have asked for some clarification. Tell me if I am wrong, but from what I understand, you want to record your game to make a trailer or gameplay video?

    Time.captureDeltaTime allows you to provide a target framerate. When Time.captureDeltaTime is > 0, the editor will wait whatever time it needs until all work is done. If a frame takes 1 second to render, it will wait 1 second before incrementing the time by Time.captureDeltaTime. So if you call play, even if the frame is not ready, it will wait for the frame to finish decoding and rendering before continuing to the next frame. We made this precisely for doing high-quality recordings for the Recorder package.

    However, this method doesn't work if you want to record a real player playing a game. If it is what you are trying to do then it is another issue.
    You can right-click and reimport if you want to see the warning. Btw starting a recorder is generally not instant. Are you sure the recorder is not preparing during the 1-2 frames? I don't know about AVProMovieCapture but I would also check that. Because if you start the recorder at videoPlayer.frame == 2, I don't understand how could they be 1-2 missing frames. At worst the recorder should record too many frames not miss some.
     
  5. taowenbo1993

    taowenbo1993

    Joined:
    Feb 12, 2022
    Posts:
    12
    No gameplay involved - just recording an animation/trailer, i.e., some game objects on top of a video.
    If so, does that mean I can use the exact code here for my purpose? I.e. I can switch to 2022.2, set VideoPlayer to be play on awake & wait for first frame, and then I don't need to have an IF condition in Update() to check when the first frame becomes available?
     
  6. The_Island

    The_Island

    Unity Technologies

    Joined:
    Jun 1, 2021
    Posts:
    502
    Yes this should work. Btw if it is too painful to upgrade, I often saw people using this workaround to make it work. With the same principle, when Time.captureDeltaTime > 0, every update needs to be completely done before going forward. So if you seek using
    videoPlayer.frame++
    and call play just after, you are forcing it to render this frame. At least, this is what I read. It is not my code, but I found another discussion internally about this workaround.
    Code (CSharp):
    1. using UnityEngine;
    2. using UnityEngine.Video;
    3.  
    4. public class VideoStepper : MonoBehaviour
    5. {
    6.     private VideoPlayer videoPlayer; // sibling video player
    7.     private bool ready = false;
    8.  
    9.     private void Start() {
    10.         videoPlayer = GetComponent<VideoPlayer>();
    11.         videoPlayer.playOnAwake = false;
    12.         videoPlayer.playbackSpeed = 0f;
    13.         videoPlayer.isLooping = true;
    14.         videoPlayer.prepareCompleted += OnVideoPrepared;
    15.         videoPlayer.Prepare();
    16.     }
    17.  
    18.     private void OnVideoPrepared(VideoPlayer videoPlayer) {
    19.         videoPlayer.Play();
    20.         ready = true;
    21.         Debug.Log("Video done prepared.");
    22.     }
    23.  
    24.     private void Update() {
    25.         if (ready)
    26.         {
    27.             videoPlayer.frame++;
    28.             videoPlayer.Play();
    29.         }
    30.     }
    31. }
    32.  
    If you want to try the new feature, I think it is still in beta. You will know you have the right version if you see the Update mode on the VideoPlayer. Don't forget to check that it is in Game Time. Otherwise, it will not work.

    upload_2022-8-2_0-20-59.png

    I hope it helps!
     
    Last edited: Aug 2, 2022
  7. taowenbo1993

    taowenbo1993

    Joined:
    Feb 12, 2022
    Posts:
    12
    Thank you! I'll try and report back.
     
  8. taowenbo1993

    taowenbo1993

    Joined:
    Feb 12, 2022
    Posts:
    12
    I tried the workaround - it didn't work.
    VideoPlayer.frame++; VideoPlayer.Play() won't render exactly one frame at each step when captureDeltaTime > 0.

    However, the videoplayer + captureDeltaTime approach in 2022.2 Beta worked like a charm!

    Thanks again @The_Island
     
    The_Island likes this.