Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Video how to extract frames from a video

Discussion in 'Audio & Video' started by eomerb, Mar 25, 2020.

  1. eomerb

    eomerb

    Joined:
    Mar 6, 2020
    Posts:
    31
    hello there,
    I am working on a videoplayer
    I can play videos change video.url from code
    for next step I want to show frames from video when mouse is on over the timeline (like youtube)
    I need to get frames for each second second from that video before playing
    how can I do it?
    I look through it and with ffmpeg bind I can extract them but dont want to buy that asset just for that is there any other way?
     
  2. DominiqueLrx

    DominiqueLrx

    Unity Technologies

    Joined:
    Dec 14, 2016
    Posts:
    260
    Hi!

    Unity's VideoPlayer makes this possible, but not easy.
    1. Create a VideoPlayer for the purpose of extracting these frames.
    2. Set its renderMode to APIOnly (see https://docs.unity3d.com/ScriptReference/Video.VideoPlayer-renderMode.html and https://docs.unity3d.com/ScriptReference/Video.VideoRenderMode.APIOnly.html). This is so it doesn't try to display content somewhere in the scene while your background setup is happening.
    3. Set the VideoPlayer url (or VideoClip) that you want to inspect.
    4. Call VideoPlayer.Prepare() to have the source loading bootstrapped.
    5. Enable the frameReady events by setting sendFrameReadyEvents to true (https://docs.unity3d.com/ScriptReference/Video.VideoPlayer-sendFrameReadyEvents.html) and then attaching a handler to the frameReady event (https://docs.unity3d.com/ScriptReference/Video.VideoPlayer-frameReady.html)
    6. When prepare completes (you can either wait for isPrepared to be true, or use the prepareCompleted event), invoke VideoPlayer.Pause(): this will cause the frameReady handler to be invoked when the first frame is ready.
    7. Here, you can copy the pixels from VideoPlayer.texture into a texture of your own for future display. Using https://docs.unity3d.com/ScriptReference/Rendering.AsyncGPUReadback.html is the most efficient way to do this because it's non-blocking. But it forces you to deal with asynchronous execution. There are simpler ways to read the pixels back which I'm sure you can find by googling around, if you prefer a simpler approach.
    8. Then, seek to the next frame you want to capture (1 second later, according to your description) by setting the VideoPlayer.frame (or .time) property and the frameReady event will again be invoked when the seek completes.
    9. Repeat steps 7-8 until you reach the end of the range you wanted to inspect.
    We should have an API that makes it simpler, but for now this is the only possibility.

    Hope this helps!

    Dominique Leroux
    A/V developer at Unity
     
  3. eomerb

    eomerb

    Joined:
    Mar 6, 2020
    Posts:
    31

    for a little summary I need to create another videoPlayer to run in background. pause. get the texture from frameReady handler seek to the next second get the texture from frameReady handler do that until the range I wanted
    step 7 will be a challange for me I tried to read the pixels for another thing hope I can do it
    thanks for the detailed answer
     
  4. eomerb

    eomerb

    Joined:
    Mar 6, 2020
    Posts:
    31
    @DominiqueLrx hello again,
    I am working on your answer but couldnt manage to trigger videoplayer.frameReady without playing the video.


    public VideoPlayer vp;
    public void getFrame(){
    vp.renderMode = VideoRenderMode.APIOnly;
    vp.playOnAwake = false;
    vp.url = selectedSource;
    vp.Prepare();
    vp.sendFrameReadyEvents = true;
    vp.frameReady += Vp_frameReady;
    vp.prepareCompleted += Vp_prepareCompleted;
    }
    private void Vp_prepareCompleted(VideoPlayer source){
    Debug.Log("prepared");
    vp.Pause();
    }
    private void Vp_frameReady(VideoPlayer source, long frameIdx) {
    Debug.Log("ready");
    }

    ı can mute and play at prepareCompleted and pause at frameReady but I dont think its a good way
     
  5. DominiqueLrx

    DominiqueLrx

    Unity Technologies

    Joined:
    Dec 14, 2016
    Posts:
    260
    Hi again!

    Always make sure you add the event handlers before you trigger the action that will cause it to be called, just in case the event is fired directly by the call.

    So I'd attach frameReady and prepareCompleted handlers before calling Prepare().

    Also, make sure the VideoPlayer (vp) is stopped when you do all this setup. Since it's a public member in here, you don't know its initial state so calling Stop() on it would make sure it's not currently playing.

    Let me know if this helps you get a bit further!

    Dominique
     
  6. eomerb

    eomerb

    Joined:
    Mar 6, 2020
    Posts:
    31
    I added videoplayer.Stop() when I am done
    before your answer I write the code as I mantioned I mute the videoPlayer and when it is prepared I played and when frame ready triggers I paused.
    with your anser I call prepare method after I the handlers but its not working.
    frameReady handler not triggering
     
  7. DominiqueLrx

    DominiqueLrx

    Unity Technologies

    Joined:
    Dec 14, 2016
    Posts:
    260
    Hi!

    I just gave this a try locally. Here's something that works for me, let me know if it does what you want!

    I added this script on a game object where a VideoPlayer exists and already has the right source.

    Code (CSharp):
    1. using UnityEngine;
    2. using UnityEngine.Video;
    3.  
    4. public class ExtractFrames : MonoBehaviour
    5. {
    6.     void Start()
    7.     {
    8.         var videoPlayer = GetComponent<VideoPlayer>();
    9.         videoPlayer.Stop();
    10.         videoPlayer.renderMode = VideoRenderMode.APIOnly;
    11.         videoPlayer.prepareCompleted += Prepared;
    12.         videoPlayer.sendFrameReadyEvents = true;
    13.         videoPlayer.frameReady += FrameReady;
    14.         videoPlayer.Prepare();
    15.     }
    16.  
    17.     void Prepared(VideoPlayer vp) => vp.Pause();
    18.  
    19.     void FrameReady(VideoPlayer vp, long frameIndex)
    20.     {
    21.         Debug.Log("FrameReady " + frameIndex);
    22.         var textureToCopy = vp.texture;
    23.         // Perform texture copy here ...
    24.         vp.frame = frameIndex + 30;
    25.     }
    26. }
    27.  
    Dominique
     
    Last edited: Mar 27, 2020
    marck_ozz likes this.
  8. eomerb

    eomerb

    Joined:
    Mar 6, 2020
    Posts:
    31
    @DominiqueLrx hello again
    for my project I have 2 video player components already and they have different roles in my project I didnt reduce it to 1
    I've added your videoplayer somewhere in the canvas and in script I write public Videoplayer vp so that I can match my videoplayer in unity and videoplayer in code
    so I dont use
     var videoPlayer = GetComponent<VideoPlayer>();

    the rest is same.I've also added url from code.
    I write the code you written in start to another function and called it from with a button click and start (no difference) but with pause at prepared event frames ready not triggering. playing video at prepared and pausing in frame ready works for me not gonna dig that further
    thanks
     
  9. Stranger-Games

    Stranger-Games

    Joined:
    May 10, 2014
    Posts:
    393
    Hi @DominiqueLrx

    Thank you. It is exactly what I needed. It works fine on desktop but on iOS prepareCompleted is called but frameReady is never called even though I enabled sendFrameReadyEvents explicitly.

    Here is my code.

    Code (CSharp):
    1.  
    2.             videoPlayer.url = path;
    3.             videoPlayer.Stop();
    4.             videoPlayer.renderMode = VideoRenderMode.APIOnly;
    5.             videoPlayer.prepareCompleted += Prepared;//called
    6.             videoPlayer.errorReceived += ErrorReceived;//not called
    7.             videoPlayer.frameReady += FrameReady;//never called
    8.             videoPlayer.sendFrameReadyEvents = true;
    9.             videoPlayer.Prepare();
    Thanks for advance.
     
  10. Stranger-Games

    Stranger-Games

    Joined:
    May 10, 2014
    Posts:
    393
    Hi @DominiqueLrx

    It seems that if I remove vp.Pause from Prepared, it calls FrameReady but as expected it calls it on rapid succession without waiting for setting vp.frame or a call to StepForward

    Thanks for advance.
     
    Last edited: Jun 26, 2020
  11. eomerb

    eomerb

    Joined:
    Mar 6, 2020
    Posts:
    31
    @Stranger-Games can you send the codes of frameReady and prepared methods
    I had a different problem without vp.play() frameReady wasnt called so I play at prepared and pause at frameReady
    give it a try
     
  12. Stranger-Games

    Stranger-Games

    Joined:
    May 10, 2014
    Posts:
    393
    Thank you very much.

    Thank you for the hint. I switched vp.pause to vp.play in prepared and indeed FrameReady was called but with frame index 1 and not 0.
    In frameready I called videoPlayer.Pause(); and after I finished processing the texture I called videoPlayer.Play(); and indeed frameready was called again but this time with frame index 5, and then 17, etc...

    even after I set videoPlayer.skipOnDrop = false; it still skips significant number of frames.
     
  13. Stranger-Games

    Stranger-Games

    Joined:
    May 10, 2014
    Posts:
    393
    Instead of just calling play, I did this which seems to work

    Code (CSharp):
    1. videoPlayer.Play();
    2. videoPlayer.frame = frameIndex + 1;
    I do still miss the first frame, but isn't a big deal.
    --------------
    Edit:
    Actually I do get the first frame now calling vp.play() on frameReady.
     
    Last edited: Jun 26, 2020
  14. eomerb

    eomerb

    Joined:
    Mar 6, 2020
    Posts:
    31
    I dont know why you are gonna need frames but you are not even missing a second I also used it like that dont think it is a big deal
     
  15. waldgeist

    waldgeist

    Joined:
    May 6, 2017
    Posts:
    386
    Can you please elaborate on how this is done? I understand the callback structure, but I don't understand how to copy the actual data to a Texture2D using AsyncGPUReadbackRequest.GetData().

    request.GetData<Texture2DArrray>(0) does not work. It tells me that the type has to be a non-nullable value type. So what kind of types are allowed? I also don't understand what the "mipIndex" and the "layer" are.

    The Unity docs really lack a lot of examples. If you only know the API, you're pretty much lost.
     
  16. marck_ozz

    marck_ozz

    Joined:
    Nov 30, 2018
    Posts:
    107
    Hello all!

    I was strugling with this also, but in my case I was looking to have a thumbnail of the video while playin the video.

    The solution in here works but once I play the video the thumbnail start to change with every frame ready. So, I combined the solution from @DominiqueLrx and the one from @SrTartarus in here: https://forum.unity.com/threads/video-player-preview.527061/ to get this done.

    Code (CSharp):
    1. public VideoPlayer VideoPlayback;
    2. public RawImage thumbnailVid;
    3. private Texture2D thumbnail;
    4. bool thumbnailOk;
    5.  
    6. public int vidHeight;
    7. public int vidWidth;
    8.  
    9. public void VideoPreparation(string path_)
    10.     {
    11.         VideoPlayback.url = path_;
    12.         VideoPlayback.Stop();
    13.         VideoPlayback.renderMode = VideoRenderMode.APIOnly;
    14.         VideoPlayback.sendFrameReadyEvents = true;
    15.         VideoPlayback.frameReady += FrameReady;
    16.         VideoPlayback.Prepare();
    17.         StartCoroutine(PrepareVideo());
    18.     }
    19.     void FrameReady(VideoPlayer vp, long frameIndex)
    20.     {
    21.         Debug.Log("FrameReady " + frameIndex);
    22.         VideoPlayback.Pause();
    23.         thumbnailVid.texture = vp.texture;
    24.  
    25.         thumbnailVid.texture = Get2DTexture();
    26.  
    27.         VideoPlayback.sendFrameReadyEvents = false; //To stop frameReady events
    28.         vp = null;
    29.  
    30.         thumbnailOk = true;
    31.     }
    32.  
    33.     IEnumerator PrepareVideo()
    34.     {
    35.         yield return new WaitUntil(() => VideoPlayback.isPrepared);
    36.  
    37.         Debug.Log("Video PlayBack");
    38.         VideoPlayback.Play();
    39.        
    40.         vidWidth = Convert.ToInt32(VideoPlayback.width);
    41.         vidHeight = Convert.ToInt32(VideoPlayback.height);
    42.  
    43.         VideoPlayback.isLooping = true;
    44.         VideoPlayback.renderMode = VideoRenderMode.MaterialOverride;
    45.  
    46.         Debug.Log("Video height & width: " + vidWidth + ", " + vidHeight);
    47.  
    48.         yield return new WaitUntil(() => thumbnailOk);
    49.  
    50.         VideoPlayback.Play();
    51.  
    52.         GC.Collect();
    53.     }
    54.    
    55.     private Texture2D Get2DTexture()
    56.     {
    57.         thumbnail = new Texture2D(thumbnailVid.texture.width, thumbnailVid.texture.height, TextureFormat.RGBA32, false);
    58.         RenderTexture cTexture = RenderTexture.active;
    59.         RenderTexture rTexture = new RenderTexture(thumbnailVid.texture.width, thumbnailVid.texture.height, 32);
    60.         UnityEngine.Graphics.Blit(thumbnailVid.texture, rTexture);
    61.  
    62.         RenderTexture.active = rTexture;
    63.         thumbnail.ReadPixels(new Rect(0, 0, rTexture.width, rTexture.height), 0, 0);
    64.         thumbnail.Apply();
    65.  
    66.         UnityEngine.Color[] pixels = thumbnail.GetPixels();
    67.  
    68.         RenderTexture.active = cTexture;
    69.  
    70.         rTexture.Release();
    71.  
    72.         return thumbnail;
    73.     }
    Also, credits of the "texture to Texture2d" code are for "BBIT-SOLUTIONS" from this post: https://answers.unity.com/questions/1271693/reading-pixel-data-from-materialmaintexture-return.html
     
    april_4_short likes this.
  17. threelight

    threelight

    Joined:
    Jan 23, 2019
    Posts:
    5
    Hello all,
    I'm facing a similar problem but with a slightly different context: I have a scene with a plane that shows a video via videoplayer component. I want to be able to move the video plane to a specific position and extract the current frame of the video via button click. The video plane gets cloned and remains in the desired position with the desired frame of the video. I tried the solution from DominiqueLrx but the cloned video plane always shows the last frame of the video. So no matter how often I clone the original video player instance, the cloned video plane always contains the last frame of the video instead of the current one (at the time of my button click). Any ideas what the issue could be? Thanks in advance.
     
  18. marck_ozz

    marck_ozz

    Joined:
    Nov 30, 2018
    Posts:
    107
    Sharing your code could help but I'll gess that you are making the "cloned video plane's texture" equals to "VideoPlayer.Texture" of the original video so that is espected. Try it by making it equals to a 2D texture so it will not change with every "snap shot" of the original video. Check out my solution.

     
  19. threelight

    threelight

    Joined:
    Jan 23, 2019
    Posts:
    5
    @marck_ozz : Thanks for your reply. Here is my code:

    Code (CSharp):
    1. using System;
    2. using System.Collections;
    3. using System.Collections.Generic;
    4. using UnityEngine;
    5. using UnityEngine.Video;
    6.  
    7. public class CloneGameObjects : MonoBehaviour
    8. {
    9.     public GameObject rootObj;
    10.  
    11.     private GameObject duplicate;
    12.     private Renderer rend;
    13.  
    14.     // Start is called before the first frame update
    15.     void Start()
    16.     {
    17.        
    18.     }
    19.  
    20.     public void Clone()
    21.     {
    22.         duplicate = Instantiate(rootObj);
    23.        
    24.         duplicate.transform.localScale = new Vector3(rootObj.transform.lossyScale.x , rootObj.transform.lossyScale.y , rootObj.transform.lossyScale.z );
    25.         duplicate.transform.position = rootObj.transform.position;
    26.         duplicate.transform.rotation = rootObj.transform.rotation;
    27.  
    28.         var videoPlayer = duplicate.GetComponent<VideoPlayer>();
    29.  
    30.         rend = duplicate.GetComponent<Renderer>();
    31.  
    32.         videoPlayer.Stop();
    33.         videoPlayer.renderMode = VideoRenderMode.APIOnly;
    34.         videoPlayer.prepareCompleted += Prepared;
    35.         videoPlayer.sendFrameReadyEvents = true;
    36.         videoPlayer.frameReady += FrameReady;
    37.         videoPlayer.Prepare();
    38.     }
    39.  
    40.     void Prepared(VideoPlayer vp) => vp.Pause();
    41.  
    42.     void FrameReady(VideoPlayer vp, long frameIndex)
    43.     {
    44.        Console.WriteLine("FrameReady " + frameIndex);
    45.         var textureToCopy = vp.texture;
    46.  
    47.         rend.material.mainTexture  = textureToCopy;
    48.  
    49.         // Perform texture copy here ...
    50.         vp.frame = frameIndex + 1;
    51.     }
    52.  
    53.     // Update is called once per frame
    54.     void Update()
    55.     {
    56.        
    57.     }
    58. }
    59.  
     
  20. threelight

    threelight

    Joined:
    Jan 23, 2019
    Posts:
    5
    @marck_ozz I'm trying your solution now but I don't know how to get the RawImage thumbnailVid which you defined as a public variable. So I assume in your case you set this via the inspector in unity?
     
  21. marck_ozz

    marck_ozz

    Joined:
    Nov 30, 2018
    Posts:
    107
    Yes I do.

    In my case, I play the video in a plane and have the video info in a the UI, so, "RawImage thumbnailVid" is the RawImage in my UI that I use to "fill" with the first "Frame Ready texture" of my video as "thumbnail", so you can avoid this component and use "rend.material.mainTexture" instead and then something like:

    rend.material.SetTexture("_BaseMap", Get2DTexture());
    whenever you wish to "capture" the desired frame

    Note that "SetTexture(String name, Texture) depends on the shader you're using in your plane's material, in my case "_BaseMap".
     
  22. threelight

    threelight

    Joined:
    Jan 23, 2019
    Posts:
    5
    Thank you. That's my current code but after I cloned the video plane, the original video player stops playing and all cloned video planes show the same frame (probably the last frame of the whole video).

    Any ideas where the problem could be?

    Code (CSharp):
    1. using System;
    2. using System.Collections;
    3. using System.Collections.Generic;
    4. using UnityEngine;
    5. using UnityEngine.Video;
    6. using UnityEngine.UI;
    7.  
    8. public class CloneGameObjects : MonoBehaviour
    9. {
    10.     public GameObject rootObj;
    11.  
    12.     public int vidHeight;
    13.     public int vidWidth;
    14.     private Texture2D thumbnail;
    15.     bool thumbnailOk;
    16.     private VideoPlayer videoPlayer;
    17.  
    18.     private GameObject duplicate;
    19.     private Renderer rend;
    20.  
    21.     // Start is called before the first frame update
    22.     void Start()
    23.     {
    24.        
    25.     }
    26.  
    27.     public void Clone()
    28.     {
    29.         duplicate = Instantiate(rootObj);
    30.        
    31.         duplicate.transform.localScale = new Vector3(rootObj.transform.lossyScale.x , rootObj.transform.lossyScale.y , rootObj.transform.lossyScale.z );
    32.         duplicate.transform.position = rootObj.transform.position;
    33.         duplicate.transform.rotation = rootObj.transform.rotation;
    34.  
    35.         videoPlayer = duplicate.GetComponent<VideoPlayer>();
    36.  
    37.         rend = duplicate.GetComponent<Renderer>();
    38.  
    39.         videoPlayer.Stop();
    40.         videoPlayer.renderMode = VideoRenderMode.APIOnly;
    41.         //videoPlayer.prepareCompleted += Prepared;
    42.         videoPlayer.sendFrameReadyEvents = true;
    43.         videoPlayer.frameReady += FrameReady;
    44.         videoPlayer.Prepare();
    45.  
    46.         StartCoroutine(PrepareVideo());
    47.     }
    48.  
    49.     void Prepared(VideoPlayer vp) => vp.Pause();
    50.  
    51.     // Update is called once per frame
    52.     void Update()
    53.     {
    54.        
    55.     }
    56.  
    57.     void FrameReady(VideoPlayer vp, long frameIndex)
    58.     {
    59.         Debug.Log("FrameReady " + frameIndex);
    60.         videoPlayer.Pause();
    61.        
    62.         rend.material.SetTexture("Default-Diffuse", Get2DTexture(vp));
    63.  
    64.         vp = null;
    65.        
    66.         thumbnailOk = true;
    67.     }
    68.  
    69.     IEnumerator PrepareVideo()
    70.     {
    71.         yield return new WaitUntil(() => videoPlayer.isPrepared);
    72.  
    73.         Debug.Log("Video PlayBack");
    74.         videoPlayer.Play();
    75.  
    76.         vidWidth = Convert.ToInt32(videoPlayer.width);
    77.         vidHeight = Convert.ToInt32(videoPlayer.height);
    78.  
    79.         videoPlayer.isLooping = true;
    80.         videoPlayer.renderMode = VideoRenderMode.MaterialOverride;
    81.  
    82.         Debug.Log("Video height & width: " + vidWidth + ", " + vidHeight);
    83.  
    84.         yield return new WaitUntil(() => thumbnailOk);
    85.  
    86.         videoPlayer.Play();
    87.  
    88.         GC.Collect();
    89.     }
    90.  
    91.     private Texture2D Get2DTexture(VideoPlayer vp)
    92.     {
    93.         thumbnail = new Texture2D(vp.texture.width, vp.texture.height, TextureFormat.RGBA32, false);
    94.         RenderTexture cTexture = RenderTexture.active;
    95.         RenderTexture rTexture = new RenderTexture(vp.texture.width, vp.texture.height, 32);
    96.         UnityEngine.Graphics.Blit(vp.texture, rTexture);
    97.  
    98.         RenderTexture.active = rTexture;
    99.         thumbnail.ReadPixels(new Rect(0, 0, rTexture.width, rTexture.height), 0, 0);
    100.         thumbnail.Apply();
    101.  
    102.         //UnityEngine.Color[] pixels = thumbnail.GetPixels();
    103.  
    104.         RenderTexture.active = cTexture;
    105.  
    106.         rTexture.Release();
    107.  
    108.         return thumbnail;
    109.     }
    110. }
    111.  
     
  23. cficara

    cficara

    Joined:
    Sep 27, 2017
    Posts:
    35
    Have you tried unsuscribing from the event FrameReady ?

    EDIT: Well, yeah, I'm having the same issue.



    This is because the thumbnail generation function isn't waiting for each frame to be ready, for each video, so it gets whatever is set at the texture. I will update later with the solution.

    EDIT 2: That was the issue. Here's the correct generation:



    I use FrameReady() function to wait for the texture to be ready. I added at the end of that function:
    Code (CSharp):
    1.  VideoPlayback.frameReady -= FrameReady;//unsuscribe
    so these events won't pile up unnecessarily. Also, at the end I added a custom function that sets the generated thumbnail to a button raw image, and in that same function I call VideoPrepare() for the next button, and it starts all over again.

    Thumbnail generation is not super slow, but is not really quick. I will need to save these thumbnails to disk, so they are used the next time. BUT another story for another time I guess. Thanks everyone who contributed to this!
     
    Last edited: Aug 19, 2021
  24. threelight

    threelight

    Joined:
    Jan 23, 2019
    Posts:
    5
    Thanks for your reply! Unsubscribing from the FrameReady event helps! However, no matter when I create a new 2D plane with the copied texture, the new plane always contains the same frame from from the video. Any ideas what could be the reason for this?