Search Unity

  1. Unity 6 Preview is now available. To find out what's new, have a look at our Unity 6 Preview blog post.
    Dismiss Notice
  2. Unity is excited to announce that we will be collaborating with TheXPlace for a summer game jam from June 13 - June 19. Learn more.
    Dismiss Notice

Resolved VideoPlayer can't play at correct & expected video frame rate for a given video clip?

Discussion in 'Audio & Video' started by mikejm_, Feb 10, 2023.

  1. mikejm_

    mikejm_

    Joined:
    Oct 9, 2021
    Posts:
    346
    I am attempting to create transparent background videos, which works partially if one encodes to VP8. I have made videos encoded using FFMPEG from PNG sequences. When I open them in Media Player Classic it correctly identifies their format as:

    Code (csharp):
    1. Video: VP80 720x1520 60fps [V: vp8, yuv420p, 720x1520]
    To get the transparent function in Unity, I set the clip properties to "Transcode" with "Keep Alpha". This allows the transparency to come through both in Unity Editor and Android builds.

    However, the frame rate is not played back correctly when I do this. If I debug as the video is playing:

    Code (csharp):
    1. Debug.Log("FRAME RATE " + videoPlayer.frameRate + " delta time rate " + (1f/Time.deltaTime));
    I get:

    Code (csharp):
    1. FRAME RATE 24.97496 delta time rate 60.07016
    If I set the Codec in Unity on the video clip asset to "VP8" instead of "Auto" and then play back, it gives the correct video playback frame rate (60) in Editor. However, the video clip then looks visually mangled in Android with lines all over it and wrong colors.

    I need transparent videos working in Windows/Android/iPhone.

    Is there something I'm missing? How is this supposed to be possible or is this a bug I should report formally? Thanks.
     
  2. The_Island

    The_Island

    Unity Technologies

    Joined:
    Jun 1, 2021
    Posts:
    502
    So that I understand your issue correctly, you made a clip with FFmpeg, and the clip framerate is 60fps as it should be. Then when you transcode your clip in VP8 using Unity, the framerate changes to 24.97496. You tried to change it to Auto, and the framerate was right, but you get visually "mangled" video on Android. Am I following you?

    1. When you import the clip without transcoding, what is the framerate? You can see it in the source info. I want to make sure we correctly handle the file before transcoding.

    upload_2023-2-10_16-1-49.png

    2. You know that if you already have alpha in your webm, you don't need to transcode your clip except for Android to keep alpha, right? So do you transcode on all platforms or just Android?

    3. Can you share a screenshot to understand how mangled the output is? And can you confirm that it is only when transcoding in Auto, not VP8?
     
    Last edited: Feb 10, 2023
  3. mikejm_

    mikejm_

    Joined:
    Oct 9, 2021
    Posts:
    346
    Thanks, The_Island. I have made a bug report. If it is easier, perhaps you can take a look at that and let me know what you see from it.

    For this test, I made a simple video using 29 frames of PNGs using FFMPEG to create a spinning glowing transparent cube:

    Code (csharp):
    1. ffmpeg -start_number 1 -i c:/tmp/%04d.png -r 60 -c:v libvpx -crf 10 -b:v 2M -pix_fmt yuva420p -auto-alt-ref 0 -metadata:s:v:0 alpha_mode="1" c:/tmp/output.webm
    Just for reference, I even included FFMPEG.exe and the source PNGs in the test project:
    videoissue-cubes.PNG


    The output video appears as follows inside Unity, correctly recognized as 60 fps as encoded:
    videoissue-clip.PNG

    Then I wrote the following project:
    Code (csharp):
    1.  
    2. public class App : MonoBehaviour
    3. {
    4.     VisualElement bgVE;
    5.     Label label;
    6.     VideoPlayer videoPlayer;
    7.  
    8.     // Start is called before the first frame update
    9.     void Start()
    10.     {
    11.         Application.targetFrameRate = (int)Screen.currentResolution.refreshRateRatio.value;
    12.         VisualElement rootVE = gameObject.GetComponent<UIDocument>().rootVisualElement;
    13.  
    14.         bgVE = new VisualElement();
    15.         bgVE.style.width = bgVE.style.height = new Length(100, LengthUnit.Percent);
    16.         bgVE.style.backgroundColor = Color.white;
    17.         rootVE.Add(bgVE);
    18.  
    19.         label = new Label();
    20.         label.style.color = Color.green;
    21.         label.style.fontSize = 20;
    22.         label.style.position = Position.Absolute;
    23.         bgVE.Add(label);
    24.  
    25.         VideoClip clip = Resources.Load<VideoClip>("output");
    26.  
    27.         RenderTexture rt = new RenderTexture((int)clip.width, (int)clip.height, 32);
    28.  
    29.         videoPlayer = gameObject.AddComponent<VideoPlayer>();
    30.         videoPlayer.clip = clip;
    31.         videoPlayer.targetTexture = rt;
    32.         videoPlayer.isLooping = true;
    33.         videoPlayer.Play();
    34.  
    35.         VisualElement videoVE = new VisualElement();
    36.         videoVE.style.width = videoVE.style.height = new Length(100, LengthUnit.Percent);
    37.         videoVE.style.backgroundImage = Background.FromRenderTexture(rt);
    38.         videoVE.style.unityBackgroundScaleMode = ScaleMode.ScaleToFit;
    39.         bgVE.Add(videoVE);
    40.     }
    41.  
    42.     // Update is called once per frame
    43.     void Update()
    44.     {
    45.         string debug = "";
    46.         if (videoPlayer != null) {
    47.             debug += "VIDEO FPS: " + videoPlayer.frameRate + " \nSCREEN FPS: " + (1f / Time.deltaTime);
    48.         }
    49.         label.text = debug;
    50.     }
    51. }
    This creates the following screen display with a spinning cube and frame rate outputs:

    videoissue-project.PNG


    I then ran the following test settings under "Default" with the following results:

    SETTING 1: TRANSCODE, AUTO OR H264 CODEC, KEEP ALPHA:

    Android Result:
    SCREEN FPS 60, VIDEO FPS 30 (terrible jerky motion, does not run in any remotely normal way)
    Video looks mostly correct otherwise, transparency permitted but with sharp alpha clipping - no glow effect evident.

    Windows Editor Result:
    SCREEN FPS 60, VIDEO FPS 25.58075 (motion is smooth and normal)
    Video looks correct except again transparency is permitted only with sharp alpha clipping - no glow effect evident.

    SETTING 2: NO TRANSCODE:

    Android Result:
    SCREEN FPS 60, VIDEO FPS 30 (terrible jerky motion, does not run in any remotely normal way)
    Video has green lines all over, has black background instead of alpha, glow is evident against black background, generally looks ridiculous

    Windows Editor Result:
    SCREEN FPS 60, VIDEO FPS 60 (motion is smooth and normal)
    Video looks correct except transparency is permitted again with only sharp alpha clipping - no glow effect evident.

    SETTING 3: TRANSCODE, VP8 CODEC, KEEP ALPHA:

    Android Result:
    SCREEN FPS 60, VIDEO FPS 60 - Gives correct frame rate output, but nothing is visible - video is not rendered, just a big black box.

    Windows Editor Result:
    SCREEN FPS 60, VIDEO FPS 25.58075 (motion is smooth and normal but wrong frame rate)
    Video looks correct except again transparency is permitted only with sharp alpha clipping - no glow effect evident.

    I don't have an iOS/iPhone device to test but I will need this working in iOS also.


    TO SUMMARIZE:

    (1) No way to currently get smooth transparent video playback at correct frame rate on Android (very big problem).
    (2) Can get smooth transparent playback on Windows by not Transcoding there.
    (3) Unsure what the outcome would be on iOS/iPhone.
    (4) Even when transparency is allowed, it is only with a sharp alpha edge - under no circumstances is glow manifested.

    Thanks for any further feedback or suggestions.

    Bug Report: IN-31790
     
    Last edited: Feb 11, 2023
  4. The_Island

    The_Island

    Unity Technologies

    Joined:
    Jun 1, 2021
    Posts:
    502
    Thank you for the bug report! So I just checked your projects, and here is what I found.

    First, the sharp alpha clipping - no glow effect seems to be an issue with the input. If you check the textures, you see the alpha in the original file is clipping the glow.

    upload_2023-2-13_14-2-38.png

    Now for the wrong fps. I don't know if you saw the console, but you should have seen this warning when you imported your clip.
    Video Decoding Error : 39 defective frames out of 68


    I am still unsure why it happened, but I found two ways to fix it. You can either add -vsync cfr to your FFmpeg command, so your clip has a constant framerate or removes the -metadata:s:v:0 alpha_mode="1". Both options will fix the fps issue and keep the alpha channel.

    By the way, I want to reiterate that you don't need to transcode your clip on any platform other than Android to keep the alpha channel. We already detect if there is alpha in the file, and we should output the texture correctly. So only override the clip on the Android platform, not all of them.

    upload_2023-2-13_15-9-58.png

    I hope this helped!
     
  5. The_Island

    The_Island

    Unity Technologies

    Joined:
    Jun 1, 2021
    Posts:
    502
    Ok, I think I found the root of the issue. This is the FFmpeg command, you used to convert your 29 images to a webm.
    Code (CSharp):
    1. ffmpeg -start_number 1 -i ./%04d.png -r 60 -c:v libvpx -crf 10 -b:v 2M -pix_fmt yuva420p -auto-alt-ref 0 -metadata:s:v:0 alpha_mode="1" ./output.webm
    In this command, you use the -r to control the framerate but you are actually changing the output framerate, not the input framerate. The default framerate in FFmpeg is 25 fps. So what happens is FFmpeg calculates the input duration 29/25 = 1.16s and then you ask FFmpeg to change the framerate to 60 fps which it does but doesn't change the file duration only the frame duration. So in your file header, the duration is 1.16s but the sum of all 29 frames is just ~0.483s. So we complain that there are 39 missing frames in your files that we couldn't read.

    What you really want to do is to use -framerate and remove the -r like this.
    Code (CSharp):
    1. ffmpeg -framerate 60 -start_number 1 -i ./%04d.png -c:v libvpx -crf 10 -b:v 2M -pix_fmt yuva420p -auto-alt-ref 0 -metadata:s:v:0 alpha_mode="1" ./output.webm
    Do this instead of my previous workaround.
     
    Last edited: Feb 14, 2023
    mikejm_ likes this.
  6. mikejm_

    mikejm_

    Joined:
    Oct 9, 2021
    Posts:
    346
    Thanks for the detailed assessment and solution! I will need to do some more confirmation tests, but I can definitely get at least a few videos running at full speed now with transparency on Android using just the Override & Transcode with Auto Codec on Android setting. I left default/windows/iOS with no transcode/override setting.

    I will need to post on a Blender forum to clarify how to get the alpha to manifest a glow like this.

    Just to clarify one last thing: Would this type of video encoding and setting be expected to work in iOS? Ie. Would I need an override/Transcode setting for iOS or could I leave that alone with nothing same as Windows?

    Thanks again. Very much appreciated.
     
  7. The_Island

    The_Island

    Unity Technologies

    Joined:
    Jun 1, 2021
    Posts:
    502
    No need to transcode just like Windows.

    The only exception is Android because Android is the only platform with its own VP8 implementation. Every other platform shares the same code for VP8. We did this to get access to hardware-accelerated codecs, and we thought Google, the creator of VP8/webm, would support all features natively on Android but they don't. At the very least, they didn't when we first implemented it.
     
    mikejm_ likes this.