Search Unity

  1. Good news ✨ We have more Unite Now videos available for you to watch on-demand! Come check them out and ask our experts any questions!
    Dismiss Notice

Unity Frame accurate video or image sequence playback

Discussion in 'Film, TV, Animation & Cinematics' started by Matjio, Mar 26, 2020.

  1. Matjio

    Matjio

    Unity Technologies

    Joined:
    Dec 1, 2014
    Posts:
    32
    Hi,

    We have regular requests from users to be able to playback videos or image sequences in an accurate way.
    Usage can be the playback of 2D animations or 2D effects (for example when mixing 2D and 3D animation) or to record a video plate in sync with graphics in Unity.

    We currently have not started any work to have built-in support for that in our video player. The gist of it is that many platform-supplied playback engines follow their own internal clock and don’t give the option to sync to an external source. To get this capability, we need a different implementation that is likely to have lower performance. It’s a performance-vs-flexibility trade off.

    In the meantime, you have various solutions depending on your use case:
    -HAP playback (again Keijiro's amazing contributions!): https://github.com/keijiro/KlakHap
    -Convert your video to an image sequence and use the new Streaming Image Sequence package (Preview): https://docs.unity3d.com/Packages/com.unity.streaming-image-sequence@0.0/manual/index.html
    -Use the trick described at the bottom of this thread to sync any video player with your rendering: https://github.com/Unity-Technologies/GenericFrameRecorder/issues/78 (The video source frame rate and the target framerate need to match then)

    We would be interested to get your use cases, your needs, and your feedback on the alternatives above.

    Best,
    Mathieu
     
  2. volvis

    volvis

    Joined:
    Jan 10, 2013
    Posts:
    11
    Hello! I can share some of our video playback experiences in our B2B projects.

    We're constantly balancing between providing striking visuals and often quite low-end business laptops that the applications are usually run on. We have a skilled VFX and mo-graph team in-house, and we would really benefit if we could utilize them to the fullest.

    Our latest use cases (done on 2019.2) have involved

    - Looping video in the background and showing canvas UI on top. This mostly works, but I do wish the video player could render directly to RawImage without requiring setting up RenderTextures. Every clip requires its own RenderTexture asset, because you sometimes see a frame of another clip before the right one loads. It's a hassle and prone to mistakes during development.

    - Using WebM with alpha to display animations that are too complicated to build by other means. I recall this causing issues with performance, so it's off the table for now.

    - Using PNG sequences for infographic animations. Common challenge was packing the sprites optimally and keeping the texts from blurring. We're experimenting with slicing each frame into smaller chunks, comparing if the chunk shows change from the previous frame, and reusing previous chunks as much as possible. (We got the atlas count down about 60%, but there are still issues to be sorted out)

    - Matching video and timeline animations. A common need is to be able to reliably play, loop and pause between sections in film and control the flow and pacing. This, well, did not work too well as you might guess, but we will look into the HAP repo you linked. Also, if we had frame perfect match with Timeline, we could use the Unity text rendering to support more screen sizes (it's anything between 800px wide and 4K).

    - In an earlier project, we experimented with SVG sequences. I recall the scene loading improved compared to PNG sequences in that one case, but the lack of anti-aliasing was a bit jarring to the eye.

    Those are some use cases that I can recall.. Hope this helps.
     
  3. Matjio

    Matjio

    Unity Technologies

    Joined:
    Dec 1, 2014
    Posts:
    32
    Thanks @volvis , very good list of use cases!
    If possible, could you give more context on the project(s) itself for us to get the full picture?

    - Looping video in the background and showing canvas UI on top. This mostly works, but I do wish the video player could render directly to RawImage without requiring setting up RenderTextures. Every clip requires its own RenderTexture asset, because you sometimes see a frame of another clip before the right one loads. It's a hassle and prone to mistakes during development.
    Very good point. We will discuss internally to see how we could improve this.
    Note that you could clear the render texture when you finish the previous clip or before starting the new clip.
    By the way, here is a good hint on how to setup rendertextures to playback videos with best quality:
    https://forum.unity.com/threads/aliasing-on-video-played-back-on-a-texture-in-vr.849820/

    - Using WebM with alpha to display animations that are too complicated to build by other means. I recall this causing issues with performance, so it's off the table for now.
    For best performances you might need to transcode the original import format:
    https://docs.unity3d.com/Manual/VideoTransparency.html
    https://docs.unity3d.com/Manual/VideoSources-FileCompatibility.html

    - Using PNG sequences for infographic animations. Common challenge was packing the sprites optimally and keeping the texts from blurring. We're experimenting with slicing each frame into smaller chunks, comparing if the chunk shows change from the previous frame, and reusing previous chunks as much as possible. (We got the atlas count down about 60%, but there are still issues to be sorted out)
    If you are on a PC, you could just go with uncompressed or less compressed PNGs and stream them from disk (not put them in the Unity project which will compress them). Of course that takes a lot more space on disk.

    - Matching video and timeline animations. A common need is to be able to reliably play, loop and pause between sections in film and control the flow and pacing. This, well, did not work too well as you might guess, but we will look into the HAP repo you linked. Also, if we had frame perfect match with Timeline, we could use the Unity text rendering to support more screen sizes (it's anything between 800px wide and 4K).
    Yes please check the different methods I mentioned above. Would this scenario be to playback at real-time or to output a video from Unity Editor? Image sequence streaming and HAP playback were both used successfully for similar usage at real-time.

    - In an earlier project, we experimented with SVG sequences. I recall the scene loading improved compared to PNG sequences in that one case, but the lack of anti-aliasing was a bit jarring to the eye.
    Did this happen using our SVG importer?
    https://docs.unity3d.com/Packages/com.unity.vectorgraphics@2.0/manual/index.html
     
  4. volvis

    volvis

    Joined:
    Jan 10, 2013
    Posts:
    11
    - If possible, could you give more context on the project(s) itself for us to get the full picture?
    I think we can, but not on a public forum. Will discuss with the team.

    - If you are on a PC, you could just go with uncompressed or less compressed PNGs and stream them from disk (not put them in the Unity project which will compress them). Of course that takes a lot more space on disk.
    I'm under the impression that loading PNG files from disk stalls the main thread while the file is converted to texture. We sometimes have 1080p "slides" in the streaming assets, and pulling even one causes the UI to hiccup. We're exploring precaching the files on boot, but I don't see how this would work at 30 frames per second.

    - Yes please check the different methods I mentioned above. Would this scenario be to playback at real-time or to output a video from Unity Editor? Image sequence streaming and HAP playback were both used successfully for similar usage at real-time.
    It's for runtime. Will explore those options. It seems that utilizing stock footage is becoming more central in our productions.

    - Did this happen using our SVG importer?
    Yes it did.
     
  5. Ivan_br

    Ivan_br

    Joined:
    Nov 10, 2015
    Posts:
    6
    Hi Mathieu,

    Some of the video playback issues I have had are:

    1. One project we use the Oculus Quest with multiple (6-8) 4K 360 videos which are of slight graphical variations from one another. We need to seamlessly switch between these videos as a result of user input and the switch needs to be at the exact frame. We worked around it by switching between images and synching up the audio to start at the same time and hope nothing goes out of sync.

    If there were to be a method where we could preload videos and switch to a specific frame it would make things easier.


    2. I use Unity to create VFX videos with augmented reality in the Unity editor. I have a way of getting a raw video (without the augmented objects) from a file to play in Unity and use an AR sdk on top of that to process for AR markers, which adds virtual objects on top of the located markers. Sometimes I need to manually adjust virtual objects in my scene, and this may take a while, so it would be nice to have more control over the speed in which the frames in the video play at.

    It is also common that the project requires chroma-keying in the unity editor, in which a video is placed in the background. So I now would have 2 videos, one in the foreground which will be used to process for markers and be chromakeyed, and another video which will be the new background, these also have to be in sync, along with all the rest (animations, audios, virtual lighting, etc.). This all needs to occur while the Unity Recorder is also running in order to export all of this as a video.

    If I could scrub through the video frames as if I were to scrub through the Unity timeline, for instance, it would make it much simpler to make these edits at my own pace, especially if I need to adjust multiple objects per frame.

    If there was a way to sync the video playback to my own function, for instance, this would also give me more control.

    Hope that helps.

    Thank you. Best,
    Ivan
     
    Last edited: Apr 3, 2020
  6. cineconcerts

    cineconcerts

    Joined:
    Dec 18, 2019
    Posts:
    9
  7. Ivan_br

    Ivan_br

    Joined:
    Nov 10, 2015
    Posts:
    6
    Hi @Matjio, are there any updates on this?

    Thank you.
     
    LostPanda likes this.
  8. LostPanda

    LostPanda

    Joined:
    Apr 5, 2013
    Posts:
    149

    I feel that unity has given up and ue ndisplay has been very perfect.
     
unityunity