Official Frame accurate video or image sequence playback

Hello! I can share some of our video playback experiences in our B2B projects.

We're constantly balancing between providing striking visuals and often quite low-end business laptops that the applications are usually run on. We have a skilled VFX and mo-graph team in-house, and we would really benefit if we could utilize them to the fullest.

Our latest use cases (done on 2019.2) have involved

- Looping video in the background and showing canvas UI on top. This mostly works, but I do wish the video player could render directly to RawImage without requiring setting up RenderTextures. Every clip requires its own RenderTexture asset, because you sometimes see a frame of another clip before the right one loads. It's a hassle and prone to mistakes during development.

- Using WebM with alpha to display animations that are too complicated to build by other means. I recall this causing issues with performance, so it's off the table for now.

- Using PNG sequences for infographic animations. Common challenge was packing the sprites optimally and keeping the texts from blurring. We're experimenting with slicing each frame into smaller chunks, comparing if the chunk shows change from the previous frame, and reusing previous chunks as much as possible. (We got the atlas count down about 60%, but there are still issues to be sorted out)

- Matching video and timeline animations. A common need is to be able to reliably play, loop and pause between sections in film and control the flow and pacing. This, well, did not work too well as you might guess, but we will look into the HAP repo you linked. Also, if we had frame perfect match with Timeline, we could use the Unity text rendering to support more screen sizes (it's anything between 800px wide and 4K).

- In an earlier project, we experimented with SVG sequences. I recall the scene loading improved compared to PNG sequences in that one case, but the lack of anti-aliasing was a bit jarring to the eye.

Those are some use cases that I can recall.. Hope this helps.

 

- If possible, could you give more context on the project(s) itself for us to get the full picture?
I think we can, but not on a public forum. Will discuss with the team.

- If you are on a PC, you could just go with uncompressed or less compressed PNGs and stream them from disk (not put them in the Unity project which will compress them). Of course that takes a lot more space on disk.
I'm under the impression that loading PNG files from disk stalls the main thread while the file is converted to texture. We sometimes have 1080p "slides" in the streaming assets, and pulling even one causes the UI to hiccup. We're exploring precaching the files on boot, but I don't see how this would work at 30 frames per second.

- Yes please check the different methods I mentioned above. Would this scenario be to playback at real-time or to output a video from Unity Editor? Image sequence streaming and HAP playback were both used successfully for similar usage at real-time.
It's for runtime. Will explore those options. It seems that utilizing stock footage is becoming more central in our productions.

- Did this happen using our SVG importer?
Yes it did.

 

Hi Mathieu,

Some of the video playback issues I have had are:

1. One project we use the Oculus Quest with multiple (6-8) 4K 360 videos which are of slight graphical variations from one another. We need to seamlessly switch between these videos as a result of user input and the switch needs to be at the exact frame. We worked around it by switching between images and synching up the audio to start at the same time and hope nothing goes out of sync.

If there were to be a method where we could preload videos and switch to a specific frame it would make things easier.


2. I use Unity to create VFX videos with augmented reality in the Unity editor. I have a way of getting a raw video (without the augmented objects) from a file to play in Unity and use an AR sdk on top of that to process for AR markers, which adds virtual objects on top of the located markers. Sometimes I need to manually adjust virtual objects in my scene, and this may take a while, so it would be nice to have more control over the speed in which the frames in the video play at.

It is also common that the project requires chroma-keying in the unity editor, in which a video is placed in the background. So I now would have 2 videos, one in the foreground which will be used to process for markers and be chromakeyed, and another video which will be the new background, these also have to be in sync, along with all the rest (animations, audios, virtual lighting, etc.). This all needs to occur while the Unity Recorder is also running in order to export all of this as a video.

If I could scrub through the video frames as if I were to scrub through the Unity timeline, for instance, it would make it much simpler to make these edits at my own pace, especially if I need to adjust multiple objects per frame.

If there was a way to sync the video playback to my own function, for instance, this would also give me more control.

Hope that helps.

Thank you. Best,
Ivan

 

I can't find where to download the streaming image sequence package.

Edit: Found it. https://github.com/unity3d-jp/StreamingImageSequence

 

Hi @Matjio, are there any updates on this?

Thank you.

 

I feel that unity has given up and ue ndisplay has been very perfect.

 

Hi, I've been trying to make short film regarding a drive scene but I've having problems with as animating the car starting from the tyre rotation to making turns. How can i go about this? I don't know how to code.

 

if you dont know how to code your using bolt or playermake or other visual script tool

 

Hi Mattieu, is this review still an ongoing process? I'm definitely going to try this Hap solution. I tried the streaming image sequence a few months ago and it crashed the project I was working on, but my memory is kind of hazy so would have to revisit it to remember exactly why.

Similar to other use cases here, I need reliable and correct matching between VideoPlayer, Timeline and Unity Recorder. I use chroma keying on either prerecorded or live footage in most of my projects to get real-life actors into a Unity scene.

More control over Unity Recorder would be very useful too, like bitrate control, etc.

What would also be really useful is to be able to set either the game view in the editor or the actual built game as a virtual webcam to use as input for, for example, OBS for live streaming.

 

OK I have an issue I have exported transparent/Alpha .mov files from spine 2d animating program and I tried to import them into unity to use them as in game animations and they imported the to asset folder the show up in the project folder but when I drag them on to the hierarchy it deosnt show and I get a global transcode error, plus the info window shows the specs but it’s not able to play in the source window as seen in photos provided. Maybe unity deosnt support this kinda of usage I’m super confused and I have a deadline to meet can someone help me?

 

For everybody interested in network synced/manually-driven video playback: http://u3d.as/2KgC

 

Hi,

I don't currently need multi machine sync, but is the $129 version of the plugin able to playback stuff synchronized frame accurately to Unity (ie. does it playback without frame drops?) or do I need to buy this version to ensure that the time runs in sync with Unity's frame clock?

The reason I'm asking is that we're shooting film/ads/whatever stuff against a LED volume, and if the system occasionally drops frames, we can't use it.

br,
Sami

 

Hi Sami!
Generally the simple version is able to play without framedrops (if machine specs match the video, there are no other heavy apps running, etc), but I can imagine a situation where playback starts not exactly the same Unity frame when you call Media.Play (you need to check). The main aim in the simple version is smooth playback.

It also worth noting that simple playback is using a clock instance which is tied either to the audio stream timeline if having audio, or to the internal clock which is running on its own and not synced with Unity, unlike the pro sync version. If a hiccup is happening in Unity main loop, the video timing should be unaffected by that. Another thing worth noting: even in the simple version the video texture is being updated (actually replaced in latest versions) within the Unity frame.