Video How to know when VideoPlayer is waiting for new frames (if source is URL)?

We have a game where a video from a URL is synced with gameplay. We want to pause gameplay whenever the video stops due to a slow network, and resume when it resumes. It's rhythm-based, so timing is crucial.

However, there doesn't seem to be a way to know this. Each time we get a frameReady callback we know the video has a usable frame to display, but then the best we can do is to check in Update how long it has been since the last frameReady callback, and assume the video has stopped if past a "certain time".

The very best we can do is to compare the clockTime every Update with the clockTime at the time of the last frameReady callback PLUS the duration of a video frame. Even then, we have to wait another Unity frame, to make sure that the VideoPlayer has time to check the frame on its own.

(frameReady seems to be called after all our code, no matter script execution order. It's called after LateUpdate and even after WaitForEndOfFrame, which is already the last resort. No calls that we make in frameReady even show up in the Profiler; it's like a ghost method that runs after everything but still on the main thread(?).)

Why is there not a callback for when the video fails to play the next frame because there is no next frame? We even have callbacks for clock resyncs but not one for the video pausing because it has no more data...? Any ideas?

 

We have been trying everything to solve a lot of problems with the VideoPlayer, especially on iOS and macOS. The implementation is very broken sometimes, and very inconsistent between platforms.

We've reported these problems in the following bug report. Case number 1283295.


Title: Lots of bugs and platform inconsistencies in Unity's VideoPlayer


1. What happened

We've been trying to understand Unity's VideoPlayer on different platforms, in order to synchronize gameplay/subtitles with videos. We found A LOT of problems in the VideoPlayer that make it pretty much impossible to achieve that objective reliably.

We spent many hours organizing the problems into this single report. It is a multi-issue report!

Forum thread:
https://forum.unity.com/threads/how...iting-for-new-frames-if-source-is-url.976038/


2. How we can reproduce it using the example you attached

For each problem, play the scene on the given platforms and test the steps:

NOTES:
- All tests use Freerun timeReference unless specified.
- All tests use VideoClip "Video - frames" unless specified (referenced by default).
- "Any video" means URL, VideoClip, transcoded, non-transcoded, etc.

[===== Sure seem like bugs =====]

- Windows, any video.
- Prepare, Seek.
- seekCompleted happens but no frameReady. Not even StepForward causes new frames to happen.
- Press Play now. Nothing will happen but .clockTime will be counting from 0 to the time of the seeked frame (100 in our examples). When it gets there it finally calls frameReady from then on.
- (i.e. If we want a seek to work after a Prepare without Play, we must Pause before the seek)

- iOS/macOS, any video.
- Prepare, Seek.
- seekCompleted happens but no frameReady (would need a Play, Pause, or StepForward).
- Switch to another app window on macOS or press the Home button on iOS and then return to the app: a frameReady happens.

- NOTE: This one uses InternalTime timeReference.
- Windows/Android/macOS/iOS, any video.
- Prepare, Pause after prepare completes, Play.
- The video starts at some arbitrary frame like 231.
- If, however, you DON'T call Prepare beforehand, it Plays from frame 0 as expected.

- iOS/macOS, any video.
- Prepare, Play, Pause, Seek.
- When seeking, one or more frameReady calls will happen and report the new frameIdx (all equal). However, .frame and .time DO NOT update. Then we get the seekCompleted event, still on the wrong .frame.
- The frameReady for the right frame never happens. Now we need to StepForward or Play to get the FOLLOWING frameReady.
- NOTE: Pause doesn't generate a frameReady for this case, although it does work if you just Prepare and Seek.

- iOS/macOS, "Test - Transcoded Video (forced 30fps)".
- Prepare, Play, Pause before frame 15. Then Play again and see that it continues just fine.
- Prepare, Play, Pause after frame 15 but before 30 (ex.: 22). Then Play again and see that it jumps to 30, stays there a few Unity frames, and then continues.
- This happens on multiples of 30. The first 15 frames (ex.: 30-44) are fine, then the next 15 (ex.: 45-59) jump to the next multiple of 30 (ex.: 60).
- NOTE: For the clip "Video - frames" this also happens, but only every 60 frames instead of 30 (i.e. it jumps if video is in the 15 frames preceding a multiple of 60 frames).

[===== Video-specific (might or might not be bugs) =====]

- Windows, "Test - Non-Transcoded" OR "Test - Transcoded Video (forced 30fps)".
- Prepare, then multiple Steps.
- No frameReady happens for the first few calls, then frameReady happens for frame 0.

- Windows, "Test - Non-Transcoded" OR "Test - Transcoded Video (forced 30fps)".
- Prepare, Play, Pause, Seek to frame 100.
- frameReady happens for frame 99 (may be 100 for some videos), although the video displays something like frame 97 or 96.
- Stepping afterwards will always produce a frameReady that is that number of frames ahead of the video.

- Windows, "Test - Transcoded Video" (but not always).
- Prepare, Play, Pause.
- Step multiple times. Most of the time(?), you'll see that the frame only updates each X steps (when frameReady happens).

[===== Other platform inconsistencies =====]

- Prepare, Seek.
- Windows/Android: seekCompleted happens, then frameReady happens for the new frame (.frame is now the new frame).
- iOS/macOS: seekCompleted happens and that's it. The frameReady for the new frame never happens unless we Play or StepForward. Also, if the video is Playing before Seek, frameReady happens for the old frame before seekCompleted (why??).

- Prepare, Seek multiple times to different times (using the time slider).
- Windows/Android: a seekCompleted happens for each time we seeked, followed by one frameReady in the new frame.
- iOS/macOS: only one seekCompleted happens.

- Prepare, Seek, Step (to ready the right frame), Play.
- iOS/macOS: The video reports isPlaying True but is still stuck on the same frame for a while. While this happens the clockTime never updates until the very last frames before frameReady.
- Windows/Android: clockTime always seems update fine while isPlaying is True.

- Prepare, Play.
- Windows/Android/macOS(!): frameReady always happens after everything in the Unity frame (even after WaitForEndOfFrame).
- iOS: frameReady always happens before Unity Update.
- (script execution order doesn't matter)

 

Tagging @DominiqueLrx @cguertin for possible help again. I've tried getting help about this two weeks ago (either for the original problem or for all the bugs in the VideoPlayer). Now I've deleted the old posts and re-posted with our updated findings/bug report.

 

Got a reply to the bug report. The first 4 issues were confirmed:

1. Reproduced (link: https://issuetracker.unity3d.com/product/unity/issues/guid/1297646/)
2. Reproduced (link: https://issuetracker.unity3d.com/product/unity/issues/guid/1302462/)
3. Reproduced (link: https://issuetracker.unity3d.com/product/unity/issues/guid/1297648/)
4. Reproduced (link: https://issuetracker.unity3d.com/product/unity/issues/guid/1302463/)

The others were not reproducible on the tested devices. We'll be trying to collaborate to find out why.

 

Unity 2019.4.13f1 causes a NEW and BIG problem, and it's LTS :'( Reported as issue 1304061.

UPDATE: The team created 3 new issues for it.

https://issuetracker.unity3d.com/is...when-audio-output-mode-is-set-to-audio-source
https://issuetracker.unity3d.com/is...-a-delay-when-videoplayer-dot-pause-is-called
https://issuetracker.unity3d.com/is...ed-with-audio-output-mode-set-to-audio-source

 

We have been trying to sync up videos over a network and pretty much had most of these issues, its very difficult to reliably control the video player in 2019 LTS right now.

 

Yep! Our code does its very best to try to find network issues on its own (such as the same video frame remaining for multiple Unity frames, but that's unreliable) but even then we have to sync the gameplay to the video by speeding up/slowing down, since we may have found the network problem "too late". Syncing to subtitles I'm sure has the same kinds of problems.

Thanks for reminding me of this thread. I'll summarize the current updated links for all the issues:

1. https://issuetracker.unity3d.com/product/unity/issues/guid/1297646/
2. https://issuetracker.unity3d.com/product/unity/issues/guid/1302462/
3. https://issuetracker.unity3d.com/product/unity/issues/guid/1297648/
4. https://issuetracker.unity3d.com/product/unity/issues/guid/1302463/
5. Fixed somewhere between Unity 2019.4.8f1 (still has the problem) and 2019.4.17f1 (no problem). See the issue.
6. https://issuetracker.unity3d.com/product/unity/issues/guid/1316857/
7. https://issuetracker.unity3d.com/product/unity/issues/guid/1316931/
8. Discarded. We can't reproduce anymore.
9. May be issue 2 again.
10. Discarded. We can't reproduce anymore.
    
11. https://issuetracker.unity3d.com/product/unity/issues/guid/1316930/
12. Cannot reproduce?

Plus the ones found on Unity 2019.4.13f1:
https://issuetracker.unity3d.com/is...when-audio-output-mode-is-set-to-audio-source
https://issuetracker.unity3d.com/is...-a-delay-when-videoplayer-dot-pause-is-called
https://issuetracker.unity3d.com/is...ed-with-audio-output-mode-set-to-audio-source

Thanks to Tautvydas Kubolis (and maybe more people on the Unity bug-checking team) who had to be extremely patient trying to reproduce these bugs on all the reported devices. It was a lot of testing.

- Daniel