[RELEASED] MPMP - cross platform video solution

Yeah, I tried this earlier and the frame delay fixed it. Would be a great addition for the next release, we have situations where we need to load the videos as soon as they are ready.

 

Ok. Already in my current develop branch. Comes with the next release.

 

Hello again.
I'm trying to use the asset in macOS and finding some issues in the seek functionality.
I'll explain. I have tried both the skee with the bar and the seek functions I shared in the first page and what I found is that the player misses the time by a lot. With a 33 seconds video I'm only able to seek to the beggining, 1 point near the middle and to the end.

I'm wondering if it is a problem with AVFoundation and the way it handles seeking or if I'm doing something wrong (although it works flawlessly in windows)...

 

Seeking accuracy heavy depends on the codec and platform you use. The media framework only can seek exactly to keyframes. So you should test your video with different encoding settings to get the best results.(For example H264 is not the best codec for good seeking.) When encoding your video you should make as much keyframes as possible. But this is always a difficult tradeoff between quality and filesize.
But i will look into the AVFoundation code. As far as i remember i can use different seek behaviours. I look if i can improve this.

 

Ok, tested in IOS since they use the same AVFoundation and the same issue exists. I also noticed that once the movie begins a lot of artifcats can be seen... almost looks like the artifcats that the old vlc used to have when seeking... maybe the issue is the same?

I'm currently testing with mp4. I'll also take a look at the encoding on my side.

 

Ok, the issue is definetely with the keyframes as another video actually produces a much better result. However, in windows, even the first video worked flawlessly, different seeking methods maybe?

Also, it appears that in IOS and MacOS the first frame is being ignored. (shows white in IOS until the second keyframe and in mac it tries to interpolate it seems and I get the artifacts...)

 

Ok, I have some new findings that might help.

The first keyframe actually exists and it's located at the first frame of the movie. As such, the issue seems to be that it is not being read.

I'm loading the movie on the Start function and I can certify that it is working correctly once loaded (after the OnLoaded event all the info appears correctly even if the texture is not being shown)
Also, in Mac, the interploation is working as I gert an image which is basically a single artifcat at the beginning as apparently the texture isn't updated on the first frame.

I can see 2 possibilities here, either the loader is somehow skipping the first frame which in turn leads to the issue with the artifacts or it can be just a timing issue with the player trying to play the movie before it gets loaded and hence missing some frames...

 

Anybody know what the highest video resolutions / bit rates are appropriate for ios, android and GearVR?

I get 1080p h264 30 fps playing in everything but my experiments with other resolutions are hit or miss. I tried the suggestions from purple pill but I'm failing with them:

“Optimal” encoding settings
After studying how the big guys do it, we decided that the following encoding settings would be optimal for now:

Platform Codec Resolution FPS Avg. bitrate
Gear VR h.265 3840×2160 30 10 – 20Mbps
Cardboard Android h.264 (Baseline, level 4.2) 3840×2160 30 20 – 30Mbps
Cardboard iOS h.264 (Baseline, level 3.1) 1920×1080 30 10 – 14Mbps
Oculus Rift h.265 / h.264 4096×4096 60 40 – 60Mbps​

Guidance on video codec, maximum bitrate and resolution would help me greatly. Thanks

 

Would I be able to use this plugin for a 360 video player?

 

Should be no problem from a technical standpoint as you work in Unity with a normal material that you can assign to a mesh of your choice. To get a seamless video texture it is more a problem of the right texture mapping of your mesh and the video footage. An you have to check the performance as i assume it require a large video resolution.
On my todo list is a new Demo scene that should show a video mapped onto a sphere where the normals are flipped so you can stand inside the sphere and look at a video . If somebody has some opensource 360° video footage it would be great if he could post a link.

 

I will add for the next release an option to use another seek modus for iOS/OSX. This uses a tolerance value so the seeking will be more accurate but in the AVFoundation documentation is mentioned that it could cause some decoding delay.

 

That would really help me out immensely as I'm trying to build out a prototype as I dive into this field. I'll keep trying to get figure out how to flip the normals on a sphere.

 

New issue. _OnLoaded callback just bombs my android build

On the _OnLoaded call I just tried to set my object size (hoping to get a "mantain native ratio" functionality) and I get INTERNAL_set_sizeDelta can only be called from the main thread.

That being said, it works on all the other platforms /even though the warning is also shown) and I'd believe it is not related to the plugin itself but on how Unity handles the events.

In Android the app just plainly crashes...

 

To bad, i only tested it with scaling a normal gameobject and had no problems .(API Demo scene ) . So you try to scale a rectTransform? Seems to be not threadsafe.

 

I urge you to try again then xD

Even normal gameobjects make use of Unity's API which is not thread-safe so it's possible it worked by chance...

I'm trying to get some sort of calculations done and then passing them to the main thread...

 

Ok, so, some yields and a few control variables and It's working well now. But might be good to provide this info as an heads up with the callbacks

A bug I noticed though was the MPMP.GetDuration()
At least in my build it is always returning 0. Only on Android though :x

 

I will change the callbacks so the where called from the main thread (when the update method is called)
The GetDuration should work on Android but it could be that you call it in the wrong moment. I have looked into the code and realized that i could remove some state check so the duration could be accessible right when onLoaded is called.

 

I'm calling it at the same time as I call in other platforms... can it be that my Android takes longer to process some code and the variable simply isn't updating?

 

After some testing I can surely say that I cannot get the duration. I used it for the seeking I previously posted but at this moment I just went with a bit of an hack by changing based on percentages instead of direct values.

 

I made some test with debugging the duration at the Update method of MPMP and i have no problems getting the duration after the video is loaded.
My call is:

Code (CSharp):

1.  Debug.Log("Duration:" + GetDuration());

 

I finally found the issue with the GetDuration on Android. There was an int/double conversion problem so the values where corrupted. A fix comes in the next version. I also change the general duration unit for all platforms to seconds.(In the documentation i wrote milliseconds by mistake)

 

Awesome work
Any ETA of when we can expect the new version to hit the store?

 

If there are no other problems i should have a new demo version in a couple of hours. The official version will perhaps have a delay of one week (The 1.1 version is still in review )

 

I uploaded a new version (the demo version is already updated online):

Version 1.2 - 2016.02.11

• Events are now called from the main thread
• SetPlaybackRate implemented
• OnInit event implemented
• Fixed issue with OnError events on Windows
• SeekToWithTolerance for iOS/OSX
• Fixed issue with GetDuration on Android and iOS/OSX. The values are now available at OnLoaded in the right unit (seconds)
• ScaleFlipY.cs renamed to ScaleFlip.cs. The script has now the axis mode property for selecting which axis to flip
• Added a 360° demo scene for watching panorama videos
• Documentation update

On iOS/OSX there is a second seek mode available that specifies a tolerance value (default 0) for more
precise seeking but could cause some decoding delay. To enable this seek mode you have to enable the
SEEK_TOLERANCE definition at the top of the MPMP_API.cs script.
The new 360° demo scene is just a sphere with the right uv (sphere mapping) and the normals are flipped so you can point your camera inside the sphere and watch the video. I made a test with some footage from here: http://www.360heros.com/vr/ (Dragster sample)

Hopefully the 1.1 review is skipped and you get the 1.2 version directly

 

theres an issue with your demo... I downloaded it from the link in the asset store and imported it into a blank unity 5 project. Ran the MPMP.Basic scene and ran it. Clicked play (nothing happens), click load and it crashes unity..

 

On which platform are you? Please read the system requirements and the installation process in the manual.pdf

 

It's running on windows in unity and platform selected is Windows standalone as well..


So on windows it required directx 11? that seems a bit limiting..

Also, on android it requires openGL 2? s if you use OpenGL 3 or Auto graphics api it won't work?

 

I'm working on a OpenGL 3.0 version. (Btw. MPMP changes automatically the graphics API to OpenGLES 2.0 when publishing for mobile.)

 

So since i am using windows inside a VM, and can't use directx11 in the editor... that means i can't use MPMP?

 

On Windows you can publish the scene for mobile without running it in the editor or use the OSX Unity editor

 

tried it in OSX... get this error:

EntryPointNotFoundException: UpdateCallbacks
monoflow.MPMP.__Update () (at Assets/MPMP/Scripts/MPMP.cs:400)
monoflow.MPMP.Update () (at Assets/MPMP/Scripts/MPMP.cs:347)


Has this thing been out long?

 

actually all 3 example scenes get this error on OS X..

 

Did you have used a fresh new project with the latest demo package?
Normaly this can't happen as the call to UpdateCallbacks() is inside a preprocessor directive :

Code (CSharp):

1. #if (UNITY_STANDALONE_WIN || UNITY_EDITOR_WIN)        
2.                 UpdateCallbacks(_id);
3. #endif

Under normal circumstances this code isn't available under OSX.
Or do you have Windows as target platform under OSX? Than i have to make an additional check that this code should not be executed. (bug)
So you can change your target platform to OSX or iOS and it should work.

 

do you plan on fixing the limitations?

 

Yes, but i can't make any statements about the time frame. Windows 7 is on top of my priority list.

 

Just FYI, i am actually looking for a solution to this:

http://forum.unity3d.com/threads/cross-platform-video-playing-from-url.385503/

 

I guess the problem with this video is that it is not encoded with the faststart option (Progressive streaming) so you can't play the video without complete downloading it. MPMP support progressive streaming so you can watch the video while downloading it. Best is you re-encode the video and do a test with MPMP.

 

For all users on OSX who are facing the 'EntryPointNotFoundException: UpdateCallbacks' exception in the editor:
Please change your target platform to OSX, iOS or Android . A fix for this issue comes with the next version.

 

how do you re-encode it? i am only finding one app that does it and it seems to not work..

 

The easiest way is to use Quicktime Pro :
1. Open your movie with the Quicktime player Pro
2. choose File > Export.
3. Choose "Movie to QuickTime Movie" from the Export pop-up menu.
4. Click Options and select video and sound compression options appropriate for web delivery.
5. Make sure the "Prepare for Internet Streaming" checkbox is selected and Fast Start appears in the pop-up menu.

Or perhaps use this tool : MP4 FastStart

 

Just tested the new version... Android works wonderfully now.
Also tested the IOS issue I had but it still ignores the first key frame of the video :/

Appart from that minor issue, amazing job

 

Here is a little video that shows a possible VR video player solution with MPMP:

The footage from the video is from http://www.360heros.com/vr/

The demo setup comes with the next MPMP version.

 

Version 1.2 is now available at the Assetstore

 

Hi, this looks amazing. Do you plan to implement hot spots? For example from X time to Y time show a button on a certain point of the video? If not I can make my own when I get round to the project I have in mind. Just curious as it would save even more time

 

I will think about it. Some script without any fancy features are not complicated but i have some more complex stuff in mind. But if you want a really user friendly solution (with timeline etc.) you can end in a product of your own.

 

Hi _monoflow, how many instances of mpmp can be run at the same time? I'm running a test to see if it's possible (however unlikely) to run 12 different videos at the same time, and apply the 12 textures across two skybox's for mobile VR.

This would allow us to get around the limit of only being able to show one 4k video on some platforms (which when you split the resolution for VR, is actually quite awful compared to what most people's expectations are when they first try mobile VR video). Your example VR scene works quite well for just getting a single video to play on an inverted sphere, however for VR the best possible quality can only be achieved using skyboxes since you can theoretically set each of the skybox's cube faces to a high quality texture/video.

This solution of having 12 textures over two skybox's works great for making fantastic looking VR 360 pictures, so I'm attempting to replicate that with video.

The problem I'm running into is that although the application itself doesn't dip too far in FPS in editor, the videos all play at a very low FPS, which makes this unusable. Can you talk about this idea and let me know if it's simply impossible with current technology?

 

Well i think you are trying to push the limits too far I never tested such a hardcore setup and i guess that this is for mobile an impossible task. The amount of MPMP instances that can run at the same time depends mainly on your graphics hardware capabilities ( don't forget texture memory!) And we all know that this is on mobile a bottleneck. I could imagine that on standalone with a high power graphics card (SLI/Crossfire mode?) it could work. I always use GPU-Z for tracing the usage of my GPU and look at the process manager to watch if the CPU has some problems so you can better judge if you need a better CPU or GPU.

 

Gotcha, is there anything I could set in the Unity project to increase performance that would effect MPMP?

Also, your my personal hero for making this asset. Thanks!

 

I just tried out GPU-Z with the following results. I'm new to this program, but does the following graph mean I've reached the GPU limit? This is a test scene of 6 panels (instead of the full 12) and I'm seeing single digit performance FPS within the videos themselves (I'm eyeballing it). The Unity scene is reporting a fast FPS though.

 

I made some stress test by myself with different setups. (Radon HD7750)
I can play 3x1080p without a problem but when i add a fourth the fps drops or is instable.
With 720p video i can play 7 instances at the same time at almost 60fps.
You can tweak the performance a bit if you change the fps property of MPMP. (look at around line 77 in MPMP.cs)

Code (CSharp):

1. private float fps = 1 //60f;

change it to 1/30f . This is the refresh rate for calling into the native plugin.
The GPU usage is not too high like in your case . There seems to be a bottleneck how many pixels altogether could be processed. Later i will test it on another machine.