MoviePlayer for Unity (plays MJPEG)

Hi, thank you for your reply. I really love the asset, it is a lifesaver.

#pragma strict did get rid of the debug warnings, but still does not allow acces to the play variable. Your second suggestion worked flawlessly and re-moving after an update is not a big problem at all.

you did a very good job and I Unity should learn from it allot! Stuff like this should be standard.

Keep it up!

 

Thank you, you praise me too much

 

Hi!
We upgraded your plugin recently and we still dont see the MoviePlyer actions in Playmaker. What should we do?

 

Please find "Assets/MoviePlayer/Playmaker actions.unitypackage" and import it. It will extract playmaker actions and one sample scene.

 

It's support .mp4 format -? Android 2.3 + - ? can I playing a few videos at same time ?

 

No mp4, only mjpeg (and some more that you care only if you know what you're doing). It does work on Android 2.3. Multiple videos at same time depends on video size and device performance. Primary target for this plugin is desktop where performance and file size are not issues, but it works on most if not all Unity build targets and devices.

 

There's a game concept I kind of want to prototype using this plugin, but one thing I'm kind of curious about accomplishing is transparentized movies; for instance, overlaying camera video on a static background. One possibility for this I know some people have used (when they're doing full coding of the entire video pipline) is a full-color video file mixed with a grayscale "alpha" video through some sort of filter or shader. I'd be curious if you think this could be a possibility with the interfaces the plugin provide? (I don't really expect it to be built into the plugin)

 

The video framebuffer is a regular Texture2D that gets its contents updated. Anything you can do with a texture in Unity can be promoted to use video instead of a still image. Transparent movies and various effects are no problem if you use the right shader. It could be color-key transparency, it could be two videos playing in sync, one feeding shader color channels, another feeding transparency (alpha) channel, or it could even be one video that's encoded into mpng that already has alpha in it (rgba per pixel), but mpng decoding performance is much worse than mjpeg.

 

Hi, is there a trial version of the plugin? I need to build an application that will run in Oculus Rift and it must have high quality (at least 1080p each eye, 60fps), so I would like to test it before buying. Maybe a watermarked version?

Thanks

 

1080p@60fps for both eyes can't do, but 720p@60fps for both eyes should work nicely on a good machine. You can maybe use the Standalone Feature Demo (links are in the first post of this thread) with Oculus too.

 

I can confirm, I have done exactly that oculus setup for a customer with cubemapped movies in a box projection for each eye to create a 360 degree movie. A trial is not needed. This plugin is a nobrainer for any kind of movie material you need in your game.

 

Made a video about how to use custom uGUI together with this plugin. It's moderately advanced and contains scripting, but first part of it is dead easy.

 

Hi! I bought this and got everything working correctly. My only problem is I have no coding experience and I would like to use this asset for cut scenes only. In other words I have no idea how to get it where when the video in done playing I can change scenes by calling this script:

using UnityEngine;
using System.Collections;

public class LoadLevel : MonoBehaviour {
void Example() {
Application.LoadLevel("Main Menu");
}
}

And then move on to the next scene.

Is there a simple fix I missed? Or is there I way you could help me out with the scripting to accomplish this? Again I have no experience with scripting unfortunately.

 

Thank you. For doing something when playback stops you can write a component and attach it to movie player game object. Something like this (Javascript).

Code (csharp):

1. var sceneName : String;
2. function OnStop() { Application.LoadLevel(sceneName); }

And in C# (alternatively you could register to GetComponent<MoviePlayer>().OnStop event, but that's a bit too much code for this simple task)

Code (csharp):

1. using UnityEngine;
2. public class LoadSceneOnStop() {
3.   public string sceneName;
4.   public void OnStop() { Application.LoadLevel(sceneName); }
5. }

 

Thank you! This is by far the fastest and most reliable video player (and the only cross-platform) I've found in the AssetStore or unity itself!
I thought that maybe I could make it even faster by loading multiple frames on multiple cores and the same time, but there is a problem:
We all know that you can't call Texture2D.LoadImage from another thread. This function does 2 things: Decodes the image to RGB format, and sends it to GPU. The second part (the part that cannot be done on a separate thread) can be done with Texture2D.LoadRawTextureData and is like 1ms in my tests. The decoding part can in theory be done on a separate thread but here's the problem: Any JPEG library that I found is a lot (x10) slower than the Unity's decoder. So I'm stuck here. I just wanted to let you know. Maybe you have another solution to this.

 

If nothing there is a new feature on the roadmap for 5.3 that can help:

Graphics: Async Texture Upload
A new feature that enables asynchronous loading of Texture Data from disk and enables time-sliced upload to GPU on the Render-thread. This reduces wait for texture GPU uploads in the main thread.

 

I've looked into it and your suggestions are sound. The major issue however is that it'd need platform native code that I've been avoiding.

Unity 4 and 5 both use FreeImage library internally to decode jpeg data streams. FreeImage in turn is built linking against libjpeg. Maybe UT has modified FreeImage or libjpeg, but I'm assuming that's not the case.

Therefore by calling Unity functions this plugin is ulitmately using libjpeg. There are more than 2x performant alternatives there like libjpeg-turbo which is used by major web browsers. It's designed to be compadible with libjpeg so I believe it'd be wise for Unity to switch to using FreeImage that's compiled against libjpeg-turbo instead of just libjpeg. I'm waiting for that day and that's why I haven't brought in platform native code to boost performance. Native code is always a hassle when supporting multiple build targets.

An interesting 5 year old and going FreeImage forum thread that can be summed up with the last post in there: "after all that years... is there any chance that libjpeg-turbo gets integrated by default into FreeImage?"
http://sourceforge.net/p/freeimage/discussion/36111/thread/fb51778d/

 

Wow! I didn't know any of this. Thanks for the reply

 

Hi,
I am looking for an avi player for Unity, and this plugin looks really promising as it supports the MJpeg compression. Currenlty I am recording a video using Opencv from my webcam. The video plays fine in vlc but is distorted in Quicktime. I am attaching the sample video with this post, could you please test it and let me know if it works fine with this plugin so that I can buy it.

Thanks
Andy.

 

bump!!

 

Does play, but it's "wide". The issue is that the video is interlaced 15fps, but is played back as 30fps non-interlaced using even and odd fields as full frames. This plugin doesn't support interlaced content (interlacing is now an awful remnant of analog TV era that should be avoided whenever possible, imho), but you could force it to be 4:3. Then the pixels wouldn't be square any more however.

 

Yesssss, buttery-smooth playback even at sizes of 2048x1024! Thanks for saving my bacon!

 

Cool, it's nice to hear that. That day may be soon when I can drop the safe 720p recommendation on desktops.

 

Hi,
I am willing to buy your plugin but I need to be sure if it will meet my need. I have a game where there is an interactive tutorial for players on the standalone build. However on the mobile platform (windows, android, ios) for the lack of screen space I can't function that tutorial properly. My goal is to make a video tutorial for mobile platform instead. So people can just watch the video and learn the game mechanics. Here is what I need your player to do to meet my need.

1. I would need it to fire an event at the end of the video, so I know that the player has finished watching the tutorial.
2. Can I manually build a "-30 sec" button? So player can go back 30 seconds to rewatch a part of video?
3. Will this player run in full screen on all three mobile platform(windows phone, android, iPhone)?
4. My game runs in portrait mode, so I also want the fullscreen video to run in portrait mode. Is that possible?

Please let me know your answer on these questions. And if all is "yes" I am more than happy to buy your plugin
Thanks

 

1,2,3,4 all possible, but it may not be the best use case for this plugin. High res mjpeg encoded clips will need larger files than more advanced video codecs for the same quality. On desktop it's not a problem, but on mobile it may be if the clips are long.

 

my tutorial will be only 3 min long. it needs to be 480p at the very least. What do you think would be the file size for this video?

 

Depends on the content, but I'd say not less than 100MB for a typical 3min clip, 480p at 30fps. If the content has identical video frames or the framerate can be reduced (maybe because each frame is hand drawn and there's a lot less of them than video frames) then the video file may become a lot smaller.

Best way to find out is to try what you've got and see the result. VLC can play it. Here's how to do it with FFmpeg

Code (csharp):

1. ffmpeg -i in.avi -vcodec mjpeg -qscale 4 -an out.avi

 

Thanks for your reply. I guess video tutorial is not really an option for me then. Thanks for your support nonetheless

 

Your asset seems very useful and includes most of the features that I am searching for: Video scrubbing and capturing frames from the clip.

What it seems to be lacking is the ability to shoot video on mobile (ipad, iphone) and use this video directly in an app.

Are you aware of a solution that would bridge this gap of converting video to mjpeg within my app, so that I could then utilize your asset?

 

I'm not aware of other plugins that can capture screen on mobiles so that it can be played back with this plugin. They use other video codecs that produce smaller files than mjpeg, but due to that they loose instant seeking possibility. Maybe, try searching "video capture" in Asset Store and see what's there. Only a few seem to be capable of even recording on mobile platforms.

The hard way to do screen capture is to script together Texture2D.ReadPixels for screen reading, Texture2D.EncodeToJpeg for encoding individual frames and then use AviRemux (from in this plugin) to put it all in an AVI container file. It'll probably work on desktop, but on mobiles it's more than likely to run into performance and file size problems.

 

Thanks for the quick reply. So far I haven't stumbled on a asset that would allow scrub and frame capture from video either. I am not planning on capturing the actual video via app but use system video camera for capture. From this video few frames would need to be extracted via scrub feature. Still few pieces of the puzzle are missing but your asset definitely is one of them.

If only there was an mobile ios asset for ffmpeg.

 

There's something out there
http://stackoverflow.com/questions/4725773/ffmpeg-on-android
http://stackoverflow.com/questions/21903789/add-ffmpeg-library-in-ios

But what would you do with ffmpeg, just convert a captured file into mjpeg or integrate it somehow? I'm just curious.

 

Thanks for digging up the ffmpeg/ios links. So far I've built ffmpeg libraries from sources on PC and dll can be used in the app without scrubbing. With ffmpeg saving the frames from video is easy.I am a bit concerned about the ios/ffmpeg combination as there might be unexpected issues in the long run making app support hard. Your asset is there fore very interesting. If only there was support/ability to use "normal" videos.

Usage would be pretty much as you described:
- take video with system video camera
- in the app choose the video
- scrub video for specific frames
- save chosen frames one by one
- use saved frame images in Unity.

All of this should be done in the device within reasonable time.

Have you considered creating another plugin for avi/mpeg -> mjpeg conversion?

 

Is the system video camera listed in WebCamTexture.devices? If so, then I can throw together a quick prototype to see if it can be efficiently captured to a mjpeg video directly, all inside unity. In fact earlier versions of this plugin could do uncompressed camera capture, but because Unity API couldn't be used to encode it back then, that functionality was dropped.

Using mpeg is a tricky business, not because it's technically difficult, but there is MPEG LA with its patent pool collecting royalties. Often end users don't need to care, but library/framework/engine writers do. That's why Unity uses Quicktime for importing videos and OGG/Theora for playback. Imho, it's best to stay clear from mpeg codec family. There are now license and quality-wise better alternatives to mpeg like VP9. It hasn't completely matured yet, but Google is behind it and many web browsers already support it too. Last time I checked it was as experimental codec in ffmpeg. One day I make time and bring VP9 to Unity as a plugin.

I like to keep things simple, no potential patent issues, no potential GPL poisoning issues, no potential royalty fee issues to whoever uses this plugin.

 

Judging by this it would seem to be: http://docs.unity3d.com/ScriptReference/WebCamTexture.html.

Need to give it a try later in the evening as well. Good point on licensing issues!

 

1st gen Nexus 7 tablet (2012) test. Video capture with Unity own API is not too satisfying, barely usable, but the tablet is rather old too. Newer ones may be better. On desktop this approach seems good. Used this script TempScreenCapture.cs

Camera capture resolution 640x480. Pixel copy from camera to texture 25ms, encoding 42ms, mjpeg encoded video frame size ~10..40k. Assuming optimistic 35% overhead we get max ~10fps capture framerate, ~250kB/s data rate.

Screen capture resolution 1280x720. Pixel copy from screen to texture 150ms, encoding 140ms. Practically unusable.

 

Nice work. Tried your script and on laptop no problems getting 30fps using 480p. With 720p frame rate is dropping closer to 15fps. Probably it will be almost impossible to get reliable 30fps on mobile even on newer ones.

I was playing little with coroutines. Haven't used them in any real scenario so not sure if they would help in this situation but what if the wct.GetPixels32() result would be put separate data structure (e.g. static List<Color32[]>) and then processed one by one in separate thread using StartCoroutine(coroutineFunction());. That would free the main thread to only capture video frames.

Some memory constraint is probably needed to limit how many frames can be queued but maybe it would work on shorter clips with decent frame rate?

 

Coroutines won't help, they do code scheduling within main Unity thread. A non-native code thread won't help either, because it can't call anything from UnityEngine namespace and all the processing needs to be coded in managed C# which is multiple times slower than code in UnityEngine namespace (which is written mostly in C++ afaik). For performance the encoding part and maybe the capturing part too has to be done outside Unity in a native code library (or separate process). That library in turn could use hardware acceleration if that's available. Native code and especially hw acceleration would either kill portability between build targets or cause a lot of code maintenance overhead, because each platform needs it's own native library. For one build target it's not too bad, but supporting many is a challenge

For your case, if the device has a lot of RAM, then you could maybe capture a thousand textures without encoding or encapsulating within a container format. Then just draw them to make a movie or encode afterwards which would take longer than capture. This is of course if all is done within Unity and without touching native code. With native code, especially if optimized for specific device, one can do wonders. I could take and make it work, but it'd be separate work from this plugin and it'll be more than a few days.

Sorry for long rant

 

Didn't sound like a rant to me

http://answers.unity3d.com/questions/280597/new-thread-vs-startcoroutine.html supports your comments on threads vs coroutines. Probably starting separate threads might help a little but not enough to give performance needed. For video encoding/decoding the native plugin is most likely needed.

For my own needs in the pc environment ffmpeg is enough to get started as it is enough to convert few seconds of video to still pictures and this I already have in my Unity project. Main target platform is ios and UNITY_IOS hooking is needed. AVFoundation is providing necessary apis but this is still academic discussion since I don't have mac yet to build these

After I have the stills captured from the video could your plugin be used for scrubbing and choosing specific ones?

Really appreciate your valuable comments and advice!

 

If you've got a thousand pictures, then it won't do, it has to be one file. But a good thing is that if you can't encode it to mjpeg in avi, then just concatenating jpg picture files will create a file that is close enough to elementary mjpeg video stream, and this plugin is able to load it. Since that concatenated file doesn't contain video metadata, you may want to override default framerate and video aspect ratio from your scripts. But even if you don't do that, you can still play back the video. All of it works on iOS.

Code (csharp):

1. // That's how to override video metadata after the clip has loaded
2. var vsi = GetComponent<MoviePlayer>().movie.demux.videoStreamInfo;
3. vsi.height = 1280;
4. vsi.width = 720;
5. vsi.framerate = 25;
6.  
7. // but it's possible to set these before loading the clip like this:
8. var vsi = ...;
9. GetComponent<MoviePlayer>().Load(..., new LoadOptions { videoStreamInfo = vsi });
10.  
11. // That's how to access currently visible video frame Texture2D
12. Texture2D fb = GetComponent<MoviePlayer>().framebuffer;

Please take a look at this video about how to make a seek bar too

 

Thanks again for info and tutorial. The seek speed of your plugin is impressive!

Not really happy about the idea of creating a platform specific plugin but that is most likely needed. Would be optimal to have it out of the box in Unity (with your plugin of course ). I am currently doing more or less what you are suggesting above and as the number of frames will not be too big it shouldn't be an issue (10s * 30fps), even the total size of the files should be reasonable with 1080p.

For end user it not optimal either as the position of the video file needs to be located via system camera app outside of my app. Maybe later on that needs to be resolved (and it is not a big deal really as there are other issues to tackle too).

 

Just to document here if someone finds it useful regarding IOS video frames to stills. Unity's IOS Native asset provides Gallery access (among other things) to retrieve video clip. After finding correct location in video file AVFoundation can be used retrieve frames at specific times. Haven't used IOS AVAssetImageGenerator but seems straightforward.

Then only thing left to do then is to present user with ability to choose correct frames e.g. via Movie Player asset.

 

What you're doing should be simpler, but it can't be helped. I hope it'll all play nicely along in the end. Let me know if there's anything I can do or how it went

 

@ShuuGames Do you know if there is a way to change Playback speed? I'm trying to implement a rewind/fast-forward feature, and was hoping that there would be some type of built-in functionality.

 

For this there is moviePlayer.framerate that you can change. It has to be set in script.

 

@ShuuGames I was trying to access that in a script, but it said that was a read-only index accessor. Do you know is there another way to change the framerate?

 

It used to be read only. In latest version in Asset Store it can be written too. If updating is not an option, then you can use this method. It does exactly what the framerate setter does, written as a function:

Code (csharp):

1. void ChangeFramerate(MoviePlayer mp, float newFramerate) {
2.   if (mp.movie != null && mp.movie.demux != null && mp.movie.demux.videoStreamInfo != null) {
3.     mp.videoTime = newFramerate == 0 ? 0 : mp.videoTime * mp.framerate / newFramerate;
4.     mp.movie.demux.videoStreamInfo.framerate = newFramerate;
5.   }
6. }
7.  

 

@ShuuGames Thanks for the info. I'll look at the update and if not, I'll implement your method. Do you know if this would work for rewinding, or would that require different logic?

 

Ok, it looks like I figured out rewind. I added a bool to MoviePlayer called isReverse, then in HandlePlayheadMove() I subtracted the time if isReverse is true. Then in HandleLoop() I added a check that loops if they reach the beginning of the video and sends it to the end of the video if isReverse is true. That worked perfectly. Thanks for including the source. That really helps with extending the functionality.

 

I'm glad it worked out. That's the beauty of full source. If you've made changes to plugin sources, then please be careful when updating this asset from Asset Store later on or you could send me the changes an I'll integrate them into the main code tree