Mobile Movie Texture

DTreble. Is there any chance that you could support sound also?

 

Did you miss these lines in the instructions:

Please move the Plugins directory from here Assets/MobileMovieTexture/Plugins to be in the root Assets/ directory
Also move the StreamingAssets directory movies in Assets/MobileMovieTexture/StreamingAssets to be in the root Assets/ directory
​

Thanks for buying it!

The asset store wont let me put the Plugins or Streaming Assets in the right place :-(

 

is it possible to have a 1.1.2 demo version. The attached file in posting #1 is still version 1.1.1. Would like to test the keying quality before I buy it.

 

I think the watermarking in the demo version will mess up the quality of the keying, I don't think it will do it justice.

The keying works by doing a weighted distance in YCrCb space and using that as the alpha channel. You can set the top and bottom of that distance as the high and low values.

The shader has no branching logic at all. So it should be as fast as possible.

Still trying to get 1.1.2 accepted in the store :-(

 

Ok i'll see. Hopefully it won't take long till your update is online!
BTW are future updates included when I'll buy the current version?

 

If you PM me your invoice number, I'll send the latest version straight to you

Yup

 

I have a project where I use the demo-package in. Now that I've purchased the full plugin, I thought I simply import the full plugin and let it overwrite the old (demo) one.

But now I get an error saying:
In Unity:

Code (csharp):

1.  
2. Assets/MobileMovieTexture/Editor/ChromaKeyEditor.cs(46,25): warning CS0219: The variable `window' is assigned but its value is never used
3.  

In XCode:

Code (csharp):

1. "_OpenStream", referenced from:
2. __Z19RegisterMonoModulesv in RegisterMonoModules.o
3.      (maybe you meant: _OpenStreamDemo)
4. ld: symbol(s) not found for architecture armv7
5. clang: error: linker command failed with exit code 1 (use -v to see invocation)

How to fix this? Thanks.

 

Code (csharp):

1. Assets/MobileMovieTexture/Editor/ChromaKeyEditor.cs(46,25): warning CS0219: The variable `window' is assigned but its value is never used

I'll fix that warning. Sorry!

Also how did you get that? That is in version 1.1.2 which was rejected from the asset store as the reviewer didn't move the streaming assets directory :-(

Sent you a PM about the other problem.

Thanks for buying it!

 

Well, we now get a whole bunch of other errors.. Don't know what it is yet. I'm having a programmer looking into it right now. The ones I mentioned above are fixed.
Thank you for the quick answers. I was just a little bit going to fast and therefore missing some things.

 

Hi,

Do you have any plans to support audio?
This looks very interesting

 

I haven't tried yet. I have a feeling keeping audio in sync with the movie will be tough. Especially on android where audio latency in mixing buffers can vary hugely.

The movie is synced to Time.delta time, so in theory you could start the audio playing ~170 ms before the video.

Open to suggestions

 

We just bought your plugin from the Asset Store. Thanks for developing this
I hope you find a way to include audio at some point.

 

I have another quick question
Is it possible to use the movie texture on UnityGUI components such as a label or a drawtexture? That would be very handy.
If I understand the setup correctly, it's the movie shader that determines where the movie is played. I guess that makes it impossible to use it with UnityGUI, right?

 

... and a couple of additional feature requests:

- play the movie backwards (a variable we can toggle)
- jump to a specific frame and play

 

I don't think so as it needs the custom shader to convert the colors. It also uses a custom UV matrix to crop the movie.

That is exceptionally hard with the way these movies are encoded. They have a keyframe, then many delta frames that have to be played sequentially. I doubt the delta frames can be applied in reverse.

Tricky for the same reason above. Say you want frame 75, I have to find the last key frame, say it was frame 50, I then have to decode/play/apply 25 delta frames to get to 75. That is going to be one hell of a performance spike when it seeks!

Thanks again for buying it! I'm sorry all your suggestions are hard

 

Hi DTreble,

First of all, thanks for developing this life-saver! We had a big problem playing movies from an AR application on Android, using Unity's VideoPlayer: after the movie was finished, it took some 5 to 10 seconds for the application to return to the AR on some devices (e.g. 7" Galaxy Tab). First five seconds you could already hear the music from the AR playing, with the last frame of the video in view. Then 5 seconds of black... At first we though this was due to the QCAR tracking becoming active immediately, but later on we found that it was probably caused by the loss of the OpenGL Context which is mandatory on Android 2.x. In other words: the entire scene needed to be loaded into GPU memory again. This would explain why switching from a small 3D scene and back again is really quick, but from/to a big scene is not.

Ultimately we purchased your component, positioned a quad in front of the camera (with accelerometer-readout for orientation) and started an audio stream in parallel. When the framerate and resolution are kept moderately low (18 fps, 320x480), the result is good enough.

We used it for the 'interactive theatre poster' for the Rain Man play by a Dutch theatre group called De Utrechtse Spelen. A video of it is here: http://www.youtube.com/watch?v=ry0JDrx1AtE (made by the ANP, a Dutch news agency).

We also used it in an AR production for the Pre-launch campaign of the Sony Playstation Vita for Vodafone Germany https://market.android.com/details?id=de.vodafone.VodafoneAR, which for various reasons hasn't quite gotten off the ground yet. We hope to refurbish the app for the real Launch campaign, since nobody has seen it yet and it's really cool (app only available in Germany btw). In any event, you can see that your component is being put to good use!

Having said all that, please let me take the opportunity to join the chorus of voices asking for audio-support, and the ability to skip frames should the device not be able to keep up! Can you give some feedback on your plans for this?

All the best,
Lex van der Sluijs
creative technologist, twnkls | augmented reality

 

Hi,
Is there a way to test the 1.1.2 ? I'm in a project that could be very useful the chroma key feature, let me know, so I can make a decision on using it (buying) or trie to find other solution.
Tks
@mob

 

Hey TheLex, thanks for sharing your experiences. The projects look cool.

Noted

I put the ground work in the 1.1.3 version. As all the frames are deltas based on the last, you still have to decode the frame, but you can skip the upload to the gpu and the decode.

 

Shader and shader editor editor attached, it's up to you to hook them up! I will get round to updating the demo version at some point. Right now my focus is on getting the asset store to accept version 1.1.3 after they rejected 1.1.2.

 

Tks I will try it
Tks
@mob

 

Hello, very interseting!
Your library playing video files (coding by theora) from any place (I mean if place file not in StreamingAssets) or not on all platform?

Thanks!

 

Has there been any tests done on 3.5?

 

Internally it does a fopen, if you can provide a file path it can open, it will work.

Hope that helps

 

I've been running 3.5 for a while with no problems.

 

Excellent, I was JUST about to reply that i'd tested with the demo on 3.5 on iPad2 and it worked exactly how I needed it to - multiple movies, no hassle. It just worked.

This, after trying Prime31's plugin, was such a relief. Mike over at 31 was fantastic in trying to help trouble shoot, but it never did work for me. Yours works great!

One request so far: can you add a permanent getter for the movieMaterial private?

Code (csharp):

1. public Material MovieMaterial
2. {
3.     get
4.     {
5.         return m_movieMaterial;
6.     }
7. }

I added this on my end, but I know that'll get wiped out when a new rev comes out

Thanks very much for your hard work, this really saved our @$$

 

On the sample videos I converted from mov to ogv, I'm getting this green line that's not always there on playback - there are 3 videos playing, the one in the middle is being keyed:



Playing the video back in VLC, I can see a thin green line at the bottom:



Is this green showing up because of encoding settings? or is it to odd aspect ratio? Or ... ?

Thanks for your help! OUTSTANDING that I can run it in the IDE - ROCKS.

 

Excellent glad to hear that, nice to get such positive feedback.

Consider it done

Thanks for buying it!

Interesting. I was going to blame bilinear filtering in my implementation till I saw the VLC shot. Looks more like a bug with the encoder.

Our of interest, what did you use to convert it to theora, http://v2v.cc/~j/ffmpeg2theora/? The stable version has been solid for me, I don't trust the beta though, it gave me some weird frame skips.

What is your frame width x height? Internally, it rounds the width and height to multiples of 16. The decode buffer then has a 16 pixel buffer top,bottom,right and left. Definitely try a multiple of 16 on the height.

Ha ha, glad you appreciate it. Speeds up my workflow if I don't have to deploy to device.

How are you finding the keying?

 

Hello!
Thanks for answer!!!

Yes, it really work on all platforms except Android!!!
All time I have exception in AssetStream.cs when trying to open file (file mnt/sdcard/Android/Data/com.test.app/files/video.ogg not found, but file present!!!)

This line throws exception: assetManager.Call<AndroidJavaObject>("openFd", fileName))...

in Android video file places in "mnt/sdcard/Android/Data/com.test.app/files/video.ogg"

Maybe you know what happens?

It work under WEB?

Thanks!

 

Wondering if sound in .mov files would be supported at some point or is it just something I need a separate track for?

 

I downloaded a plugin for quicktime pro and it seemed to have done the job, with the exception of that line. I will try the one you pasted here. I'm on mac, I could have sworn I saw that link in reference to windows, so i'll try and let you know.

The key worked great - really great. I did have a moment in the IDE that I can't fully described, but it seemed for a minute that my shader selection kept switching back to keyed version despite my setting it to non keyed. I think selecting the material in the project panel, then changing it there made the change "stick", but I had to wonder if it was because I added permanent panel for the key editor? Anyway, it worked great as you can see from that screenshot.

Thanks for the heads up on the multiples of 16, i'll definitely keep that in mind. I had created those videos using Jing's video capture, so my screen selection was willynilly

 

For those who like a little GUI with your command line on Mac for ogv conversion, this worked great:
http://taylanpince.com/blog/posts/oggifier-gui-wrapper-ffmpeg2theora/

seems to have cleared up those green lines just fine!



Thanks again!

 

I haven't tried it yet, but is there any reason I can't use the keyed shader with a normal MovieTexture for non iOS apps?

 

You probably want to change the android section in this MobileMovieTexture.Open(). Android is hard coded to read back from a offset within the apk file. You don't want that, really your want to pass your path un modified through to the native OpenStream

So try replacing this

Code (csharp):

1.  
2. #if UNITY_ANDROID  !UNITY_EDITOR
3. <snip>
4.         m_nativeContext = OpenStream(AssetStream.GetApkPath(), (int)offset, (int)length, m_loop, pot );
5. #elif UNITY_IPHONE  !UNITY_EDITOR
6.  

With this

Code (csharp):

1. #if UNITY_ANDROID  !UNITY_EDITOR
2. <snip>
3.         m_nativeContext = OpenStream(m_path, 0, 0, m_loop, pot );
4. #elif UNITY_IPHONE  !UNITY_EDITOR

Sorry, it doesn't. However maybe the built in MovieTexture works under web?

 

I'm not sure how best to tackle this, but I know for sure that making them sync will be a nightmare.

I have to account for the size of the fmod mixing buffer (x ms) then on android the kernel mixing buffer (40ms to 170ms). The audio will need to be way ahead of the video. Which means the audio will have to be triggered separately (which you can try now) or I'd internally have to buffer video frames, decode the audio with a second copy of vorbis and pipe that out and through unity :-(

 

Glad the keying worked well. Not sure what was happening with your shader not sticking. Could it have been a "Save Project" thing?

Shouldn't be, it's not the best code, but all it really does is change shader values in the material you have selected, if it is using the chroma key shader.

I wouldn't worry about the 16 too much, if you now have an encoder that works

 

The chroma key shader works in YCrCb space and ends up in RGB, where as MovieTexture is RGB out. Wouldn't be too hard to do another version that worked for MovieTexture

 

On a side note, what's the performance on typical ios devices? is 60fps a pipe dream? I ask because if it works ok at 60fps, it's possible to get the rest of the game working fine and scale the whole thing down to 30. Just looking for a yardstick really.

 

In my tests, the 3GS runs the demo movie, 640 x 380 at slightly higher than 30fps. Note it isn't doing anything else. One of my customers is doing 1024x768 on the iPad2. I personally haven't tried to hit 60 on a device yet.

Give the demo a go. That's what it is for, it has the same performance characteristics as the paid one. Wont take too long to see if it solves your problem or not

 

One other suggestion: can you wrap an if statement around the clearing of the channelTextures indexes so that this class can co-exist when compiling/running in other build targets?

Thanks again!

Code (csharp):

1. private void DestroyTextures()
2.     {
3. #if (UNITY_ANDROID || UNITY_IPHONE)  !UNITY_EDITOR
4.         m_reallocTextures = false;
5. #endif
6.         if( m_ChannelTextures != null )
7.         {
8.                 m_ChannelTextures[0].Destroy();
9.                 m_ChannelTextures[1].Destroy();
10.                 m_ChannelTextures[2].Destroy();
11.         }
12.  
13.         m_ChannelTextures = null;
14.     }

 

Sorry, you lost me. What breaks?

Destroy textures looks like this in 1.1.3

Code (csharp):

1.     private void DestroyTextures()
2.     {
3. #if (UNITY_ANDROID || UNITY_IPHONE)  !UNITY_EDITOR
4.         m_reallocTextures = false;
5. #endif
6.         if (m_ChannelTextures != null)
7.         {
8.             for( int i = 0; i < CHANNELS; ++i)
9.             {
10.                 if(m_ChannelTextures[i] != null)
11.                 {
12.                     m_ChannelTextures[i].Destroy();
13.                 }
14.             }
15.         }
16.        
17.         m_ChannelTextures = null;
18.     }

What wants changing?

Thanks! Dan

 

i hadn't realized that thats in the DEMO version, and it's different than the final so never mind!

 

I do have a question about skipping frames, and I thought I'd read a post about it - right now it seems like its playing every frame in the iPad whether it can keep up realtime or not. In your code, it *looks* like you're testing for time, but I'm wondering - do you drop frames to keep up with realtime? Is that doable? I ask because I'm trying to play audio with it. When playing in the IDE and it can keep up, it actually works "good enough". but on iPad, my scene is rather heavy and the video playback is slower than realtime.

Can you let me know what to do to have it drop frames and keep up with realtime?

Thanks much!

 

Ok, it remains untested, but if you're feeling brave, you can pass -1 as the texture handles to DecodeAndUpload() it will just decode and not upload. You can't avoid decoding the frame, they are all deltas. However saving the upload should save a lot of work.

So the decode loop logic would look something like

Code (csharp):

1. double nextFrameTime = GetNextFrameTime(m_nativeContext);
2.  
3. while( nextFrameTime <  (m_elapsedTime - Time.deltaTime))
4. {
5.     DecodeAndUpload(m_nativeContext, -1, -1 ,-1); //Skip frames
6.     nextFrameTime = GetNextFrameTime(m_nativeContext);
7. }
8.  
9. //Do the actual frame
10. DecodeAndUpload(m_nativeContext, m_ChannelTextures[0].tex.GetNativeTextureID(), m_ChannelTextures[1].tex.GetNativeTextureID(), m_ChannelTextures[2].tex.GetNativeTextureID()  );

I do plan on getting round to this at sometime myself

 

I apply Movie material to my prefab. I also have a game object with attached MobileMovieTexture.cs and TestMobileTexture.cs
I have this code inside TestMobileTexture.cs

Code (csharp):

1. if (GUI.Button(new Rect(0, 0, 100, 100), m_movieTexture.isPlaying ? "Pause" : "Play" ))
2. {
3.    m_movieTexture.pause = !m_movieTexture.pause;
4.    if(!m_movieTexture.pause)
5.    {
6.       m_movieTexture.Play();
7.    }
8. }

When I call m_movieTexture.Play(); texture in my prefab freezes. If I drag and drop that material runtime again - it works ok.
Could you help me with that?

 

yeah I just ran across this today - I think the issue is that when you use it on a prefab, the material that comes with the prefab is an instance of the material, not the material itself that you assigned to MobileMovieMaterial.

All you have to do is assign the material of the renderer to the MMT component at Start. I literally just put this on my prefab and it takes care of it:

Code (csharp):

1.  
2. using UnityEngine;
3. using System.Collections;
4.  
5. public class SetMobileMovieTexture : MonoBehaviour
6. {
7.     public MobileMovieTexture mmt;
8.    
9.     void Start ()
10.     {
11.         mmt = GetComponent<MobileMovieTexture>();
12.         // this has to happen because this is created as a prefab at runtime and the material is instanced.
13.         mmt.MovieMaterial = renderer.material;
14.     }
15. }
16.  

I was JUST going to ask if we could add the getter/setter I've added to the code base, and you'll have to add it for this to work:

Code (csharp):

1.  
2. public Material MovieMaterial
3.     {
4.         get
5.         {
6.             return m_movieMaterial;
7.         }
8.         set
9.         {
10.             m_movieMaterial = value;
11.         }
12.     }
13.  

hth

 

Cool, it works! Thanks man!

 

Yes please!

 

Thanks for doing my support for me neoriley!

 

thanks for putting in my getter/setter requests

 

Hi,

Just bought, and until now is great, and good performance
I'm testing with a chroma key video that I had, but is a green chroma color, I have been struggling to find or match the Key Color YCrCb values for that.
Is there any standard values, or a color pickup way to get the right values ?
Tks for the great piece of software.
atmob