Extremely poor WebcamTexture performances under Android in high resolution

Hello,

Video acquisition via WebCamTexture is extremely slow (less than 3 FPS) in 1920*1080 on a Samsung Galaxy S4 Android 4.3, which should be capable of better performances...
Any idea guys to improve it? I'm under Unity 4.3.
I've tried to play with different _requestedFPS but it didn't help.
Here is the little code I've used for testing:

Code (csharp):

1. using UnityEngine;
2. using System.Collections;
3.  
4. public class FrontCameraHD : MonoBehaviour {
5.  
6.     [Range (0,5)]
7.     public int _webcamId = 1;
8.     public int _requestedWidth = 1920;
9.     public int _requestedHeight = 1080;
10.     public int _requestedFPS = 15;
11.     public GUISkin _debugSkin;
12.  
13.     private float _minFPS, _maxFPS;
14.  
15.     private WebCamTexture _wct;
16.  
17.     // Use this for initialization
18.     void Start () {
19.         Screen.sleepTimeout = SleepTimeout.NeverSleep;
20.         _wct = new WebCamTexture(WebCamTexture.devices[Mathf.Min(_webcamId,WebCamTexture.devices.Length-1)].name, _requestedWidth, _requestedHeight, _requestedFPS);
21.         _wct.Play();
22.         _minFPS = 1000;
23.         _maxFPS = -1;
24.     }
25.  
26.     // OnGUI
27.     void OnGUI () {
28.         GUI.DrawTexture(new Rect(0, 0, Screen.width/2, Screen.height/2), _wct, ScaleMode.StretchToFill, true, 1.0f);
29.         float fps = 1.0f/Time.deltaTime;
30.         if (fps < _minFPS) _minFPS = fps;
31.         if (fps > _maxFPS) _maxFPS = fps;
32.         string s = "FPS: " + fps + "\nMin: " + _minFPS + "\nMax: " + _maxFPS+ "\nRes: " + _wct.width + " * " +_wct.height;
33.         GUI.skin = _debugSkin;
34.         GUI.Box(new Rect(Screen.width/2, 0, Screen.width/2, Screen.height/2), s);
35.     }
36. }

 

Thanks for your answer, I've started to take a look on the GL_TEXTURE_EXTERNAL_OES documentation and it doesn't look that easy to implement. What I really need is not to render the texture into Unity but to get the R, G , B bytes array of the camera for some processing, if you happen to have any hint for making it.
For Unity people information: yesterday I've reported this issue as a bug, case 584816.

 

Thank you very much for the hint, this is exactly what I was looking for!

 

I read the post and found it very interesting since I'm trying to do the same thing (capture camera pixels efficiently). Could someone please details a little bit more (since I'm new to Android development) on how to use this setPreviewCallbackWithBuffer function with Unity plugin?

Thanks a lot!

 

We have run into the same poor performance issue webcamTexture exhibits on Android...it pretty much destroys any chance of our game succeeding on Android, which is a big deal for us. You can try it out for yourself - https://play.google.com/store/apps/details?id=leagueofmonkeys.knifegunbomb

What we don't understand is how Vuforia a third party plugin can succeed in delivering a smooth webcam feed(plus analysis for tracking) of resolution up to 720p when Unity's own implementation performs so poorly.

Qualcomm is listed as one of Unity's partners, why not ask them how they did it?

 

bump...

 

bump......

 

bump.........

 

Bump............

 

same in unity 5. I am starting to lose faith in this platform now.

 

globalillumination: They're probably busy working on new features that they can market rather than fixing flawed existing features

 

I've just started playing with this on my note 4, but I found that if I use the default constructor (WebCamTexture()), performance is significantly improved. Not sure if that's high-resolution or not, but it at least appears to be identical to the primary camera app...

 

I wonder if it is because a faster device. Since on my Samsung S3 i am having pretty low performance on webcam texture.
As @bitter kindly noted writing a plugin that copies the camera feed directly to GPU and getting the id of that texture seems most performant way.
But i have been reading lots (and lots) of info on this matter and i am not even close to building a plugin with my knowledge.
Any help would be appreciated.
(Please don't point to Vuforia. It is not "free")

 

bump......

 

New version of Unity 5.2.0f3 was released.
But the bug still there.
I am starting to lose faith in this platform.

 

Hate to revive a dead thread, but what if I packaged my code to the UAssetStore, would any of you be interested? I made a wrapper around iOS and Android's native cameras. It supports autofocus, tap to focus, flashlight, high resolution photos, and video capture (video capture is incomplete). And this you'll love:

On my S3, I get between 23-30FPS Live camera preview. It's done with GPU processing native side.

Oh and as an added goodie, it supports barcode detection on both platforms.

 

Would be very nice!

 

Hi Lanre, Yes please

 

Lanre, that would be incredible. I'm currently working on a wrapper for Android myself, and it's really slow going. I'm doing some pretty heavy image processing with an OpenCV Unity plugin. Coupled with the poor WebCamTexture performance, that portion of our game is essentially unusable on those devices.

If you decide to go through with it, do you have a general timeframe in mind for video capture? It would be great to avoid this Android issue and focus more time on optimizing and fine tuning gameplay as our release date approaches. Meanwhile, I'll continue my tinkering.

...Also, if you throw in exposure control, I'll totally be your best friend.

 

Hey guys. Love the feedback. I'll go on ahead and finish it. I hope 2-3 weeks isn't too long a wait time; I've got a huge time-sensitive project on hand (that uses NatCam). Video capture could be done, I mentioned that it's incomplete because I only got it running on iOS before I became occupied with the current project. As for your OpenCV needs, I'll be sure to add a little helper method that does Blits (like RenderTexture's Blit, but this time it'll be an extension method for Texture2D) so that you could do GPU processing if you wish to.

I'll look into Exposure Control. Might or might not make it into 1.0.

[EDIT]
Ohh and concerning video capture, retrieving the mp4 or mov might be an issue. If you'd like to playback the recorded video, you'll have to get an asset like MovieTexture for mobile or so. I don't intend to build that functionality into NatCam, although I might consider it.

 

a "shut up and take my money" meme would fit here probably (don't mean you to shut up...please continue to share info...i...er..oh well)

Yes please...
Need beta tester?

 

Thanks for the quick response, Lanre! Just realized I've mixed up the semantics regarding video capture, all I need personally is the camera preview, not any recording. All my project depends on is functionality similar to WebCamTexture, just with better performance on Android. That is, play/pause and retrieving the image data as a Texture2D. The OpenCV plugin I have is "OpenCV for Unity," which has a utility for converting a Texture2D to an OpenCV matrix. Of course, if you feel like getting fancy, don't let me stop ya!

Again, thanks for considering all my feedback, and best of luck with your other project.

 

Hi Bunderant. That's better for me in fact. Video capture would have taken extra time. Can I have you as a Beta tester? I say this because I need to know if Texture2D to OpenCV Mat will work. I discovered a bug with Unity's external texture API (CreateExternalTexture, UpdateExternalTexture) that doesn't allow any reading or writing on the external texture. This means that GetPixel(s), SetPixels, ReadPixels, and EncodeToJPG or PNG won't work on the camera preview. For still image capture, I made a helper function that goes around this problem (because it's a one-shot process). But or processing every frame, this will certainly be a problem.

I'm opening a new forum thread so that Unity Devs know of this. Fingers crossed, Texture2D to OpenCV Mat works. And Bariscigal, yes I'll have you as a Beta tester. Shoot me your email address.

 

iOS done. Hopefully will finish Android tomorrow. Then share to betas then submit to Asset Store

 

Hey, Lanre. I'd be more than happy to test that out for you, and I'll let you know if I have any ideas for a workaround on the external texture issues. I'll send ya my email address. Thanks!

 

Im also interested, any pricepoint in mind Lanre?

 

Hi 128bit. I'll be putting it at about $40. I'm getting NatCam to work with Android's multithreaded rendering right now

EDIT
Works

 

Hi Lanre, I'm interested to test it and buy it as well when it's out. A little question, will you provide the source code ?

 

Hi Packedbox. Unfortunately, the beta is closed. And yes of course all sources are included, it won't make sense to create such a solution without allowing customization. NatCam will be submitted for Asset Store review by Tuesday.

 

I got acess to the beta some days ago. Honestly, tested it on several devices. Did worked on each without any issues. Performance is spot on. Even on low end devices!

 

Sounds promising!

 

Yes in fact it is. We're prepping ourselves for release. And I personally can't wait to see how NatCam gets modified and improved. I'm writing a brief explanation file for the sources so that it won't look so intimidating to anyone. Soon guys, soon

 

Hi Lanre, thanks for he update can't wait to try it.

 

Hey Guys! I started a new thread for NatCam. Please put your questions, concerns, and feature requests there. It's in for Asset Store review. I expect it to be out no later than the 5th of January considering the holidays. Here's the link: http://forum.unity3d.com/threads/natcam-device-camera-api.374690/

 

Hello everyone,

I had terrible issues using WebCamTexture even using default constructor. I solved it enabling the option "Auto Graphics API" in Player settings, specifically in "Other Settings". I don't know if it is the root cause for all of you but it is works for me. See attached image.

 

Thank you ChristianBG! This gave me a huge improvement. When I enabled Auto Graphics API, it gave me some warnings about other settings, but I squelched those by changing Color Space to Gamma and Lightmap Encoding to Low Quality with no ill side effects for my project.