Search Unity

Extremely poor WebcamTexture performances under Android in high resolution

Discussion in 'Android' started by DeltaCygniLabs, Jan 10, 2014.

  1. DeltaCygniLabs

    DeltaCygniLabs

    Joined:
    Nov 27, 2013
    Posts:
    14
    Hello,

    Video acquisition via WebCamTexture is extremely slow (less than 3 FPS) in 1920*1080 on a Samsung Galaxy S4 Android 4.3, which should be capable of better performances...
    Any idea guys to improve it? I'm under Unity 4.3.
    I've tried to play with different _requestedFPS but it didn't help.
    Here is the little code I've used for testing:

    Code (csharp):
    1. using UnityEngine;
    2. using System.Collections;
    3.  
    4. public class FrontCameraHD : MonoBehaviour {
    5.  
    6.     [Range (0,5)]
    7.     public int _webcamId = 1;
    8.     public int _requestedWidth = 1920;
    9.     public int _requestedHeight = 1080;
    10.     public int _requestedFPS = 15;
    11.     public GUISkin _debugSkin;
    12.  
    13.     private float _minFPS, _maxFPS;
    14.  
    15.     private WebCamTexture _wct;
    16.  
    17.     // Use this for initialization
    18.     void Start () {
    19.         Screen.sleepTimeout = SleepTimeout.NeverSleep;
    20.         _wct = new WebCamTexture(WebCamTexture.devices[Mathf.Min(_webcamId,WebCamTexture.devices.Length-1)].name, _requestedWidth, _requestedHeight, _requestedFPS);
    21.         _wct.Play();
    22.         _minFPS = 1000;
    23.         _maxFPS = -1;
    24.     }
    25.  
    26.     // OnGUI
    27.     void OnGUI () {
    28.         GUI.DrawTexture(new Rect(0, 0, Screen.width/2, Screen.height/2), _wct, ScaleMode.StretchToFill, true, 1.0f);
    29.         float fps = 1.0f/Time.deltaTime;
    30.         if (fps < _minFPS) _minFPS = fps;
    31.         if (fps > _maxFPS) _maxFPS = fps;
    32.         string s = "FPS: " + fps + "\nMin: " + _minFPS + "\nMax: " + _maxFPS+ "\nRes: " + _wct.width + " * " +_wct.height;
    33.         GUI.skin = _debugSkin;
    34.         GUI.Box(new Rect(Screen.width/2, 0, Screen.width/2, Screen.height/2), s);
    35.     }
    36. }
     
  2. bitter

    bitter

    Unity Technologies

    Joined:
    Jan 11, 2012
    Posts:
    528
    The current implementation isn't very performant. 3fps for grabbing a 1920*1080 camera feed is probably the best you will get. Basically every frame have to be decoded in full into system memory by the cpu and then uploaded as a texture to the GPU, and on top of that you get the overhead of actual Camera implementation in Android. There is an optimized way of rendering a camera preview feed on Android but that doesn't work on Gingerbread and it requires you to use GL_TEXTURE_EXTERNAL_OES which means you need custom shaders etc for sampling the texture.

    There are some things we can do to enhance performance but for now you basically need to write your own plugin to get good camera feed performance for high resolution video.
     
  3. DeltaCygniLabs

    DeltaCygniLabs

    Joined:
    Nov 27, 2013
    Posts:
    14
    Thanks for your answer, I've started to take a look on the GL_TEXTURE_EXTERNAL_OES documentation and it doesn't look that easy to implement. What I really need is not to render the texture into Unity but to get the R, G , B bytes array of the camera for some processing, if you happen to have any hint for making it.
    For Unity people information: yesterday I've reported this issue as a bug, case 584816.
     
  4. bitter

    bitter

    Unity Technologies

    Joined:
    Jan 11, 2012
    Posts:
    528
    Ah, ok.. but if you need the raw data you won't be helped by rendering directly into a texture.

    I would suggest writing a java plugin that use the setPreviewCallbackWithBuffer(Camera.PreviewCallback cb) function to grab data from the camera. That way you can distil the data before handing it over to Unity/your game code.

    And thanks for the bug report ;0)
     
    DeltaCygniLabs likes this.
  5. DeltaCygniLabs

    DeltaCygniLabs

    Joined:
    Nov 27, 2013
    Posts:
    14
    Thank you very much for the hint, this is exactly what I was looking for!
     
  6. kikoland

    kikoland

    Joined:
    Nov 4, 2013
    Posts:
    7
    I read the post and found it very interesting since I'm trying to do the same thing (capture camera pixels efficiently). Could someone please details a little bit more (since I'm new to Android development) on how to use this setPreviewCallbackWithBuffer function with Unity plugin?

    Thanks a lot!
     
  7. exerion

    exerion

    Joined:
    Jun 29, 2011
    Posts:
    54
    We have run into the same poor performance issue webcamTexture exhibits on Android...it pretty much destroys any chance of our game succeeding on Android, which is a big deal for us. You can try it out for yourself - https://play.google.com/store/apps/details?id=leagueofmonkeys.knifegunbomb

    What we don't understand is how Vuforia a third party plugin can succeed in delivering a smooth webcam feed(plus analysis for tracking) of resolution up to 720p when Unity's own implementation performs so poorly.

    Qualcomm is listed as one of Unity's partners, why not ask them how they did it?
     
  8. exerion

    exerion

    Joined:
    Jun 29, 2011
    Posts:
    54
    bump...
     
  9. exerion

    exerion

    Joined:
    Jun 29, 2011
    Posts:
    54
    bump......
     
  10. exerion

    exerion

    Joined:
    Jun 29, 2011
    Posts:
    54
    bump.........
     
  11. exerion

    exerion

    Joined:
    Jun 29, 2011
    Posts:
    54
    Bump............
     
  12. globalillumination

    globalillumination

    Joined:
    Feb 2, 2015
    Posts:
    1
    same in unity 5. I am starting to lose faith in this platform now.
     
  13. exerion

    exerion

    Joined:
    Jun 29, 2011
    Posts:
    54
    globalillumination: They're probably busy working on new features that they can market rather than fixing flawed existing features :(
     
    TheGameLearner likes this.
  14. michaelgburton

    michaelgburton

    Joined:
    Sep 19, 2014
    Posts:
    4
    I've just started playing with this on my note 4, but I found that if I use the default constructor (WebCamTexture()), performance is significantly improved. Not sure if that's high-resolution or not, but it at least appears to be identical to the primary camera app...
     
  15. bariscigal

    bariscigal

    Joined:
    Oct 26, 2009
    Posts:
    46
    I wonder if it is because a faster device. Since on my Samsung S3 i am having pretty low performance on webcam texture.
    As @bitter kindly noted writing a plugin that copies the camera feed directly to GPU and getting the id of that texture seems most performant way.
    But i have been reading lots (and lots) of info on this matter and i am not even close to building a plugin with my knowledge.
    Any help would be appreciated.
    (Please don't point to Vuforia. It is not "free")
     
  16. khacpm

    khacpm

    Joined:
    Nov 19, 2013
    Posts:
    3
    bump......
     
  17. khacpm

    khacpm

    Joined:
    Nov 19, 2013
    Posts:
    3
    New version of Unity 5.2.0f3 was released.
    But the bug still there.
    I am starting to lose faith in this platform.
     
    DeltaCygniLabs likes this.
  18. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,773
    Hate to revive a dead thread, but what if I packaged my code to the UAssetStore, would any of you be interested? I made a wrapper around iOS and Android's native cameras. It supports autofocus, tap to focus, flashlight, high resolution photos, and video capture (video capture is incomplete). And this you'll love:

    On my S3, I get between 23-30FPS Live camera preview. It's done with GPU processing native side.

    Oh and as an added goodie, it supports barcode detection on both platforms.
     
    Bunderant likes this.
  19. jcarpay

    jcarpay

    Joined:
    Aug 15, 2008
    Posts:
    539
    Would be very nice!
     
  20. exerion

    exerion

    Joined:
    Jun 29, 2011
    Posts:
    54
    Hi Lanre, Yes please :)
     
  21. Bunderant

    Bunderant

    Joined:
    May 29, 2013
    Posts:
    21
    Lanre, that would be incredible. I'm currently working on a wrapper for Android myself, and it's really slow going. I'm doing some pretty heavy image processing with an OpenCV Unity plugin. Coupled with the poor WebCamTexture performance, that portion of our game is essentially unusable on those devices.

    If you decide to go through with it, do you have a general timeframe in mind for video capture? It would be great to avoid this Android issue and focus more time on optimizing and fine tuning gameplay as our release date approaches. Meanwhile, I'll continue my tinkering.

    ...Also, if you throw in exposure control, I'll totally be your best friend.
     
  22. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,773
    Hey guys. Love the feedback. I'll go on ahead and finish it. I hope 2-3 weeks isn't too long a wait time; I've got a huge time-sensitive project on hand (that uses NatCam). Video capture could be done, I mentioned that it's incomplete because I only got it running on iOS before I became occupied with the current project. As for your OpenCV needs, I'll be sure to add a little helper method that does Blits (like RenderTexture's Blit, but this time it'll be an extension method for Texture2D) so that you could do GPU processing if you wish to.

    I'll look into Exposure Control. Might or might not make it into 1.0.

    [EDIT]
    Ohh and concerning video capture, retrieving the mp4 or mov might be an issue. If you'd like to playback the recorded video, you'll have to get an asset like MovieTexture for mobile or so. I don't intend to build that functionality into NatCam, although I might consider it.
     
  23. bariscigal

    bariscigal

    Joined:
    Oct 26, 2009
    Posts:
    46
    a "shut up and take my money" meme would fit here probably :) (don't mean you to shut up...please continue to share info...i...er..oh well)

    Yes please...:)
    Need beta tester?
     
  24. Bunderant

    Bunderant

    Joined:
    May 29, 2013
    Posts:
    21
    Thanks for the quick response, Lanre! Just realized I've mixed up the semantics regarding video capture, all I need personally is the camera preview, not any recording. All my project depends on is functionality similar to WebCamTexture, just with better performance on Android. That is, play/pause and retrieving the image data as a Texture2D. The OpenCV plugin I have is "OpenCV for Unity," which has a utility for converting a Texture2D to an OpenCV matrix. Of course, if you feel like getting fancy, don't let me stop ya!

    Again, thanks for considering all my feedback, and best of luck with your other project.
     
    bariscigal likes this.
  25. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,773
    Hi Bunderant. That's better for me in fact. Video capture would have taken extra time. Can I have you as a Beta tester? I say this because I need to know if Texture2D to OpenCV Mat will work. I discovered a bug with Unity's external texture API (CreateExternalTexture, UpdateExternalTexture) that doesn't allow any reading or writing on the external texture. This means that GetPixel(s), SetPixels, ReadPixels, and EncodeToJPG or PNG won't work on the camera preview. For still image capture, I made a helper function that goes around this problem (because it's a one-shot process). But or processing every frame, this will certainly be a problem.

    I'm opening a new forum thread so that Unity Devs know of this. Fingers crossed, Texture2D to OpenCV Mat works. And Bariscigal, yes I'll have you as a Beta tester. Shoot me your email address.
     
    Bunderant likes this.
  26. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,773
    iOS done. Hopefully will finish Android tomorrow. Then share to betas then submit to Asset Store :)
     
    Bunderant and jcarpay like this.
  27. Bunderant

    Bunderant

    Joined:
    May 29, 2013
    Posts:
    21
    Hey, Lanre. I'd be more than happy to test that out for you, and I'll let you know if I have any ideas for a workaround on the external texture issues. I'll send ya my email address. Thanks!
     
  28. 128bit

    128bit

    Joined:
    Oct 8, 2014
    Posts:
    117
    Im also interested, any pricepoint in mind Lanre?
     
  29. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,773
    Hi 128bit. I'll be putting it at about $40. I'm getting NatCam to work with Android's multithreaded rendering right now :D

    EDIT
    Works :)
     
    Last edited: Dec 1, 2015
  30. Packedbox

    Packedbox

    Joined:
    Jun 20, 2013
    Posts:
    17
    Hi Lanre, I'm interested to test it and buy it as well when it's out. A little question, will you provide the source code ?
     
  31. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,773
    Hi Packedbox. Unfortunately, the beta is closed. And yes of course all sources are included, it won't make sense to create such a solution without allowing customization. NatCam will be submitted for Asset Store review by Tuesday.
     
  32. 128bit

    128bit

    Joined:
    Oct 8, 2014
    Posts:
    117
    I got acess to the beta some days ago. Honestly, tested it on several devices. Did worked on each without any issues. Performance is spot on. Even on low end devices!
     
    MrEsquire likes this.
  33. jcarpay

    jcarpay

    Joined:
    Aug 15, 2008
    Posts:
    539
    Sounds promising!
     
  34. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,773
    Yes in fact it is. We're prepping ourselves for release. And I personally can't wait to see how NatCam gets modified and improved. I'm writing a brief explanation file for the sources so that it won't look so intimidating to anyone. Soon guys, soon :)
     
    jcarpay likes this.
  35. Packedbox

    Packedbox

    Joined:
    Jun 20, 2013
    Posts:
    17
    Hi Lanre, thanks for he update can't wait to try it.
     
  36. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,773
    tolstenko likes this.
  37. ChristianBG

    ChristianBG

    Joined:
    Nov 13, 2019
    Posts:
    1
    Hello everyone,

    I had terrible issues using WebCamTexture even using default constructor. I solved it enabling the option "Auto Graphics API" in Player settings, specifically in "Other Settings". I don't know if it is the root cause for all of you but it is works for me. See attached image.
     

    Attached Files:

  38. GBoswood

    GBoswood

    Joined:
    Dec 26, 2017
    Posts:
    1
    Thank you ChristianBG! This gave me a huge improvement. When I enabled Auto Graphics API, it gave me some warnings about other settings, but I squelched those by changing Color Space to Gamma and Lightmap Encoding to Low Quality with no ill side effects for my project.
     
unityunity