Search Unity

  1. Good news ✨ We have more Unite Now videos available for you to watch on-demand! Come check them out and ask our experts any questions!
    Dismiss Notice

NatDevice - Media Device API

Discussion in 'Assets and Asset Store' started by Lanre, Dec 17, 2015.

?

Should we add exposure controls in v1.3? This means dropping support for iOS 7

Poll closed Jun 10, 2016.
  1. Yes

    9 vote(s)
    75.0%
  2. No

    3 vote(s)
    25.0%
  1. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,419
    This is a bug that affects Android O only. We have fixed it on our end. I will be publishing the update on the Asset Store within the next few days.
     
  2. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,419
    The picture and photo resolutions are different; as a result, the aspect ratios are also different. This can be subverted by setting the preview and photo resolutions to have the same aspect ratio.
     
  3. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,419
    Hi guys. I apologize for being off the radar for the past few days. I have been completely occupied with another project. Anyway, I'm back with exciting news, fixes, and a new API in the works that many of you will want to get your hands on. More info will be coming shortly :D
     
  4. Xenc

    Xenc

    Joined:
    Feb 2, 2016
    Posts:
    13
    Thanks for the fast response!
    Is this just setting the Preview Resolution and Photo Resolution at the same value, e.g. Highest? If I set them both equal in the MiniCam sample, the full screen is being filled with the captured image, but the image is zoomed in. Am I missing something?

    Kind regards
     
  5. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,419
    You need to make sure that they have the same aspect ratio. The highest preview resolution may be different from the highest photo resolution (check with the DeviceCamera.PreviewResolution and DeviceCamera.PhotoResolution getters). You can try setting the width and height manually instead of a preset (there are overloads that take width and height).
     
  6. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,419
    Hi guys. A lot of you have asked if NatCam supports recording videos with UI overlays. The answer up till now has been no...but not anymore.

    Check out the NatCorder Video Recording API. It's currently in the works and is going into open beta this coming week.
     
  7. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,419
    Hi guys. I'm excited to announce that NatCam Pro is currently on sale on the Asset Store for the entire week. Grab it for half the price!
     
  8. Joits

    Joits

    Joined:
    Jul 12, 2016
    Posts:
    26
    Hi Lanre
    I'm very interested in the asset and went ahead and bought the pro version.
    I need the camera feed for a simple VR app. The problem im facing is the framerate on the phone is absolute terrible (I reckon around 10 fps), when i try to run the sample scenes (SimpleCam). The profiler tells another story where it reports a frame rate consistently around 60.
    Do you know what causes this discrepancy?
    I'm using an Samsung S7 for testing.
     
  9. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,419
    Some S7 and S7 Edge models have a graphics driver bug that causes the framerate to dip when NatCam tries to get the preview data from the GPU. The workaround is to open NatCamAndroid.cs (in NatCam>Pro>Plugins>Managed>Platforms>Android) and turn off the PreviewData flag. This will cause the camera preview to be fluid, but you will not be able to use the PreviewBuffer, PreviewFrame, and PreviewMatrix API's.
     
  10. Joits

    Joits

    Joined:
    Jul 12, 2016
    Posts:
    26
    Thank you Lanre for the quick response. Your fix solved the issue. Looking forward to develop using the asset!
     
    Lanre likes this.
  11. learningbank_aj

    learningbank_aj

    Joined:
    Jun 2, 2017
    Posts:
    7
    Hello, how is the support for GearVR?
    I tried the VR Cardboard example and it appears to be working fine.
    But when i try to build using "Oculus" SDK in the VR settings the demo scenes just appear white (I guess its the preview panel) when loaded.
    Are there any steps that im missing?
     
  12. Xenc

    Xenc

    Joined:
    Feb 2, 2016
    Posts:
    13
    Hi Lanre,

    I tested this in the Minicam demo scene, by editing the code in NatCamBehaviour as shown in screenshot below. I've set a resolution with ratio 4:3.
    upload_2017-11-21_17-17-35.png

    I traced the logs via Logcat:
    LG tablet sets both preview and photo resolution to 1280x800, aspect ratio 24:15.

    upload_2017-11-21_17-30-30.png

    On our Samsung S6 edge:
    Preview resolution has been set to 1024x768, aspect ratio 4:3.
    Photo resolution has been set to 1280x720, aspect ratio 5:3
    upload_2017-11-21_17-22-14.png

    Is there any way to set both preview and photo resolution the same, so both visualizations are the same for the app user?

    Kind regards
     
  13. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,419
    I don't think you're missing anything. Can you send me the logs from logcat? That will contain the cause of this issue.
     
  14. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,419
    The resolution you set, 1152x864, is not widely supported on Android devices. As a result, NatCam will set a resolution closest to that (which is different on different cameras and devices). In my experience, 1280x720 is probably the most common supported resolution, so try that instead.
     
  15. Xenc

    Xenc

    Joined:
    Feb 2, 2016
    Posts:
    13
    Hi Lanre,

    As you proposed, I changed the preview and photo resolution to 1280x720.
    Now on the LG tablet, Natcam sets the resolution to 1280x720 for preview, and to 1280x768 for photo.
    On the other side, it now works on my Samsung s6 edge smartphone.

    upload_2017-11-23_14-58-52.png

    As you can see, the aspect ratio still isn't the same for preview and photo resolution, although I'm setting them manually to the proposed 1280x720.
    What a user expects is that the taken picture has the same resolution and aspect ratio as the preview.

    Is it possible to have both preview and photo resolution set at the same resolution and aspect ratio, for any (or at least most) android devices?

    Kind regards
     
  16. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,419
    Different devices and cameras support different sets of preview and photo resolutions. As a result, you cannot enforce one picture and photo resolution. For most Android devices though, 1280x720 is your absolute best bet.
     
  17. rattlesnake

    rattlesnake

    Joined:
    Jul 18, 2013
    Posts:
    138
    Hello Lanre ! I just spotted a very strange bug (on several different android devices).
    If you take the miniCam scene with auto-rotation activated :

    If you switch like 6 times from Portrait to Landscape, then a big zoom is appearing more and more.
    Thank you if you can put an eye on this, it's critical for my project :/

    Best,
    Seb
     
  18. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,419
    This might not be a bug in NatCam's preview rendering pipeline. Are you using a NatCamScaler? If so, what is the scale mode on it? Chances are that it is currently on ScaleMode.Fill. If you set it to ScaleWidth or ScaleHeight, then you should not encounter this issue anymore.
     
  19. rattlesnake

    rattlesnake

    Joined:
    Jul 18, 2013
    Posts:
    138
    Thanks for pointing me in the right direction :)
    ScaleMode.None seems to work properly in my case but I will do further experiments !

    Strangely the "bug" of ScaleMode.Fill occurs only on android and not in IOS though.

    Best,
    Seb
     
  20. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,419
    It actually depends on the aspect ratio of the preview, not necessarily iOS vs. Android.
     
    rattlesnake likes this.
  21. blamejane

    blamejane

    Joined:
    Jul 8, 2013
    Posts:
    228
    I just updated from 1.5b4 to 1.6 and Xcode fails with NatCamRenderDispatch.h file not found. I've completely removed NatCam asset and then imported 1.6 again, but the same problem occurred.

    Build settings: NATCAM_16;INATCAM_C;NATCAM_CORE;NATCAM_PRO

    Can you help?
     
  22. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,419
    Can you check that the file is in NatCam>Core>Plugins>iOS? If it is not there, let me know. If it is, then delete your Xcode project (in case you have been building by appending) and rebuild it from Unity.
     
  23. blamejane

    blamejane

    Joined:
    Jul 8, 2013
    Posts:
    228
    Files are there (.mm, .h)

    Close and re-open Unity (5.5.1f1),
    New Unity project,
    switch to iOS,
    import natcam 1.6,
    build and and run,

    fails in Xcode same error
     
  24. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,419
    When you build to Xcode, in the project hierarchy, check that the files are included. See attached screenshot.
     
  25. blamejane

    blamejane

    Joined:
    Jul 8, 2013
    Posts:
    228
    Files are not all there (see image). I can add them, but how do I get Unity to include them automatically?


     
  26. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,419
    Unity should copy those files verbatim. Including the remaining files should let you build, but Unity should be doing this for you.
     
  27. Denaton

    Denaton

    Joined:
    Apr 6, 2011
    Posts:
    6
    Is this one fixed? I just updated my phone to 8.0 and the CapturePhoto stop working, i first trough it was something i had coded wrong.

    I have updated to NatCam 1.6f1 and it still wont work.
     
  28. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,419
    It has been fixed, but the update is not live on the Asset Store yet. Email me with your invoice number and I will send you the update.
     
  29. Bakwon

    Bakwon

    Joined:
    Jul 20, 2017
    Posts:
    4
    Hi Lanre,
    I recently bought NatCam core. It looks like a great asset and our company is very keen to use it. We are really after the liquid smooth preview in our unity app...
    Unfortunately I have a problem with the performance and I'm hoping it might be something basic I am getting wrong?
    The preview image appears jerky - like something is going on in the background. Its not terrible, but its not smooth or usable either.
    I am using NatCam Core 1.6f1
    I am deploying from unity to Samsung Galaxy S6 - Android 7.0.
    I am using the SimpleCam example that comes with NatCam.
    I have tried all different frame rates and resolutions but it does not make any difference.
    I tried setting exposure mode to locked and focus mode to off.

    Please let me know if you have any ideas or if I could provide any more details to help.

    Thank you
     
  30. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,419
    I would need to gather more information. What version of Unity are you using? What framerate do you get (I have attached scripts that measure framerate). And just to confirm, are you using Core or Pro?
     

    Attached Files:

  31. Bakwon

    Bakwon

    Joined:
    Jul 20, 2017
    Posts:
    4
    Thank you for the quick response. Please could you briefly explain the framerate scripts? I have added them to my project and tried adding FramerateMonitor.cs to my 'NatCam' camera gameobject. Ive linked the preview text and app text properties to a UI text box within my scene. Am I on the right path? Sorry! Im still pretty new to Unity...

    I am using Unity 2017.2.0f3
    I am using NatCam Core
     
  32. Joits

    Joits

    Joined:
    Jul 12, 2016
    Posts:
    26
    Hi Lanre
    I'm currently trying out a simple AR app using Oculus SDK with VR enabled.
    Works fine so far but the one thing which bothers me is the apparent uneven aspect ratio of the camera feed rendered in the two eyes (it looks like a 3:2'ish aspect ratio) I have tried to set the preview to a 1:1 aspect ratio resolution but to no avail.
    I was wondering if it is possible to change the aspect ratio of the camera to a 1:1 to fit a better AR experience without a distorted preview.
    I read something about it might be a hardware limitation due to the preview size of the camera, is this correct?. Im using a Samsung S8.
    See attached reference squared grid. The distortion is clearly visible.
    Thanks in advance :)
     

    Attached Files:

  33. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,419
    Hi. Sorry, I should have explained. Your setup so far is correct. Make sure the 'Monitor' checkbox is ticked then run the app. You should see your UI text display the framerate on the screen. The 'Preview Text' shows the framerate of the NatCam preview while the 'App Text' shows the framerate of the Unity application.
     
  34. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,419
    Hey there. It is definitely a hardware limitation. I have never seen a mobile camera that supports a square preview or photo size. You have to manually correct for this in your application; I am not knowledgeable on how exactly you would go about this.
     
  35. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,419
    Hi guys.

    For everyone who has ever wondered if NatCam can record videos with UI and gameobjects overlaid, we have finally created an API that will let you do just that and more. Check out the NatCorder API.
     
  36. Bakwon

    Bakwon

    Joined:
    Jul 20, 2017
    Posts:
    4
    Thanks for your reply.

    I have checked the framerates:
    The app framerate is pretty much a constant 30fps (or 60fps if I enable that option)
    However, the preview framerate varies. This sounds strange, but i discovered that the framerate varies as the lighting changes.
    very low lighting - preview = average 17fps
    very high lighting - preview = average =30fps
    I have tried it many times.
    It is repeatable.
    It appears that light intensity coming into the camera is inverse linearly proportional to the preview framerate.
    I am having a look into the focus and exposure settings again. Maybe they are kicking in.
     
  37. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,419
    This is exactly what I thought. Cameras work by accumulating light on a sensor before producing an individual frame. Because mobile devices almost always have fixed-aperture cameras, they must expose for longer in darker lighting conditions before they can issue a frame. This is a hardware limitation (or better said, an artifact); NatCam is not the performance bottleneck. As you can see, NatCam will render preview frames as fast as the camera can supply them. Over the years of developing NatCam, we have made sure to squeeze out the most performance from mobile devices.
     
  38. upressplay

    upressplay

    Joined:
    Aug 8, 2017
    Posts:
    24
    I am working on an app using OpenCV for Unity for facemasks and props. It uses a WebCamTextureToMat helper that is using Unity's WebCamTexture. We are running into exposure issues on a variety of Android devices. Sometimes the Front Camera, sometimes the back. Hence why we purchased NatCam.
    Questions
    Will NatCam.play have better results than WebCamTexture? or is it just an extension?
    Do we need to do NatCam.Play or can we just adjust the exposure with DeviceCamera.FrontCamera.MaxExposureBias?

    The later doesn't seem to hold, unless I call NatCam.Play.

    I guess I would want to read if an exposure setting is too low, and adjust accordingly at runtime. Without having to replace how the WebCamTextureToMat helper is being used. Would require a ton of reverse engineering.

    Thoughts?
     
  39. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,419
    Yes, much much better. NatCam is a full camera API, not a wrapper around WebCamTexture. NatCam will only fall back to WebCamTexture when it does not have native support (like on desktop platforms).
    Yes. You must either use NatCam or WebCamTexture, never both. Set NatCam.Camera to DeviceCamera.FrontCamera, set the exposure, then call NatCam.Play.
    When using NatCam with OpenCV, the workflow is somewhat different. NatCam comes with the OpenCVBehaviour base class that provides an easy way of interacting with OpenCV.
     
  40. Bakwon

    Bakwon

    Joined:
    Jul 20, 2017
    Posts:
    4
    Thank you for this explanation. I understand what is happening now. With this information I am trying to overcome the slow framerate. We are making a VR/AR app so framerate is really important.

    Firstly I have tried locking the exposure and turning auto focus off by adding code to the NatCamBehaviour but it has not made a difference.
    I would expect this to help stabilise the framerate? (at the cost of image quality)

    Secondly, according to Samsung, the Galaxy S8 is capable of Full HD video @ 60fps. So I have also tried changing the NatCam Behaviour to Full HD preview resolution and smooth framerate (I also tried 'highest framerate')
    Again this made no difference to the monitored framerate.

    I am a bit confused! Please could you advise.

    Thank you again for the quick and quality support.
     
  41. rattlesnake

    rattlesnake

    Joined:
    Jul 18, 2013
    Posts:
    138
    Hi lanre !
    I have a Noob question I can't figure it out :/

    I'm trying to use an other plugin in order to rotate/mirror the photos more quicky using GPU.
    So I try to make some conditions with the orientation variable.

    For instance on Android, when you take a front photo in portrait mode, you will get :
    orientation => "Rotation_90, Mirror".

    So I tried to match this case :
    Code (CSharp):
    1. if (orientation == Orientation.Rotation_90 || orientation == Orientation.Mirror)
    2. {
    3.     Debug.Log("orientation => Orientation.Rotation_90+mirror"); //Doesn't work
    4. }
    5.  
    6. if (orientation == Orientation.Mirror)
    7. {
    8.     Debug.Log("Mirror only"); //Doesn't work
    9. }
    What is the right syntaxe ?
    Thank you very much :)
    Seb
     
  42. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,419
    Not necessarily. Even if you lock the exposure, the camera still has to accumulate enough light before issuing a frame.
    Not necessarily. The camera must still accumulate enough light before issuing a frame. Locking the exposure will (theoretically) not help with this.
    I'm not sure what could be causing the device to not reach 60FPS. One useful thing to do is to set the application framerate to 60 (so that you can actually see as many frames as are produced by the camera/NatCam).
     
  43. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,419
    The Orientation enum is a set of bit flags. You can check if a specific flag is set by using bitwise-AND:
    Code (CSharp):
    1. if ((orientation & Orientation.Mirror) != 0) { // The mirror flag is set
    2.  
    3. }
    4.  
    5. if (((orientation & (Orientation.Rotation_90 | Orientation.Mirror)) != 0) { // Rotation 90 and mirror
    6.  
    7. }
     
    rattlesnake likes this.
  44. upressplay

    upressplay

    Joined:
    Aug 8, 2017
    Posts:
    24
    I'm trying to replace the following in OpenCV for Unity. How would I go about that with NatCam? I'm not seeing how I inject those values. Got the cam rendering in the environment.

    webCamTextureMat = webCamTextureToMatHelper.GetMat ();
    colors = new Color32[webCamTextureMat.cols () * webCamTextureMat.rows ()]; texture = new Texture2D (webCamTextureMat.cols (), webCamTextureMat.rows (), TextureFormat.RGBA32, false);

    I was expecting something like this, coming from OpenCVBehaviour.cs.
    Mat matrix;
    webCamTextureMat = NatCam.PreviewMatrix(ref matrix);
    colors = NatCam.colors;
    texture = NatCam.texture;

    Where is my disconnect?
     
  45. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,419
    OpenCVBehaviour already does this for you. In the OnMatrix function, you receive a matrix containing the current preview frame. Then you can use it for drawing. To convert to a texture, call FlushMatrix then access the 'texture' variable.

    We will review the design of OpenCVBehaviour in the next minor update.
     
  46. upressplay

    upressplay

    Joined:
    Aug 8, 2017
    Posts:
    24
    Got it working, sort of. Added the following, extended the class OpenCVBehaviour and got it running.

    public override void OnMatrix () {

    FlushMatrix();
    webCamTextureMat = matrix;
    }

    The facial recognition is really off now. It's like the material stops being recognized and the scaling is off. I am doing it on Update. Attached is my file.

    if (useNatCam) {
    NatCam.PreviewMatrix (ref matrix);
    rgbaMat = matrix;
    FlushMatrix();
    } else {
    rgbaMat = webCamTextureToMatHelper.GetMat ();
    }
    Not sure if I need to do the calculation on another function, rather than Update?

    File got a bit messy with NatCam integration. This is happening on line 462. When I turn the useNatCam back to false, it's working with the pre-established WebCamTextureToMatHelper that came with the Opencv for Unity. Any thoughts would be appreciated. The WebCamTextureToMatHelper's performance on mobile is spotty across our device range.

    What type of updates are you going to be making to OpenCVBehaviour?
     
  47. upressplay

    upressplay

    Joined:
    Aug 8, 2017
    Posts:
    24
    Tried to edit the attachment and the forum blocked it.
     

    Attached Files:

  48. rattlesnake

    rattlesnake

    Joined:
    Jul 18, 2013
    Posts:
    138
    Hello Lanre,

    I tried :
    Code (CSharp):
    1. if (DeviceCamera.Facing == Facing.Front)
    2. {...}
    But I got this error :
    An object reference is required to access non-static member `NatCamU.Core.DeviceCamera.Facing'
     
  49. upressplay

    upressplay

    Joined:
    Aug 8, 2017
    Posts:
    24
    So, I saw this post on your Tutorial.

    "Nice to talk with you, I have solved my problem. It seems that WebCamTextureHelper is different from NatcamHelper, because of Natcam preview update is independent of Update(). So I implement it in another way and it proves that Natcam runs well with OpenCV. Thanks a lot again!"

    Independent of Update()? Do you know what his solution was? I think we are trying to do the same thing. Integrate NatCam Pro in lieu of WebCamTextureHelper. Do I need to put my facial recognition logic, currently in Update, into an extended function OnFrame()?
     
  50. upressplay

    upressplay

    Joined:
    Aug 8, 2017
    Posts:
    24
    I think you have to do.
    if (NatCam.Camera.Facing == Facing.Front)
    Which is the active camera.

    Otherwise you can reference the front facing camera
    DeviceCamera.FrontCamera
    DeviceCamera.RearCamera
     
    rattlesnake and Lanre like this.
unityunity