NatDevice - Media Device API

Yup! NatCam has the OnFrame event which is fired when the preview updates (when a new frame gets issued by the camera). The OnFrame event is invoked as frequently as the minimum of the camera framerate and the application framerate. The reason for this is that if the camera framerate is lower than the app framerate, then you might see several app frames (Update() gets invoked several times) even though the camera preview has not updated. In this case, there is no benefit to performing the vision calculations on the same camera frame several times. When the camera is updating more frequently than the application (probably because the camera framerate is higher than Unity's), then you can't actually see all the camera frames that are issued in between app frames (Unity will only update what's on screen as frequently as the app framerate). So any calculations you do for camera frames that won't actually make it to screen will be wasted computation. This is why you should perform your calculations in NatCam's OnFrame event.

Yes, you should. If you are using OpenCVBehaviour (in a class that derives from it), then use the OnMatrix function as it will automatically get invoked by OpenCVBehaviour on every new camera frame (from NatCam's OnFrame event).

 

@upressplay Can you send me an email? We have decided to deprecate OpenCVBehaviour, and instead replace it with something more convenient to use. I believe that forcing you to move all your vision computation logic to OnMatrix is far too intrusive. I'd ask that you don't make any changes to your code just yet. I want to design an OpenCV interop API that will be very easy for you and others like you to use. I'll send you updated NatCam builds as we design and implement the new API.

 

Lanre, wanted to check to make sure you received my email. And if there is a timeframe on the update? I have a deadline on Friday. Really need to find a fix soon.

 

I got your email and have responded. Sorry for the delay.

 

Good morning Lanre,
I definitly have troubles with the scaleModes for the preview.
I tried all the types and have different issues, so I decided to compile a little video in order to show you (the youtube link is not referenced):

This test is made with the basic miniCam scene. The only modifications are made by changing the scaleMode and Preview/Photo resolutions.

The best choice for me is the scaleMode.Fill but as you can see I have this strange zoom issue.
Thank you in advance

Seb

 

Also, I try to record a thumbnail with the replayCam.
i saw your exemple :
https://gist.github.com/olokobayusuf/c7cb42e28280128fb712eedca25f5ba8

But I don't understand what sharing is ?
Seb

 

Hi Seb.

Thanks for the very informative video. We will be uploading an update today or tomorrow. We have deprecated NatCamPreview and the ScaleMode enum. For scaling, we want devs to move to the AspectRatioFitter component in UnityEngine.UI. It essentially does what NatCamPreview does, so there's no need for NatCam to duplicate that functionality. I suggest moving to it.

Regarding the stretching when you take a photo on iOS in landscape left, we have fixed it in the upcoming update.

This example is not for NatCam.

 

Thank you, can't wait to try it

So is there any builtin solution to get the thumbnail ?

 

Nope, NatCam does not offer this functionality.

 

Switched over to IOS to build to iPhone and started getting the following. I don't recall if I got this error before or after NatCorder was added.

EntryPointNotFoundException: NCCoreRegisterCallbacks
NatCamU.Core.Platforms.NatCamiOS..ctor () (at Assets/NatCam/Core/Plugins/Managed/Platforms/iOS/NatCamiOS.cs:57)
NatCamU.Core.NatCam..cctor () (at Assets/NatCam/Core/Plugins/Managed/NatCam.cs:123)
Rethrow as TypeInitializationException: An exception was thrown by the type initializer for NatCamU.Core.NatCam
NatCamU.Core.DeviceCamera.get_Device () (at Assets/NatCam/Core/Plugins/Managed/NatCamDevice.cs:275)
NatCamU.Core.DeviceCamera.get_Facing () (at Assets/NatCam/Core/Plugins/Managed/NatCamDevice.cs:25)
NatCamU.Core.DeviceCamera.<DeviceCamera>m__0 (NatCamU.Core.DeviceCamera c) (at Assets/NatCam/Core/Plugins/Managed/NatCamDevice.cs:300)
System.Linq.Enumerable.First[DeviceCamera] (IEnumerable`1 source, System.Func`2 predicate, Fallback fallback)
System.Linq.Enumerable.FirstOrDefault[DeviceCamera] (IEnumerable`1 source, System.Func`2 predicate)
NatCamU.Core.DeviceCamera..cctor () (at Assets/NatCam/Core/Plugins/Managed/NatCamDevice.cs:300)
Rethrow as TypeInitializationException: An exception was thrown by the type initializer for NatCamU.Core.DeviceCamera
TRU.Photobooth.MasksProps.OnChangeCameraButtonClick () (at Assets/TRU/Photobooth/Scripts/MasksProps.cs:1128)
TRU.Photobooth.MasksProps.StartNatCam () (at Assets/TRU/Photobooth/Scripts/MasksProps.cs:114)
TRU.Photobooth.MasksProps.Update () (at Assets/TRU/Photobooth/Scripts/MasksProps.cs:1017)

 

Unity crashed. I restarted and the error went away. Might have something to do with another 3rd party we are using.

 

Hi,

I got a strange error trying to take a photo on a Google Pixel phone (8.0):

AndroidJavaException: java.lang.NoSuchFieldError: no "Ljava/lang/Object;" field "mBuffer" in class "Landroid/graphics/Bitmap;" or its superclasses
java.lang.NoSuchFieldError: no "Ljava/lang/Object;" field "mBuffer" in class "Landroid/graphics/Bitmap;" or its superclasses
at com.unity3d.player.ReflectionHelper.nativeProxyInvoke(Native Method)
at com.unity3d.player.ReflectionHelper.a(Unknown Source:0)
at com.unity3d.player.ReflectionHelper$1.invoke(Unknown Source:6)
at java.lang.reflect.Proxy.invoke(Proxy.java:913)
at $Proxy8.onPhoto(Unknown Source)
at com.yusufolokoba.natcam.NatCam.onPictureTaken(NatCam.java:222)
at android.hardware.Camera$EventHandler.handleMessage(Camera.java:1106)
at android.os.Handler.dispatchMessage(Handler.java:105)
at android.os.Looper.loop(Looper.java:164)
at android.os.HandlerThread.run(HandlerThread.java:65)
at UnityEngine.AndroidJNISafe.CheckException () [0x00000] in <filename unknown>:0
at UnityEngine.AndroidJNISafe.GetFieldID (IntPtr clazz, System.String name, Syst

Is there a way to fix it?

 

We have submitted an update that fixes this issue.

 

Hi,

NatCam seems to have a problem with iPhone X for previewing or CapturePhoto. Even if I put the NatCamPreview component on Preview Image, the image is skewed when rotating the device. Also when capturing photo, it returns flat purple image.

It happens when using Minigram scene from plugin version 1.6.1 on iPhone X with rear camera. Capturing photo works on face camera.

 

Can you email me with more information. Specifically, can you share screenshots? Also, remove NatCamPreview and lock the orientation of the app (in Player Settings). Send me the full logs from Xcode too, and make sure that NatCam's verbose mode is on.

 

Ok I will write an email now, jfyi screenshot is attached here.

 

I've just sent an email with more information. Thank you in advance.

P.S. by the way the same problem happened in 1.5.3 which is the reason why I updated to newest version.

 

I saw your email. Regarding the aspect ratio, I recommend using the AspectRatioFitter UI component as opposed to NatCamPreview; we are deprecating this component in the upcoming update. Here is an example for how to use it.

Regarding the purple screen when you take a photo, I saw in the logs that you are using OpenGL ES2 as opposed to Metal. GLES2 cannot handle textures as big as the photo captured, so you should either reduce the photo resolution or switch to the Metal API.

 

Thank you for your reply.

I have tried using AspectRatioFitter component to the Preview but I still got the skewed image on iPhone X.

I have just also tried setting the graphics api to Metal but I still get this purple screen on iPhone X.

It really seems like device dependent.

Please check accordingly,

Thank you.

UPDATE 1: I have also tried changing the NatCam Photo Resolution to lower than Highest such as Full HD but it didn't fix the purple view.

 

This is correct. The iPhone X seems to be taking photos with very very high resolution. Can you set the preview resolution to lowest or medium and then set the photo resolution to lowest and let me know if the issue persists? Also, simply check 'Auto Graphics API' instead of manually setting Metal (and excluding GLES2).

 

I have just tried, checking "Auto Graphics API" and set both preview resolution and photo resolution to medium and it worked without screen being purple. It seems to work as well when I set the preview resolution to medium and photo resolution to highest which is a bit weird.

The other problem for skewing preview on iPhoneX (image attached) is still there which also happens on iPhone 7 plus (I've just checked).

Thank you,

Update: by the way I'm using World Space Canvas for camera preview image which is necessary for my app usage.

 

We've removed the NatCamPreview component, so you have to make sure that the aspect ratio of your RawImage matches that of the camera preview using the AspectRatioFitter UI component. See this example on how to do so.

 

Thank you the updated version is way better and stable
I will test it more deeply but great job !!!

 

@Lanre Evaluating use of natcam for an ar project - and thus far have been unable to get it to work on an galaxy s6 on android vs 6.0.1. I have tried the simple cam scene and the intro tutorials but no luck with getting camera pass through with the built APK on this device. Is there something that is likely overlooked that could be the cause? Have tested this in empty projects using 5.6.4f1 and 2017.1.1f1. Also - in my testing we have been able to get it working on Kindle Fire HD 8 / Nexus 5 / iPad 2 without issue. Thanks ahead of time.

 

It's impossible to tell what might be going on without seeing logs. Can you upload the full logs from logcat?

 

Happy Holidays everyone! Thank you for all the support in bringing NatCam up to what it is today.

 

Hi Lanre
I'm developing a Gear VR application and currently looking into differen

Thanks for the reply Lanre
I've been looking into vuforia and have some ok results, however your plugin is way faster in terms of FPS.
The issue with Natcam is the distortion because of the 1:1 aspect ratio of NatCam. Vuforia on the other hand has "fixed" this by changing aspect ratio to fit the cameras, see attached image.
Is there a way to replicate this using NatCam? I've tried to fiddle with the aspect ratio fitter but with little luck.
Using a Samsung S8 if that helps.

 

I am not well versed in (virtual) camera calibration parameters. It seems like Vuforia is doing something behind the scenes to stretch out the aspect ratio of each view. I don't know what it is doing exactly. You might get answers by asking on the Vuforia forums.

 

Hi guys. We deprecated the OpenCVBehaviour class in the last update to the Asset Store. But the VisionCam example was not updated to reflect this. I have attached an updated VisionCam script. Use it to replace the current VisionCam.cs in the NatCam>Pro>Examples>VisionCam directory.

 

Hi Lanre

Im trying to display a transparent texture in front of the preview RawImage, however the texture it's being rendered behind the camera preview no matter how i order the render queue. Is there a way to fix this? I'm using worldspace canvases for the texture and NatCam.

I have also tried to blur the camera feed using an image with a fast blur shader. Again, placed in front of the preview RawImage (done to avoid per pixel operations with the feed), whenever I enable this blur feature it simply disables the camera feed entirely. Do you know what the cause is?
Thanks

 

Hey there. Are you using NatCam 1.6f1? NatCam 1.5 had a bug with depth so I need to make sure that it isn't the reason. If you are on 1.6f1, are you using a custom shader for your transparent texture? Is it a RawImage? Have you tried using a static texture (anything other than NatCam.Preview) on your preview RawImage to see whether it is the NatCam preview texture that is causing this? I highly doubt that the NatCam.Preview texture is the cause, so it might be in your setup.

It isn't possible to perform a blur without doing per-pixel operations. Your blur shader is probably performing a grab pass, which is just a more expensive 'screenshot' of the NatCam preview. If you are using a third party plugin for the blur shaders, then reach out to them for support. I am not very verse in Unity's graphics pipeline.

 

Thank you for the quick reply.
I fixed the issue with NatCam preview being on top and it turned out to be an issue with my setup. Specifically the canvas layer orders were mixed up. I'm using 1.6f1, so no issue there.

The blur layer was also fixed by the sorting the canvasses correctly.
Thanks and sorry for the inconvenience!

 

Sounds good!

 

Hi Lanre,
I updated to version 1.6f1 and imported the plugin in a fresh project. The preview works fine, but when the photo is taken it was distorted or mirrored. I attached some screenshot, the resolution is set to lowest just to test.
Front Photo:

Front Preview:


Rear Photo

Rear Preview


Thanks

 

This is because device cameras always return pixel/photo data in a 'natural' orientation (landscape left). As a result, the client (NatCam) has to correct the data for orientation so that the image produced will be upright. This is why the `PhotoCallback` has an `Orientation` parameter, which tells you how to orient the image to be upright. If you would like to display the photo, use the `NatCamPanel` component on a RawImage, then set its `orientation` variable.

If you would like to physically rotate the texture to be upright, use the Utilities.RotateImage function. I recommend that you read more about this in the Photos tutorial.

 

Hi Lanre,
I totally understand your point by providing the 'natural' image. But you have to consider the client side, I purchased this plugin because I was tired of checking all devices rotation and mirrored orientations. I just want to display a photo.

I just built the MiniCam scene on the android device, which already use this line of code. So it's bug then? Do you want the logcat?

Thanks

 

This was entirely a performance consideration. In the earlier versions of NatCam, the API would first physically rotate the image before returning it to the photo callback. The problem was that it took an excessive amount of time (given that tens of millions of pixels had to be moved around in irregular memory access patterns). So after a number of versions of going back and forth, we settled on the current API and provided tools for simply displaying the photo upright and actually physically rotating the photo.

This sounds like a bug. Can you email me screenshots and logs? Thanks.

 

Hi, I'd like to know is there any other way to check permission if user disable permission after first time the app is started? Thanks

 

Hi Lanre, why did you deprecate the save picture function? Is there a solution to save the picture to an album on the device of the user?

 

Yes, there is:

Code (CSharp):

1. bool hasPermissions = NatCam.Implementation.HasPermissions; // User has granted the app camera permissions

 

The SavePhoto API was deprecated with NatCam Extended. To save photos to the device gallery, I recommend Capture and Save Pro.

 

Hi Lanre,

I have an augmented reality iOS app that relies on NatCam. I'm getting ready to submit to Apple, however I'm trying to setup with the preview framerate/focus/etc for the best AR experience using NatCam. One of the things that has bugged me is the constant focusing that goes on with focus mode set to AutoFocus. I'm trying to minimize the flickering and focusing. I have just downloaded the latest NatCam Pro, and was following your tutorial series Starting off #1 and Controls #2.

In the Controls video you mention setting FocusMode.SoftFocus is best for VR. I'm assuming AR falls into that category, but when I try to build I get this error:

Error CS0117: `NatCamU.Core.FocusMode' does not contain a definition for `SoftFocus' (CS0117) (Assembly-CSharp)

Is this an Android only setting? What do you suggest for AR apps?

Here's my current settings:

Code (CSharp):

1.     void Start () {
2.  
3.         #if UNITY_EDITOR
4.         NatCam.Camera = DeviceCamera.FrontCamera;
5.         #else
6.         NatCam.Camera = DeviceCamera.RearCamera;
7.         #endif
8.  
9.         // Set the camera's preview resolution
10.         NatCam.Camera.SetPreviewResolution(ResolutionPreset.HD);
11.         // Set the camera's photo resolution
12.         NatCam.Camera.SetPhotoResolution(ResolutionPreset.HD);
13.         // Set the camera's framerate
14.         NatCam.Camera.SetFramerate(FrameratePreset.Default);
15.  
16.         // Lock the rear camera's exposure
17.         DeviceCamera.RearCamera.ExposureMode = ExposureMode.Locked;
18.  
19. //        NatCam.Camera.FocusMode = FocusMode.AutoFocus;
20.         NatCam.Camera.FocusMode = FocusMode.SoftFocus;
21.         // Play the camera preview
22.         NatCam.Play();
23.  
24.         NatCam.OnStart += OnStart;
25.     }
26.  

 

Hi Lanre,

When I set the Framerate with

Code (CSharp):

1. NatCam.Camera = DeviceCamera.RearCamera;
2. NatCam.Camera.SetFramerate(FrameratePreset.HighestFramerate);
3. NatCam.Camera.SetPreviewResolution(ResolutionPreset.Medium);
4. NatCam.Camera.SetPhotoResolution(ResolutionPreset.FullHD);
5. NatCam.OnStart += OnNatCamStart;
6. NatCam.Play();
7.  

and afterwards try to read it with

Code (CSharp):

1. Debug.Log ("Framerate: " + NatCam.Camera.Framerate);
2.  

I get the following error:

AndroidJavaException: java.lang.NoSuchMethodError: no non-static method with name='getFramerate' signature='()F' in class Ljava.lang.Object;
java.lang.NoSuchMethodError: no non-static method with name='getFramerate' signature='()F' in class Ljava.lang.Object;
at com.unity3d.player.ReflectionHelper.getMethodID(Unknown Source)
at com.unity3d.player.UnityPlayer.nativeRender(Native Method)
at com.unity3d.player.UnityPlayer.c(Unknown Source)
at com.unity3d.player.UnityPlayer$c$1.handleMessage(Unknown Source)
at android.os.Handler.dispatchMessage(Handler.java:107)
at android.os.Looper.loop(Looper.java:207)
at com.unity3d.player.UnityPlayer$c.run(Unknown Source)
at UnityEngine.AndroidJNISafe.CheckException () [0x00000] in <filename unknown>:0
at UnityEngine.AndroidJNISafe.CallStaticObjectMethod (IntPtr clazz, IntPtr methodID, UnityEngine.jvalue[] args) [0x00000] in <filename unknown>:0
at UnityEngine.AndroidReflection.GetMethodMember (IntPtr jclass, System.String methodName, System.String signature,
java.io.EOFException

(Filename: Line: 390)

 

Can you post a link with this example? It really will be helpful for me. Thanks

 

Is that possible to share me with your opencvforunity&NatCam example? It is really helpful for me?.Thanks 987460880@qq.com

 

Hi Lanre,

I've got a question about the barcode scanning which requires a Color32[], I've read that there should be an overloaded method of the PreviewBuffer which takes in a Color32[], but I can't seem to find that function in 1.6, has this been removed? If so, how would I go about converting the preview to a Color32[] to scan for barcodes?

Best regards

 

We removed this constant because it only applies on Android (no other platform has an explicit soft autofocus). However, it is still implemented on Android:

Code (CSharp):

1. final int
2. AutoFocusBit = 1 << 1,
3. SoftFocusBit = 1 << 2,
4. MacroFocusBit = 1 << 3;

So if you want to set soft focus, do this:

Code (CSharp):

1. NatCam.Camera.FocusMode = (FocusMode)(1 << 2 | FocusMode.AutoFocus); // Soft focus needs autofocus bit set

Note that this only applies on Android. On iOS, it will set normal autofocus (which is why you need to set the autofocus bit)

 

This was oversight on my part. The fix is pretty easy. Open NatCamDeviceAndroid.cs in NatCam>Core>Plugins>Managed>Platforms>Android and go to the GetFramerate function. Change this:

Code (CSharp):

1. return natcam.CallStatic<float>("getFramerate", camera);

to this:

Code (CSharp):

1. return natcam.CallStatic<int>("getFramerate", camera);

 

We removed it because there was already infrastructure in place to get a Color32[]. Simply use the NatCam.PreviewFrame function passing in a Texture2D, then call GetPixels32() on that texture.

 

I bought a pro version and I use it for development.
I want to adjust the aspect ratio, the frame rate and bit rate to reduce the size of the file saved when saving the movie. My preferred aspect ratio is 720x405, the frame rate is 30fps and the bit rate is 2Mbit/s.
Please explain how to code. thanks