NatDevice - Media Device API

Hi, thanks for the answer to the other questions. I have another one. :\
Have you tested natcam on a Galaxy S8? we just had some reports from a testing house that the app is crashing when it's supposed to open natcam. We've already sent a request for logcats (we don't have any S8 here) but not sure how long will it take and maybe you already have a fix for that. We have the app running just fine on S6, S7, Nexus6P just to name a few.
I'll update as soon as we get the logs.
Thanks!

 

I have not tested with the S8 because we don't have it. Send me the logs when you have them.

 

Hi I'm working on an App and I experience very low frame rate on a Galaxy S7.

Anything higher than Medium resolution turn the preview into a slideshow

It works just fine on Nexus 5, Nexus 5X, Galaxy S6 Edge so I doubt it has something to do with my code.

The phone is a S7 SM-G930F

Android 7.0
Kernel version 3.18.14-11105523
Version Number: NRD90M.G930FXXU1DQD7

 

This is because of a graphics driver issue in the S7/S7 Edge. The workaround in NatCam is disabling the preview data pipeline. To do so, open NatCam>Professional>Plugins>Managed>Platforms>Android>NatCamAndroid.cs and disable the ReadablePreview flag. Note that this will disable the NatCam.PreviewBuffer, NatCam.PreviewFrame, and NatCam.PreviewMatrix API's as they depend on the preview data pipeline.

 

Hi Lanre.

I had upgraded to Natcam Propecianal from Natcam Core today.
SimpleCam or MiniCam Example work very wall in NatCam Core Package.
But,
work very slow, with NatCam Professional Package. (About 1 to 2 fps)

I found the part that makes it slow.

Code (CSharp):

1. INatCamBase.cs
2.  
3.         [MonoPInvokeCallback(typeof(Native.StartCallback))]
4.         protected static void OnStart (IntPtr texPtr, int width, int height) {
5.             instance.dispatch.Dispatch(() => {
6.                 #if NATCAM_PROFESSIONAL
7.                 if (instance.preview == null) instance.InitializePreviewBuffer();
8.                 #endif
9.                 instance.preview = instance.preview ?? Texture2D.CreateExternalTexture(width, height, TextureFormat.RGBA32, false, false, texPtr);
10.                 if (instance.preview.width != width || instance.preview.height != height) instance.preview.Resize(width, height, instance.preview.format, false);
11.                 if (instance.onStart != null) instance.onStart();
12.             });
13.         }

#if NATCAM_PROFESSIONAL​

if (instance.preview == null) instance.InitializePreviewBuffer();
#endif

It is very slow in the code that initializes the PreviewBuffer.

I skipped this initializes part so it got faster.



My Android Device is Samsung Galaxy S7 and Android OS version is 7.0

 

See my post above.

 

Hi Lanre, How am I supposed to get the camera live feed frame data then ?
Or event presenting the camera image to the user ? Using the CapturePhoto API doesn't seem to be an option for real time preview.

I'm a bit puzzled by this driver bug, that mean's that no AR App can work smoothly with a high res buffer on the S7 phones ?

 

If you need the preview data, then you would need to keep the ReadablePreview flag on.

Use the NatCam.Preview texture. This texture is independent of the preview data pipeline (it is part of the rendering pipeline, off which the preview data pipeline operates).

I imagine that AR SDK's use the raw YUV data directly from the camera, without having to convert to RGBA and reading back that data (which NatCam must do to be able to use the preview texture in Unity). The Y plane of the YUV format contains luminance (greyscale image), which is all that is needed for tracking.

 

The second part of the NatCam Tutorial Series is up! Read NatCam Tutorial Series 2: Control.

 

A lag occurs when capturing high resolution photo.
It suggests a mode that is only directly saved on the native side for faster photo capture. does not send photo to Unity Side. Similarly StartRecording()

I'm developing a camera app using your plugin, but I'm having a hard time with the problem of saving it.

 

Could you then give a fast access to that greyscale buffer/preview/matrix ?
It's exactly what I need to give to OpenCV.

in the logs of NatCam 1.2 I see:

+ Added NativePreviewCallback and NatCamNativeInterface.EnableComponentUpdate() to get access to the raw luma (Y) and chroma (UV) buffers from the camera.

But I can't find it in the code.

 

I am not sure what your question here is.

 

We deprecated the YUV buffer support in 1.3. There were a few problems with implementing it:
- On Android, the process of converting YUV to RGBA is streamlined with the SurfaceTexture API's (as opposed to uploading the individual buffers to OpenGL ES). This simplified and optimized the rendering pipeline greatly.
- On iOS, using BGRA instead of YUV removes the need for a rendering pipeline entirely, so we scrapped it.

If you wish, I can send you the native source projects so that you can implement YUV data capture, but we will not be adding it in the release versions.

 

I'm sorry my bad english.

NatCam.CapturePhoto (OnPhoto) has a few lags (1~2sec) at its highest resolution, on android device(my galaxy s7).
but no lag on my iOS device(ipad2 air).

If you do not pass a Photo(Texture2d) in OnPhoto() callback like StartRecording (), Could reduce the lag?

Thank you for your kind response.

 

This is an Android issue in general. One workaround is to reduce the lag is to reduce the photo resolution.

The photo must be returned to the user as a Texture2D, which is why we pass it in. We won't be changing this.

 

Hi, i need to place an object in front of the camera(between of the unity camera and canvas with preview), but it blinks, how can i fix this?

sorry for my english

 

That sounds like a bug we are looking into. A workaround might be enabling multithreaded rendering in Player Settings

 

it help, thank you

 

Hello, we have been using Natcam in our project for couple of months now and it works great on iOS mobile.
Now we are focusing on supporting Android mobile as well, but discovered our logic stops at a certain moment when running on Android.
I've printed some logs on the screen and had a look and it seems that calling NatCam.Camera in the Start function of Monobehaviour breaks the flow.

Code (CSharp):

1. void Start()
2. {
3. GameManager.Instance.StatisticsManager.PushMessage("This is printed on screen");
4. GameManager.Instance.StatisticsManager.PushMessage("NOT printed on screen = " + NatCam.Camera.ExposureBias);
5. GameManager.Instance.StatisticsManager.PushMessage("NOT printed on screen");
6. }

All code that happens at the line where I call NatCam.Camera and all code after it doesn't get called anymore.
Mind that this does work in the Editor and on iOS. Have I forgot to initialize something specific for Android?
Hope you could help.

Kind regards

 

You never set NatCam.Camera before accessing it and checking its exposure bias. This should lead to a null reference exception, which would explain why your other code stops.

 

Ok, I'll check this out today, but why does this work on iOS and Editor without null ref exception?
I even tried following output, but this didn't output anything and all code after it was ignored.

Code (CSharp):

1. GameManager.Instance.StatisticsManager.PushMessage("does natcam camera excist = " + (NatCam.Camera != null));

 

On iOS, it defaults to camera 0. I have fixed this so that it will throw a nullref instead. On the Editor, I believe it does the same.

Can you email me the logs from logcat?

 

The third part of the NatCam Tutorial Series is up! Check out NatCam Tutorial Series 3: Photos.

 

Hello Lanre,
With the lastest version I have a strange bug on android (only from time to time, not always).
=> When I take a photo the preview becomes very pixalised, and the recorded photo aswell.

I have this :
AndroidJavaExeption : "Camera is being used after Camera.release() was called"
AndroidJavaExeption : SetParameters failed

If I try to take other photos then, the issue finally disapear.

Do you have any idea of what's going on ?

Edit : It's because of the PinchCam.cs... I will send you an e-mail

Seb

 

Hi. I'm having some issues with an older version (a beta you sent me by email, 1.4 I think). We still have not updated to the latest as the API has changes considerably. When we generate a build without Development symbols, the image returned by NatCam.Preview is all white. Any idea? I can send you more details by email if you need.

 

I sent a mail but here seems to be a better place for other to benefit from the answer.

We need to scan a QR code on ios (from ipad 2) and android devices (from Android 4.0.1).
We already have the code to scan the QR code using the library XZing, but we need to feed it with pixel bytes from the webcam texture (32 bits for each pixel).

Using Unity default WebcamTexture we met 2 issues :
- The autofocus does not work, ending up with blurred textures.
- The light correction does not work, ending up with very dark image on some devices.

Here 3 questions

• Which version of your plugin do you recommend
• We saw that to work on Samsung 7 we should disable the professional feature, then can NatCam.Preview still be used ?
• If not, would it work to use a unity Camera to take a screenshot of the screen and use this screenshot ?
• In this case, should we still go for the professional version of the plugin or stick to NatcamCode ?
  

Thanks !

 

What device does this occur on? I am very sure that this bug is specific to the device. Can you send me the full logs from logcat?

Send me the logs. I am not sure what could cause this.

 

Can you send me full logs? I typically don't provide support for older versions of NatCam, so I strongly advise you to update to 1.5.

 

NatCam Pro. Since you need access to the preview data, you will need NatCam Pro which has the NatCam.PreviewBuffer API.

NatCam.Preview can still be used (it is not affected by the preview data pipeline). But disabling the ReadablePreview flag means that you will not be able to get the preview data to send to ZXing, which would seem to pose a problem for your application. The workaround I will recommend is lowering the resolution of the camera preview when your app is being run on the S7 (so that you can keep the ReadablePreview flag on and maintain a decent framerate on the S7).

The NatCam.PreviewBuffer API provides a handle to the native preview buffer (void*, not a managed byte[]). You can use the API's in the System.Runtime.InteropServices.Marshal class to copy the data into a byte[] or Color32[], then use it.

 

I mentioned that we need need to support Android 4.0.1 but it seems it is not possible with NatCam Pro. Am I wrong ?

 

Actually I thought of another way to fix our issue,
Maybe we can only use the Core part of your plugin, which seems to be compatible with Android 4.0.1, and instead of getting access to the camera preview, take a screenshot of the texture in unity.

What do you think ?
If this would work, then we could still use the plugin.

 

Thanks, initializing the natcam camera before using it, fixed it indeed!
On our way of porting our project to iOS, we came across the problem that the captured picture gets stretched on Android. To be sure it wasn't some forgotten setting or param, I've opened the Minigram scene in your sample projects. When capturing the picture, the picture gets stretched according to the live feed.

 

Hi, there are some troubles with using natcam and vuforia in one project, when i start scene whith natcam after the scene with vuforia it throws exeption:

AndroidJavaExeption: java.lang.RuntimeExeption: getParametres faild(empty parametres)

 

This is possible. The drawback is that it will not be as fast as NatCam.

NatCam Pro requires API level 18+ on Android because of the recording API's. You can modify the AndroidManifest.xml file in NatCamProfessional.aar (in Assets>NatCam>Professional>Plugins>Android) to reduce the API level requirement. An aar file is practically a zip file, so unzip it using the command line.

 

I will check MiniCam. In your project, make sure that you are scaling your UI panel according to your photo when you apply the photo on it. You can use the NatCamPreview component to do this.

 

You must make sure that you completely release Vuforia before starting NatCam (and vice versa). I believe that the call for Vuforia is this, but I'm not too sure:

Code (CSharp):

1. VuforiaUnity.Deinit();

 

Hello, I sent you the logs on your mail with the full scene.
the bug occurs on my new OnePlus 5, but also on another android phones of my colleague.
But maybe the code of my PinchCam.cs enters in conflict with MiniCam.cs ?

You have the full scene on your mail.

Seb

 

We experienced a crash on LG "Optimus G" when NatCam try to access the camera, Android version 7.1.1, Unity 5.6.1p4, NatCam professional 1.5.

I attached the log
Could you tell us if you have any hint ?

 

We also have a weird bug on Samsung S 4. The Unity UI preview (RawImage) is completly distorted. But we can still scan a QR code which means the camera preview is probably right. We start NatCam.Play not in start of the scene, but few seconds after the scene started. Could it be the reason ? It worth noting it does not happen on some other device we can test on. See the attached picture.

 

Are you using PinchCam and MiniCam in the same scene? Make sure that you are not doing so. I am looking into the logs now.

 

You seem to be forcing OpenGL ES2 on the device even though it supports ES3. NatCam will not run with ES2 on a device that supports ES3 (due to some Android restrictions). In Player Settings, give ES3 higher precedence over ES2 (or leave the 'Automatic Graphics API' option on).

 

What device does this happen on (model number)? I am not sure what could be causing it, so send me the logs.

 

The model is GT-I9500
Android 4.2.2
Attached the logs.

Here the code we run to start the camera and barcode detection :

Code (CSharp):

1. using System;
2. using System.Collections;
3. using NatCamU.Core;
4. using NatCamU.Core.UI;
5. using NatCamU.Core.Utilities;
6. using NatCamU.Extended;
7. using UnityEngine;
8.  
9. public class NatCamQRCodeReader : MonoBehaviour
10. {
11.     [Header("Preview")] [SerializeField] private NatCamPreview _preview;
12.     [SerializeField] private NatCamCustomFocuser _focuser;
13.  
14.     [Header("Debugging")] [SerializeField] private Switch _verbose;
15.  
16.     private bool _mirrored;
17.  
18.     public event Action<string> OnBarcodeDeteted;
19.  
20.     public void StartCamera()
21.     {
22.         StartCoroutine(StartCameraCoroutine());
23.     }
24.  
25.     private IEnumerator StartCameraCoroutine()
26.     {
27.         yield return Application.RequestUserAuthorization(UserAuthorization.WebCam);
28.         if (!Application.HasUserAuthorization(UserAuthorization.WebCam))
29.         {
30.             CameraAccessUI.LoadCameraAccessUI().DisplayPanel();
31.             yield break;
32.         }
33.  
34.         Debug.Log("started nat cam qr code reader.");
35.         NatCam.DetectBarcode(OnBarcodeDetected, BarcodeFormat.QR, false);
36.  
37.         NatCam.Verbose = _verbose;
38.         NatCam.Camera = DeviceCamera.RearCamera ?? DeviceCamera.FrontCamera;
39.  
40.         if (NatCam.Camera == DeviceCamera.FrontCamera)
41.         {
42.             _mirrored = true;
43.         }
44.  
45.         if (!NatCam.Camera)
46.         {
47.             Utilities.LogError(
48.                 "Both Rear Camera and Front Camera returned null. The device camera may not be supported or authorized.");
49.             CameraAccessUI.LoadCameraAccessUI().DisplayPanel();
50.             yield break;
51.         }
52.  
53.         _focuser.SetFocus();
54.         NatCam.Camera.ExposureMode = ExposureMode.AutoExpose;
55.  
56.         NatCam.Camera.SetPreviewResolution(ResolutionPreset.MediumResolution);
57.         NatCam.Camera.SetFramerate(FrameratePreset.Default);
58.         NatCam.OnStart += NatCam_OnStart;
59.         NatCam.Play();
60.     }
61.  
62.     private void NatCam_OnStart()
63.     {
64.         if (_preview != null)
65.         {
66.             _preview.Apply(NatCam.Preview, _mirrored ? Orientation.Mirror : Orientation.Rotation_0);
67.         }
68.         else
69.         {
70.             Utilities.LogError("Preview RawImage has not been set");
71.         }
72.  
73.         Utilities.LogVerbose("Preview started with dimensions: " +
74.                              new Vector2(NatCam.Preview.width, NatCam.Preview.height));
75.     }
76.  
77.     private void OnBarcodeDetected(Barcode barcode)
78.     {
79.         Debug.Log("barcode detected");
80.      
81.         if (OnBarcodeDeteted != null)
82.         {
83.             OnBarcodeDeteted(barcode.data);
84.         }
85.     }
86.  
87.     private void OnDestroy()
88.     {
89.         NatCam.Release();
90.     }
91. }

 

Hello Lanre,
Hum yes I'm using PinchCam and Mini Cam in the same scene :/
I was inpired by your NatCam-Test-Suite1.5 where the miniCam GO has miniCam.cs and Preview GO has ZoomCam.cs

What is the right thing to do ? I don't know if you downloaded the scene I send you by mail but maybe it will be easier to understand what is going on

 

Hi, when i start scene with NatCam on galaxy S7, the application crushed, how can i fix this

 

How to get Color32[] from NatCam.PreviewBuffer on Android or IOS ? I am not sure of what is the format returned by NatCam.PreviewBuffer. Is it RGBA byte array ? I would like to convert it to Color32[].

 

After testing myself, it seems that it returns a byte[] with pixelX.r = byte[X * 4] , pixelX.g = byte[X * 4 +1] , pixelX.b = byte[X * 4 + 2 ] , pixelX.a = byte[X * 4 + 3].

Let me know if I am wrong, Thanks !

 

There's nothing logged, no error or warning, and initialisation looks fine. I try to upgrade, and come back to you if it's still not working.

 

Hello! I am interested in using NatCam for a project. We need to disable automatic white balancing. Is that something that is exposed by the NatCam API? If it isn't, is that something that we could get the source for on Android and iOS, so we can add in the code ourselves?

Thanks!

 

We also have a bug on Kindle Fire HD 8.9 similar to Samsung 4 with the front camera (this device only have front camera)
The Unity UI preview (RawImage) is not aligned correctly. But contrarily to Samsung 4, the texture display on it is much closer to what the camera see.
Attached are logs and screenshot