NatDevice - Media Device API

Problem solved. I used another asset solution which uses the native camera app. That better suits my needs

 

Lanre, I'm migrating my app to android and I am seeing an IllegalStateException, android.hardware.camera2.impl.CaptureSessionImpl.checkNotClosed
android.hardware.camera2.impl.CaptureSessionImpl.capture

I'm using Unity 2019.2.9f1, min target Android 8.0, target Android 9.0.

I use NatCam for my AR Main camera, but I also have "mini cam", which I adapted from your mini cam example. My biggest unknown is how to I pair NatCam to be th AR camera, along with my slide up mini-cam. I think this is where my problem stems.

For example, if you combined Gray cam with Mini-cam, is there anything you would do to use them together?

My AR camera is a render texture, the mini cam slide up also has a render texture. Do you see anything in this implementation that might be causing my problem:

Code (CSharp):

1.     public void SetupPhotoControls(IPhotoImageModelView thumbnailImage)
2.     {
3.         cameras = _natCam.cameras;
4.         // Pick camera
5.         for (var i = 0; i < cameras.Length; i++)
6.             if (cameras[i].IsFrontFacing == useFrontCamera)
7.             {
8.                 activeCamera = i;
9.                 break;
10.             }
11.         if (activeCamera == -1)
12.         {
13.             Debug.LogError("Camera is null. Consider using " + (useFrontCamera ? "rear" : "front") + " camera");
14.             return;
15.         }
16.  
17.         if (_natCam.previewTexture != null)
18.         {
19.             rawImage.texture = _natCam.previewTexture;
20.         }
21.  
22.         // only init thumb view once
23.         if (_thumbnailImageModelView == null)
24.             _thumbnailImageModelView = thumbnailImage;
25.  
26.  
27.         OnViewPhoto();
28.     }
29.  
30.     void OnViewPhoto()
31.     {
32.         rawImage.texture = _natCam.previewTexture;
33.  
34.         // Disable the check icon
35.         checkIco.gameObject.SetActive(false);
36.  
37.         // Disable the Use photo buttons
38.         if (_doneBar != null)
39.             _doneBar.SetActive(false);
40.  
41.         // Enable the photo capture buttons
42.         if (_controlBar != null)
43.             _controlBar.SetActive(true);
44.  
45.     }
46.  
47.  

Thanks!

EDIT: The AR camera is working fine. I don't see the above error unless I click the "capture photo" button.

 

Don't use an AR camera and NatCam simultaneously; only one service can have access to the cameras. This is an OS restriction on all the platforms that NatCam runs on. It extends to anything that might be using the camera: NatCam, WebCamTexture, AR SDK's, and so on.

 

Thanks for the super fast response Lanre. So is it as simple as "stopPreview" on the AR camera, and then startPreview on the Mini Cam, until it's dismissed, and then restartPreview on AR camera?

EDIT: In case I wasn't clear, I'm using NatCam for AR via:

cameras[activeCamera].StartPreview(OnStart);

 

I don't recommend merging example scenes. Instead, build a scene that does exactly what you need it to do. In any case, only have one camera running at any time.

 

Natcam version 2.3.1 crashing onResume, happens 1 out of 3 times

Code (CSharp):

1. 2020-01-01 20:35:01.546 31608-32172/com.xxxxx.zzz E/CameraCaptureSession: Session 0: Failed to create capture session; configuration failed
2. 2020-01-01 20:35:01.548 31608-381/com.xxxxx.zzz E/BufferQueueProducer: [ImageReader-480x640f1m3-31608-10] disconnect: not connected (req=1)
3. 2020-01-01 20:35:01.548 31608-381/com.xxxxx.zzz W/libEGL: EGLNativeWindowType 0x7494644010 disconnect failed
4. 2020-01-01 20:35:01.550 31608-383/com.xxxxx.zzz E/BufferQueueProducer: [ImageReader-3024x4032f1m3-31608-11] disconnect: not connected (req=1)
5. 2020-01-01 20:35:01.550 31608-383/com.xxxxx.zzz W/libEGL: EGLNativeWindowType 0x7494515010 disconnect failed
6. 2020-01-01 20:35:01.551 31608-32172/com.xxxxx.zzz E/Unity: NatCam Error: Failed to create capture session to start preview
7.     android.hardware.camera2.CameraAccessException: CAMERA_DISCONNECTED (2): waitUntilIdle:1323: Camera device no longer alive
8.         at android.hardware.camera2.CameraManager.throwAsPublicException(CameraManager.java:814)
9.         at android.hardware.camera2.impl.ICameraDeviceUserWrapper.waitUntilIdle(ICameraDeviceUserWrapper.java:179)
10.         at android.hardware.camera2.impl.CameraDeviceImpl.waitUntilIdle(CameraDeviceImpl.java:1157)
11.         at android.hardware.camera2.impl.CameraDeviceImpl.configureStreamsChecked(CameraDeviceImpl.java:441)
12.         at android.hardware.camera2.impl.CameraDeviceImpl.createCaptureSessionInternal(CameraDeviceImpl.java:667)
13.         at android.hardware.camera2.impl.CameraDeviceImpl.createCaptureSession(CameraDeviceImpl.java:515)
14.         at api.natsuite.natcam.CameraDevice$2.onOpened(CameraDevice.java:475)
15.         at android.hardware.camera2.impl.CameraDeviceImpl$1.run(CameraDeviceImpl.java:148)
16.         at android.os.Handler.handleCallback(Handler.java:873)
17.         at android.os.Handler.dispatchMessage(Handler.java:99)
18.         at android.os.Looper.loop(Looper.java:214)
19.         at android.os.HandlerThread.run(HandlerThread.java:65)
20.      Caused by: android.os.ServiceSpecificException: waitUntilIdle:1323: Camera device no longer alive (code 4)
21.         at android.os.Parcel.createException(Parcel.java:1980)
22.         at android.os.Parcel.readException(Parcel.java:1934)
23.         at android.os.Parcel.readException(Parcel.java:1884)
24.         at android.hardware.camera2.ICameraDeviceUser$Stub$Proxy.waitUntilIdle(ICameraDeviceUser.java:608)
25.         at android.hardware.camera2.impl.ICameraDeviceUserWrapper.waitUntilIdle(ICameraDeviceUserWrapper.java:177)
26.         at android.hardware.camera2.impl.CameraDeviceImpl.waitUntilIdle(CameraDeviceImpl.java:1157)
27.         at android.hardware.camera2.impl.CameraDeviceImpl.configureStreamsChecked(CameraDeviceImpl.java:441)
28.         at android.hardware.camera2.impl.CameraDeviceImpl.createCaptureSessionInternal(CameraDeviceImpl.java:667)
29.         at android.hardware.camera2.impl.CameraDeviceImpl.createCaptureSession(CameraDeviceImpl.java:515)
30.         at api.natsuite.natcam.CameraDevice$2.onOpened(CameraDevice.java:475)
31.         at android.hardware.camera2.impl.CameraDeviceImpl$1.run(CameraDeviceImpl.java:148)
32.         at android.os.Handler.handleCallback(Handler.java:873)
33.         at android.os.Handler.dispatchMessage(Handler.java:99)
34.         at android.os.Looper.loop(Looper.java:214)
35.         at android.os.HandlerThread.run(HandlerThread.java:65)
36. 2020-01-01 20:35:01.552 31608-32172/com.xxxxx.zzz E/Unity: NatCam Error: Failed to configure capture session to start preview
37. 2020-01-01 20:35:01.553 31608-32172/com.xxxxx.zzz E/AndroidRuntime: FATAL EXCEPTION: NatCam Operations Thread
38.     Process: com.xxxxx.zzz, PID: 31608
39.     java.lang.NullPointerException: Attempt to invoke virtual method 'void api.natsuite.natcam.FrameRenderer.release()' on a null object reference
40.         at api.natsuite.natcam.CameraDevice.stopPreview(CameraDevice.java:534)
41.         at api.natsuite.natcam.CameraDevice$2.onDisconnected(CameraDevice.java:505)
42.         at android.hardware.camera2.impl.CameraDeviceImpl$7.run(CameraDeviceImpl.java:244)
43.         at android.os.Handler.handleCallback(Handler.java:873)
44.         at android.os.Handler.dispatchMessage(Handler.java:99)
45.         at android.os.Looper.loop(Looper.java:214)
46.         at android.os.HandlerThread.run(HandlerThread.java:65)
47. 2020-01-01 20:35:01.568 31608-32752/com.xxxxx.zzz E/native: loge.h:15 CameraMetrics: OpenCamera: 87.807808ms

 

I'll look into this. Thanks for brining it to my attention.

 

Hello,
I experience two issues.
First, on my test devices (Saemsung galaxy S8/S9 and LG V50) calling

StopPreview()

freezes app.
I tested MiniCam example without any modification,

StopPreview()

in

SwitchCamera()

stops app.

And second, only in LG V50 preview works fine but it's too dark when I take a photo. Almost no light. Maybe some low light issue? How can I solve these?

 

Can you share the full logs from logcat? As for the exposure in photos, have you tried just enabling flash?

 

Hi,

I have the same issue as wcchoe with calling StopPreview on Samsung S8 during testing MiniCam example. Used NatCam v.2.3.1. Logs in attachment.

 

Thanks for this, I'll check it out.

 

Thank you for support Ultra Wide Camera.
But I got Xcode error.
I just open unity ver1.6f1 and import Nat Cam Version2.3.1 And build.

-----------------------------------------------------------------
Undefined symbols for architecture arm64:
"_AVCaptureDeviceTypeBuiltInUltraWideCamera", referenced from:
+[NCCameraDevice cameras] in libNatCam.a (NCCameraDevice.o)
"___isPlatformVersionAtLeast", referenced from:
+[NCCameraDevice cameras] in libNatCam.a (NCCameraDevice.o)
ld: symbol(s) not found for architecture arm64
-----------------------------------------------------------------

 

What version of Xcode are you running on? And what is your deployment target? You need Xcode 11 at least, and your deployment target should be at least iOS 11.

 

sorry I used Xcode 10.1.
I'm going to upgrade to 11. Thank you.

 

Hi Lanre,

I'm experiencing the same issue as wcchoe and meddmi with NatCam 2.3.1, calling StopPreview will freeze the app on multiple Android devices.
In addition of that, development builds will show a drastic drop of performance (very low framerate on preview) compared to the previous version. Is this expected behaviour with development builds ? Thank you.

 

Can you share the full logs from logcat? As for the performance drop, can you profile your app in the Unity profiler? I'm not sure why you are facing a drop.

 

Here the full logcat logs, tested on my honor x8.

 

Interestingly enough the logs don't seem to indicate any error or crash. I'll look more into this. Thanks for the logs.

 

Thanks for the help.
Concerning the development build performance issue, I can't reproduce it with the MiniCam sample example, only in my project after updating. Looks like I need more investigations. Could it be related with NatCorder somehow ? I'm using NatRender from NatCorder package since I'm using both in my project.

 

We were very glad that the NatMic issue (recorded volume too low) is now fixed, but now we're experiencing the same issue as wcchoe and meddmi with NatCam 2.3.1 with the app freezing. It doesn't log anything.
Also the camera is upside down and we didn't change anything in the code and only updated the library.

Tested on Samsung S8 on Android 8, Unity 2019.2.3f1

 

I saw in the review that Android's preview frame speed was slow.

I want to use 60-120fps preview on Android.

Is this possible?

* Galaxy S7~S10
* preview resolution 1280x720

 

Hello!

I was also having issues with the arm64 undefined symbols on iOS.

The fix for us was to use Unity 2019.3. There seems to be an issue with Metal in 2019.2.

2019.3 is a pre-release version, but it looks like 2019.2 projects migrate very successfully to it. We're actually still using 2019.2 (everything works fine in the editor) but have Cloud Build target 2019.3, and everything works great.

Just thought I'd share!

(Oh, also, when we upgraded to latest NatCam, it seemed to leave some of the NatCamU.Core stuff lingering in the assemblies. We had to delete and freshly import NatCam for VS to recognize that that library was no longer available.)

 

I don't think it could be related to NatRender. Both API's are supposed to use the same version of NatRender.

 

Fix coming early next week! Sorry for the inconvenience.

 

You should be able to hit a 60FPS preview easily. You can't hit more than that because the preview framerate is effectively capped at the Unity framerate, which itself can never be higher than 60.

 

@Lanre : Thanks for the reply earlier! I've got two pieces of feedback for NatCorder.

1 - Implementing a ScreenReader script that simplifies the process of capturing all of the screen's current pixels (all camera's combined post-render for that frame). I'm not sure if it would be performant but this might be a good alternative way of capturing screen space UI for some scenarios.

2 - Providing an example in your documentation for recording from a render texture, and from a Texture2D. You mention in your documentation that it's possible, but since there isn't a quick example of this, your bound to get a lot of repeat questions.

 

NatCorder comes with a ScreenInput script, but it isn't complete as it requires Unity 2019. I'm considering completing the implementation and requiring Unity 2019.

The WebCam example already demonstrates recording a WebCamTexture, and you can drop in a Texture2D in place of the WebCamTexture and it will work. As for a RenderTexture, I'll consider adding something useful.

 

Thanks @Lanre, for my scenario either having a finished ScreenRecorder script or a working example of recording from a RenderTexture will work for my needs as they should help accomplish the same thing (render world objects + UI)

 

In the mean time check out the CameraInput implementation. It already does this.

 

Hi,
Before the last update (2.3.1) my code :

cameras[activeCamera].PreviewResolution = new Resolution { width=1280, height=720 };
cameras[activeCamera].PhotoResolution = new Resolution { width=1920, height=1080 };

worked correctly and now i have the error : Cannot implicitly convert type 'UnityEngine.Resolution' to '(int width, int height)'

same thing with :

Vector2 relativePoint = new Vector2(point.x / size.x, point.y / size.y);
cameras[activeCamera].FocusPoint = relativePoint;

error : Cannot implicitly convert type 'UnityEngine.Vector2' to '(float x, float y)'

Any help plz ?

 

First, you need Unity 2018.3 at least. Your code would then become:

Code (CSharp):

1. device.PreviewResolution = (1280, 720);

 

Thx! it's ok but Flash still doesn't work on Android.

Natcam 2.3.1
Unity 2019.1.14f1

 

Hi!

Natcam does not approve valid photo resolutions, CapturePhoto() only works with preview resolutions. Edit: (only on android!)
This is not the case with newer phones, as it (partially) supports some photo resolutions. For older phones, this ceiling can be up to 960x720.

Code (CSharp):

1. public void TakePhoto()
2. {
3.             if (camDevice == null || !camDevice.IsRunning || takingPhoto) return;
4.             takingPhoto = true;
5.             camDevice.PhotoResolution = (3264,2448);
6.             camDevice.CapturePhoto(CapturedPhoto);
7. }
8.  
9. void CapturedPhoto(Texture2D photo)
10. {
11.             Debug.Log( string.Format("real photo res: {0}x{1}", photo.width, photo.height) );
12.             Texture2D.Destroy(photo);
13.             photo = null;
14.             StartCoroutine(Timeout(2f, delegate () {
15.                 takingPhoto = false;
16.             }));
17. }

DebugLog:

real photo res: 960x720

on this phone.


Is this fixable?

Thank you for your help.

 

Can you share the logs from logcat?

 

This is entirely determined by the device. There isn't anything we can do to fix it in our code.

 

Thanks for the answer. I would like to ask you more on this topic.

Why?
All resolutions are returned by CameraCharacteristics.SCALER_STREAM_CONFIGURATION_MAP. We filter this depending on what it is used for.

When deciding on a possible preview resolution, we filter to ImageFormat.YUV_420_888.
And when we check the photo resolution, we filter to ImageFormat.JPEG.

I don't think this is the case with natcam right now.
This is the only explanation why the higher resolution photo works on the samsung A50 and not on the samsung A3 (2016).

Spoiler: samsung SM-A505FN, samsung, a50

------------ Camera 0------------

Facing: LENS_FACING_BACK
deviceLevel = INFO_SUPPORTED_HARDWARE_LEVEL_LIMITED
Available capabilities: [BACKWARD_COMPATIBLE, BURST_CAPTURE, CONSTRAINED_HIGH_SPEED_VIDEO]
Output formats: [JPEG, PRIVATE, YUV_420_888]

JPEG sizes:
[5760x4320@(50.000, 0.0),
5760x3240@(50.000, 0.0),
5760x2664@(50.000, 0.0),
4304x4304@(50.000, 0.0),
3264x2448@(50.000, 0.0),
3024x3024@(50.000, 0.0),
2880x2156@(50.000, 0.0),
2880x1620@(50.000, 0.0),
2576x1932@(50.000, 0.0),
2560x1440@(50.000, 0.0),
1920x1080@(33.332, 0.0),
1440x1080@(33.332, 0.0),
1088x1088@(33.332, 0.0),
1280x720@(33.332, 0.0),
960x720@(33.332, 0.0),
800x450@(33.332, 0.0),
720x720@(33.332, 0.0),
720x480@(33.332, 0.0),
640x480@(33.332, 0.0),
352x288@(33.332, 0.0),
320x240@(33.332, 0.0)]

YUV_420_888 sizes:
[5760x4320@(33.332, 0.0),
5760x3240@(33.332, 0.0),
5760x2664@(33.332, 0.0),
4304x4304@(33.332, 0.0),
3840x2160@(33.332, 0.0),
3264x2448@(33.332, 0.0),
3264x1836@(33.332, 0.0),
3024x3024@(33.332, 0.0),
2880x2156@(33.332, 0.0),
2880x1620@(33.332, 0.0),
2576x1932@(33.332, 0.0),
2560x1440@(33.332, 0.0),
2336x1080@(33.332, 0.0),
1920x1080@(33.332, 0.0),
1440x1080@(16.666, 0.0),
1088x1088@(16.666, 0.0),
1024x768@(16.666, 0.0),
1280x720@(16.666, 0.0),
960x720@(16.666, 0.0),
800x450@(16.666, 0.0),
720x720@(16.666, 0.0),
720x480@(16.666, 0.0),
640x480@(16.666, 0.0),
480x320@(16.666, 0.0),
352x288@(16.666, 0.0),
320x240@(16.666, 0.0),
256x144@(16.666, 0.0),
176x144@(16.666, 0.0)]

Spoiler: samsung SM-A310F, samsung, a3xelte

------------ Camera 0------------

Facing: LENS_FACING_BACK
deviceLevel = INFO_SUPPORTED_HARDWARE_LEVEL_LEGACY
Available capabilities: [BACKWARD_COMPATIBLE]
Output formats: [JPEG, PRIVATE, YUV_420_888, YV12]

JPEG sizes:
[4128x3096@(0.000, 1107.4),
4128x2322@(0.000, 880.6),
3264x2448@(0.000, 767.3),
3264x1836@(0.000, 625.5),
3088x3088@(0.000, 877.0),
2160x2160@(0.000, 531.3),
2048x1536@(0.000, 423.3),
2048x1152@(0.000, 367.5),
1920x1080@(0.000, 347.2),
1440x1080@(0.000, 310.4),
1280x720@(0.000, 265.4),
960x720@(0.000, 249.1),
640x480@(0.000, 221.8),
320x240@(0.000, 205.5)]

YUV_420_888 sizes:
[960x720,
880x720,
720x720,
720x480,
640x480,
352x288,
320x240,
176x144]

 

You may have miss my question.

I said the 60 number of camera preview texture changes.
(it isn't unity rendering)

I just got an onFrame callback 30 times in a second.

It got horrible.

You probably didn't use like "CameraConstrainedHighSpeedCaptureSession"


OnFrame is called 60 times on the iOS. (iPhone6s,iPhoneX)
but Android is called 30 times. (GalaxyS7, GalaxyS10+)
(Unity 2018.4.5f1)


The code is similar to the following.

Code (CSharp):

1.  
2. CameraDevice cam = null;
3. foreach (var c in cams)
4. {
5.     if (!c.IsFrontFacing) cam = c;
6. }
7. cam.PreviewResolution = new Resolution { width = 1280, height = 720 };
8. cam.Framerate = 60;
9. cam.StartPreview(OnStart, OnFrame, ScreenOrientation.LandscapeLeft);
10.  

 

NatCorder uses SurfaceTexture.class (which AFAIK is synonymous with YUV_420) for both preview and photo. This is necessary because requesting JPEG data would make photo capture very slow, as we would have to first decode the JPEG, upload it to the GPU, perform orientation corrections, readback the pixel data from the GPU, then send it to Unity as a Texture2D. When routing both to a SurfaceTexture, we get the image data on the GPU immediately.

 

NatCam does not use the high speed capture session. NatCam can only set a framerate that is included in CameraCharacteristics.CONTROL_AE_AVAILABLE_TARGET_FPS_RANGES.

 

Hi Lanre, we'd like to update to latest version. Is there a an update on this?

 

I see a same issue on Pixel 3a... App freezes when calling StopPreview()

 

I am working to have the update ready by early next week.

 

Hey @Lanre, any idea what could cause this on an Android build? (Happens essentially when I load the scene that automatically starts up NatCam. Everything works fine in the Unity Editor targeting Android, and in iOS Editor+Build)

I didn't see any relevant logcat messages aside from this one:

E/AndroidRuntime: FATAL EXCEPTION: NatCam Processing Thread
Process: com.company.appname, PID: 9132
java.lang.UnsatisfiedLinkError: dalvik.system.PathClassLoader[DexPathList[[zip file "/data/app/com.company.appname-byR0k5cnZKPIewe_KZ1hyg==/base.apk"],nativeLibraryDirectories=[/data/app/com.company.appname-byR0k5cnZKPIewe_KZ1hyg==/lib/x86, /data/app/com.company.appname-byR0k5cnZKPIewe_KZ1hyg==/base.apk!/lib/x86, /system/lib, /system/product/lib]]] couldn't find "libNatRender.so"
at java.lang.Runtime.loadLibrary0(Runtime.java:1067)
at java.lang.Runtime.loadLibrary0(Runtime.java:1007)
at java.lang.System.loadLibrary(System.java:1667)
at api.natsuite.natrender.Unmanaged.<clinit>(Unmanaged.java:17)
at api.natsuite.natrender.Unmanaged.baseAddress(Native Method)
at api.natsuite.natcam.FrameRenderer$4.onImageAvailable(FrameRenderer.java:131)
at android.media.ImageReader$ListenerHandler.handleMessage(ImageReader.java:798)
at android.os.Handler.dispatchMessage(Handler.java:107)
at android.os.Looper.loop(Looper.java:214)
at android.os.HandlerThread.run(HandlerThread.java:67)
​

I'm gonna mess with my build settings, and will edit this message if I fix it before I get a response.

Current (relevant, Android) settings:
Color Space Gamma, Auto Graphics is OFF, Graphics APIs are Vulkan then OpenGLES3, not using any of the ES3.1+ stuff, Multithreaded Rendering, Static and Dynamic Batching are all on, minimum Android 5.0 'Lollipop' (API level 21), target API level Automatic (highest installed), IL2CPP scripting backend, NET Standard 2.0... Also using the Cardboard XR SDK via GoogleVR Plugin for Unity.

 

NatSuite API's don't support x86. On Android, they only support armv7 and arm64.

 

Thanks! Saved me a lot of hunting and random A-B testing!

 

Email me with your invoice number for a fix to the camera freeze issue on Android.

The fix has been added to the newest API, NatDevice. NatDevice succeeds both NatCam (camera API) and NatMic (microphone API). It will be submitted to the Asset Store as a NatCam update within a week.

 

Any news on this issue?

 

The updated build is mostly ready. Email me with your invoice number and I'll send it to you.

 

Hi there! I'm having an issue where only iPhone 7 devices aren't starting preview. Other iOS devices seem fine (iPhone 6 for example). I can successfully call StartPreview on a found camera. But when this line in CameraDeviceIOS is called:

Code (CSharp):

1. device.StartPreview((int)rotation, FrameDelegate, (IntPtr)GCHandle.Alloc(handler, GCHandleType.Normal));

It never calls handler and thus never calls my OnStart callback. Again, all other iOS devices we've tried seem to work. Just seems to be an iPhone 7 and iPhone 7 Plus problem. The tricky thing is that I create a clean project with just the MiniCam example and it works fine on iPhone7. But I am unable to figure out what the difference is. Thoughts?

 

Hm, I think I figured something out. I was setting the preview resolution before calling StartPreview.

Code (CSharp):

1. activeCameraDevice.PreviewResolution = ( cameraWidth, cameraHeight );
2. activeCameraDevice.StartPreview(OnStart);

If I comment out the set PreviewResolution line, it works (OnStart callback gets called). Again, this only seems to be a problem on iPhone 7 devices.