NatDevice - Media Device API

Can you share the editor crash logs?

 

Send you a pm.

 

These logs don't seem to contain a reason for the crash. When the editor crashes, there should be a window that asks you if you want to report the crash. Generate the report, and send me the information in there. That will contain the reason for the logs.

 

I'm having an issue with the MiniCam sample (NatDevice) on a Samsung Note 10+ (it works on Note 5). It seems that MediaDeviceQuery.CameraDevices() is not able to enumerate devices. Camera permission has been granted to the app.

 

I have a fix for this in the upcoming update.

 

@Lanre @Mukaujoka
We're seeing the same issue with the editor crashing when showing the camera. It's because of the missing camera permission but it's not actually requesting permission, and there isn't a way of manually adding it. We've tried completely reinstalling unity / unity hub etc.

 

Are you running macOS? If so, what version? As far as we've seen it doesn't have anything to do with permission.

 

Hi, I'm also having a crash when I start the Minicam in the Editor. Unity has Camera permissions but I'm getting this in the console..

tccd Prompting policy for hardened runtime; service: kTCCServiceMicrophone requires entitlement com.apple.security.device.audio-input but it is missing for RESP:{ID: com.unity3d.unityhub, PID[16538], auid: 501, euid: 501, responsible path: '/Applications/Unity Hub.app/Contents/MacOS/Unity Hub', binary path: '/Applications/Unity Hub.app/Contents/MacOS/Unity Hub'}, REQ:{ID: com.unity3d.UnityEditor5.x, PID[20689], auid: 501, euid: 501, binary path: '/Applications/Unity/Hub/Editor/2019.3.5f1/Unity.app/Contents/MacOS/Unity'}
default 14:36:34.463262 +0000

tccd Policy disallows prompt for RESP:{ID: com.unity3d.unityhub, PID[16538], auid: 501, euid: 501, responsible path: '/Applications/Unity Hub.app/Contents/MacOS/Unity Hub', binary path: '/Applications/Unity Hub.app/Contents/MacOS/Unity Hub'}, REQ:{ID: com.unity3d.UnityEditor5.x, PID[20689], auid: 501, euid: 501, binary path: '/Applications/Unity/Hub/Editor/2019.3.5f1/Unity.app/Contents/MacOS/Unity'}; access to kTCCServiceMicrophone denied

error 14:36:34.463596 +0000 Unity This app has crashed because it has a hardened runtime and attempted to access privacy-sensitive data without an entitlement indicating its intent to access this data. The app must have the 'com.apple.security.device.audio-input' entitlement.

You mention manually adding permissions so help in the interim, how does one do this?

 

I'm running OSX 10.14.6 btw. Never been asked to give access to Unity, or the Hub

 

We have two macs, one it's working on - and that one did request permission for the mic. You can see that Unity Hub has mic and camera permissions in the system prefs. The one it's not working on doesn't have that permission and it's not requesting it whatever we do. Both Macs are on 10.14.6.

 

This is very useful information, thanks. I'm looking into it now.

 

Just to follow up, I think the best thing to do is to reach out to Unity. It's up to Unity Hub to declare its entitlements (I can't declare them in the NatDevice bundle).

 

This is coming really late, but:

Read about the NatDevice Upgrade.

 

Hi,
On cameras(web cam) connected in windows, do you have a full control support and a unique id of the camera device (the id of the camera and not index that may be changed)?

 

`WebCameraDevice.uniqueID` returns this property.

 

Did someone already do this? I'd drop them a quick message otherwise.

 

Hi, How do I set the exposure switch from Auto to manual, and manual to Auto with Android ?
Do I use "CameraDevice.ExposureLock"?
Also what does "CameraDevice.ExposurePoint" do and how do I use it?

 

Hello, I was using NatCam with OpenCV to develop some cool image-processing apps. Looking forward to working with NatDevice. Is there any way to switch between the JPEG and RAW_SENSOR formats available in the Camera2 API using NatDevice?

 

NatDevice doesn't provide manual exposure control (ISO and shutter speed). Exposure lock is used to lock the auto exposure bias so that it doesn't change if the illumination changes. `exposurePoint` sets the exposure point of interest, the same way `focusPoint` sets the focus point of interest.

 

NatDevice (and NatCam) always provide the preview (and photo) data as an RGBA32 pixel buffer. We don't expose raw bayer data because it is out of scope for us (and pretty specific to Android).

 

Great to see that the codebase for Natmic and NatCam are being unified into one easy to use package.

Would very much love to make a feature request for :

Manual exposure control (ISO and shutter speed)

Cheers

 

Have just updated to NatDevice v1.0.1, but still seeing a crash on iPhone 6 when calling:

Code (CSharp):

1. MediaDeviceQuery natDeviceQuery = new MediaDeviceQuery(MediaDeviceQuery.Criteria.RearFacing);

It works fine on my iPad Pro however. Seems to be the same issue reported by other people on this forum. Attaching a screen grab of the crash from XCode, although it looks remarkably similar to the grabs posted by others.

Any idea what the cause could be? My iPhone 6 is running iOS 12.4.2 in case that helps?

 

I'm working with devs on fixing this. I should have it resolved in a few days. Sorry for the inconvenience.

 

Hi all,
i am trying to capture Audio devices under windows 10 and get this error:
NatDevice Error: Sample buffer delegate raised exception: System.ArgumentException: Destination array was not long enough. Check destIndex and length, and the array's lower bounds
at System.Array.Copy (System.Array sourceArray, System.Int32 sourceIndex, System.Array destinationArray, System.Int32 destinationIndex, System.Int32 length) [0x000da] in <599589bf4ce248909b8a14cbe4a2034e>:0
at System.Collections.Generic.List`1[T].CopyTo (System.Int32 index, T[] array, System.Int32 arrayIndex, System.Int32 count) [0x00013] in <599589bf4ce248909b8a14cbe4a2034e>:0
at NatSuite.Devices.MixerDevice+<>c__DisplayClass10_0.<StartRunning>b__1 (System.Single[] sampleBuffer, System.Int64 timestamp) [0x0001e] in C:\Users\xxx\Documents\GitHub\converself\record sprint\Assets\NatDevice\Plugins\Managed\MixerDevice.cs:75
at NatSuite.Devices.Internal.NativeAudioDevice+<>c__DisplayClass14_0.<StartRunning>b__0 (System.Single[] sampleBuffer, System.Int64 timestamp) [0x00002] in C:\Users\xxx\Documents\GitHub\converself\record sprint\Assets\NatDevice\Plugins\Managed\Internal\NativeAudioDevice.cs:54
UnityEngine.Debug:LogError(Object)
NatSuite.Devices.Internal.<>c__DisplayClass14_0:<StartRunning>b__0(Single[], Int64) (at Assets/NatDevice/Plugins/Managed/Internal/NativeAudioDevice.cs:55)
NatSuite.Devices.Internal.NativeAudioDevice:OnSampleBuffer(IntPtr, IntPtr, Int32, Int64) (at Assets/NatDevice/Plugins/Managed/Internal/NativeAudioDevice.cs:76)

Any idea? I can also show the script i used.
yours
Damir

 

I support. Extremely necessary thing when photographing.

As part of my task, I do not need to manually change the shutter speed, but I need to know what the current shutter speed level is - in order to make a decision: can user take a photo or it's too dark / there will be a lot of noise.

 

I'll look into this.

 

Hi,

I just got the plugin. Used to work with NatCam before, this one is slightly different.
I have 2 issues :
1) when building for iOS during iOS compilations i get the following warning :

ld: warning: ignoring file /Users/.../Desktop/Work/CleanCars/cleancars_ios/Libraries/NatDevice/Plugins/iOS/libNatDevice.a, building for iOS-armv7 but attempting to link with file built for iOS-arm64

followed by a whole bunch of errors :
Undefined symbols for architecture armv7:

"_NDCapturePhoto", referenced from:

_Bridge_CapturePhoto_m1F84A5563CCBC652F9D2A619D4E2804D2419B662 in Assembly-CSharp.o

_NativeCameraDevice_CapturePhoto_m16A1473E915F3A62DFCDA8492930D130BA3B734B in Assembly-CSharp.o

"_NDSetOrientation", referenced from:

_Bridge_Orientation_m88EA0827BC462E05D5C331F44190B550E4FDCC7F in Assembly-CSharp.o

_NativeCameraDevice_set_orientation_mEA46791BFF8934ECE876315A9ADA0C499D3F5A96 in Assembly-CSharp.o

"_NDZoomRatio", referenced from:

_Bridge_ZoomRatio_m0181FB60ECF3F2D4944F41FAB8075D9325C062D5 in Assembly-CSharp.o

_NativeCameraDevice_get_zoomRatio_m0CFEE3F10140D506951ECF529083224C1B187E51 in Assembly-CSharp.o

....

2) I also still have the crash of Unity on Mac when NatDevice tries to access the camera. Unity just quits. Will check the permissions ... as the Unity Webcamtexture works fine...

Thanks for the help !

 

Hello. I found 2 issues(also with the latest version) of your NatDevice Camera lib:
First: enabling the torch does not work on my android phone(Nokia 9).
_deviceCamera.torchEnabled = true; does not do anything. Light stays off. Getter returns true.
Same code is working on my iPhone 6s.

Second: It looks like the preview on IPhone has different orientation than the one on android.
e.g. I requested width/height 4000x3000.
On android I got 4000x3000, on iphone i got approx. 3000x4000 and preview did not work. It would by nice if width and height would have the same meaning on both systems.

Here is the output from the App(iPhone 6s):
Started camera preview with resolution 1080x1920
BarcodeScanner:.ctor()
System.Threading.ContextCallback:Invoke(Object)
...
later then:
2020-03-26 14:08:53.704014+0100 Scanner[353:9328] NatDevice: CameraDevice com.apple.avfoundation.avcapturedevice.built-in_video:0 updated format for preview resolution 3840x2160

all I do in the beginning is doubling the default resolution. But when the traces are correct width and height are swapped!

Regards, Ulrich

 

I will answer this myself

1) ld: warning: ignoring file /Users/.../Desktop/Work/CleanCars/cleancars_ios/Libraries/NatDevice/Plugins/iOS/libNatDevice.a, building for iOS-armv7 but attempting to link with file built for iOS-arm64

No-one actually needs to compile for the armv7 ... the arm64 is the way to go. In Unity/Build Settings/Player Settings
In the section Other Settings/Architecture tag arm64 here and all will be OK !

point 2) I didn't solve but it's not so important for me !

 

Are you using the latest NatDevice build from the Asset Store? Also, you might have to grant Unity microphone permissions.

 

I'll check this out.

I have two notes. First, running a preview with that resolution is practically unnecessary. 1920X1080 is enough for just about every application possible. Second, the preview resolution you set on the camera (and the one that is logged) does not respect the preview orientation. As a result it will always say WxH with W>H. The preview texture on the other hand respects the preview orientation. So if your preview resolution is 1920x1080 and your app is in portrait orientation, the preview texture will have a size of 1080x1920.

 

OK. I see I will check that.
The thing is, that I do the following:

Code (CSharp):

1. _cameraInterface.previewResolution = (_cameraInterface.previewResolution.width *2, _cameraInterface.previewResolution.height *2);
2.  
3. ScanResolutionTextField.text =
4.                 _cameraInterface.previewResolution.width + " x " + _cameraInterface.previewResolution.height + " px";

On Android this prints 2688x1512 px. Default would be half values.
On IOS(only working with default values!) it prints 1920x1080 px.
My iPhone does not work with the *2 line above!
So there should be un inconsistency between the setter and the getter of previewResolution.
No matter what orientation is set(I only use portrait mode anyway). I would guess that one of the 2 the setter or the getter does not take the orientation into account?!

 

This is because your iPhone doesn't support running a preview at that resolution. Setting a preview resolution doesn't guarantee that the camera supports it.

Your code above has nothing to do with orientation. I said orientation only affects the preview texture. It doesn't affect the values of `previewResolution` or `photoResolution`.

 

I thought of the microphone permissions but I was unable to set them... will look a bit further...

In NatCam the flash could be set to Auto and when too dark the take picture functionality made the flash work, how does this work in NatDevice ? I have my implementations working on iOS and Android now but the flash doesn't lit up any hint ??? Thanks !

 

You have to set the camera flash mode to auto. The default flash mode for all camera devices is off.

 

I'm trying to use MixerDevice in Natcorder so I can record video with mic & gameaudio. I imported Natcorder & Natdevice to my project on Windows 10 with Unity 2019.3.6f1 and used the following code

Code (CSharp):

1. /*
2. *   NatCorder
3. *   Copyright (c) 2020 Yusuf Olokoba
4. */
5.  
6. namespace NatSuite.Examples {
7.  
8.     using UnityEngine;
9.     using System;
10.     using System.Collections;
11.     using System.Linq;
12.     using NatSuite.Devices;
13.     using Recorders.Inputs;
14.     using NatSuite.Recorders;
15.     using NatSuite.Recorders.Clocks;
16.     using System.IO;
17.  
18.  
19.     public class ReplayCam : MonoBehaviour {
20.  
21.     [Header("Recording")]
22.  
23.     MP4Recorder recorder;       // Recorder that will record an MP4
24.     CameraInput cameraInput;    // Recorder input for recording video from a game camera
25.     IAudioDevice audioDevice;   // Microphone for recording user audio
26.     public AudioSource source;  //AudioSource for game audio
27.  
28.  
29.         public void StartRecording () {
30.             // Get a microphone and mixerdevice
31.             var query = new MediaDeviceQuery(); //added for game audio
32.             var microphone = query.devices.FirstOrDefault(device => device is IAudioDevice) as IAudioDevice; //added for game audio
33.             audioDevice = new MixerDevice(microphone, source); //added for game audio
34.        
35.  
36.             // Create recorder
37.             var clock = new RealtimeClock();
38.             recorder = new MP4Recorder(1280, 720, 30, audioDevice.sampleRate, audioDevice.channelCount);
39.             // Stream media samples
40.             cameraInput = new CameraInput(recorder, clock, Camera.main);
41.             audioDevice.StartRunning((sampleBuffer, timestamp) => recorder.CommitSamples(sampleBuffer, clock.timestamp));
42.         }
43.  
44.         public async void StopRecording () {
45.             cameraInput.Dispose();
46.             audioDevice.StopRunning();
47.             // Finish writing video
48.             var path = await recorder.FinishWriting();
49.             // Playback recording
50.             Debug.Log($"Saved recording to: {path}");
51.             var prefix = Application.platform == RuntimePlatform.IPhonePlayer ? "file://" : "";
52.             Handheld.PlayFullScreenMovie($"{prefix}{path}");
53.         }
54.     }
55. }

I get the following error and the gameaudio is choppy in the video on Windows. Video is not playable at all on Android.

NatDevice Error: Sample buffer delegate raised exception: System.ArgumentException: Destination array was not long enough. Check destIndex and length, and the array's lower bounds
at System.Array.Copy (System.Array sourceArray, System.Int32 sourceIndex, System.Array destinationArray, System.Int32 destinationIndex, System.Int32 length) [0x000da] in <437ba245d8404784b9fbab9b439ac908>:0
at System.Collections.Generic.List`1[T].CopyTo (System.Int32 index, T[] array, System.Int32 arrayIndex, System.Int32 count) [0x00013] in <437ba245d8404784b9fbab9b439ac908>:0
at NatSuite.Devices.MixerDevice+<>c__DisplayClass10_0.<StartRunning>b__1 (System.Single[] sampleBuffer, System.Int64 timestamp) [0x0001e] in C:\Users\xxx\Desktop\APPTest\NatCorderDevice\Assets\NatDevice\Plugins\Managed\MixerDevice.cs:75
at NatSuite.Devices.Internal.NativeAudioDevice+<>c__DisplayClass14_0.<StartRunning>b__0 (System.Single[] sampleBuffer, System.Int64 timestamp) [0x00002] in C:\Users\xxx\Desktop\APPTest\NatCorderDevice\Assets\NatDevice\Plugins\Managed\Internal\NativeAudioDevice.cs:54
UnityEngine.Debug:LogError(Object)
NatSuite.Devices.Internal.<>c__DisplayClass14_0:<StartRunning>b__0(Single[], Int64) (at Assets/NatDevice/Plugins/Managed/Internal/NativeAudioDevice.cs:55)
NatSuite.Devices.Internal.NativeAudioDevice:OnSampleBuffer(IntPtr, IntPtr, Int32, Int64) (at Assets/NatDevice/Plugins/Managed/Internal/NativeAudioDevice.cs:76)

2nd error that occurs:

Found plugins with same names, Assets/NatDevice/Plugins/Android/NatRender.aar and Assets/NatCorder/Plugins/Android/NatRender.aar. Delete the one of the duplicate plugins.
UnityEditor.AndroidPluginImporterExtension:CheckFileCollisions(String)
UnityEditorInternal.PluginsHelper:CheckFileCollisions(BuildTarget)
UnityEditor.BuildPlayerWindow:BuildPlayerAndRun()

I deleted the NatRender.aar from the Natcorder plugin as the fix for the 2nd error. Please advise if that was the incorrect course of action.

 

This is a known issue; I'm looking into it.

Yup, this is mentioned in the NatCorder README.md. I'll add it to the NatDevice README.md.

 

Hi

I am developping a VR app with Unity, working on an Android phone with Samsung Gear VR. In this app I want to be able to :

1. Show live video from the camera.
2. Take a photo when I press a button.

I already have a WebcamTexture which works fine for the camera preview at 1280x720 and 30 fps, but I want the photos to be full resolution, so a snapshot from the WebcamTexture is not ok.

I don't mind if the display goes black for a second when I take a picture, although it would be better if it could be as smooth as the photo experience we are used to.
And of course I am willing to remove the WebcamTexture, this is a temporary solution

I already tried several assets and I even tried to develop my own native Android plugin in Kotlin (using CameraX API). So far I can't get to have both tasks working in the same app : preview at low resolution and photo capture at high resolution.

Before I spend the 90$, do you think I will be able to do that with NatDevice ? How ?

Thank you

 

Hi,

Was wondering if you had made any progress on the crash bug on iPhone 6 during the call to new MediaDeviceQuery.

This bug is preventing me from releasing a new version of my app so would be good to know if a fix has been found and when it might be available, or indeed if there is anything I can do to prevent the crash using the current version of NatDevice.

 

Hi, My company buys NatDevice-Media Device API, but I followed the github NatDevice With OpenCVForUnity Example (https://github.com/EnoxSoftware/NatDeviceWithOpenCVForUnityExample) for Android build apk test, but the following problem appeared, can tell me how to deal with ?

Unity version: 2019.3.0f6

Build Settings :

 

I have been using NatCorder and NatMic to record videos of my app on iOS devices. My app is about speech recognition where voice recording functions other than NatMic is involved along with NatMic. The Problem is that when this speech-to-text function is on, sync between video and audio will be broken. From that moment and on, audio comes earlier than images. In other time, the sync works normally. So I guess it has something to do with microphone involved simultaneously with NatMic and speech recognition. Another weird thing is that it only happens on iPhones while everything is normal on iPad so far. I have tried two different speech recognition kits. One is for unity and the other is just for iOS. Wondering what to do about it. Thank you.

 

Hello Lanre.
First of all. Thanks for your fast responses!
I found the "torch issue".
The problem was that I set the flash mode to Auto.
When this is the case torch cannot be enabled on Android(on IOS it can).
Now I set the flash mode to Off and the torch mode works.
I'm not sure if this is the desired behavior but it is OK to me.
I hope this helps someone else.

 

NatDevice supports taking hi-res photos. I'm currently working on documentation to explain more on this feature. The `CameraDevice` class has a `CapturePhoto` function which will take a hi-res photo (with optional flash) and will return it in a `Texture2D` object.

 

Nope. I've been testing with two developers and the issue is beyond our code. With meticulous error checking, no error is generated when running. Instead, the iPhone 6 crashes. I'm reaching out to Apple Developer on this.

 

This is a NatShare bug, and there is an open PR with the fix.

 

If audio is coming before video, then maybe your speech to text function is computationally intensive? If this is the case, then your app isn't able to commit video frames as quickly, maybe because of a frame rate hit. I'd say to profile your app instead of conjecturing.

 

Hi, Lanre. Appreciate your guidance. We did a minimal test then by creating a new project with only NatCorder and NatMic in it. Nothing more. It works fine under most circumstances while there has been lags still in others. On some old model iPhone such as iPhone 7, if recording plays long enough like 5 minutes, there appears obvious lags between audio and video. And you can even feel them starting around 1-2 minute. Since we suppose devices like iPhone 7 are still in use of some scale, we may have to figure out a solution.

We tried pausing and restarting wthin a few frames from time to time or at some computationally intensive time. This mitigates the problem. Dont know if it's an appropriate way. So wondering if there any way of refreshing during recording to make up? Thank you.

 

Can you share your recording code? And can you upgrade to NatDevice? NatMic has been deprecated. Make sure you are using NatCorder 1.7.1 and NatDevice 1.0.1.

 

Hi,
We are running on specific Android devices (BV 5500 and BV 5500 pro). I used to use the Unity webcam on a texture but this sometimes crashed the app. So we decided to go for native Android cam support trough NatDevice.
During our final testing (not final as we have this crash ) we got the following crash... before the crash the app slowed down like crazy. Clicking on the "take a picture button" lagged so hard, the picture got takes 6-10 seconds after having pressed.

I uploaded a part of the logcat here. (crash.pdf)

I have the impression that in this case, before the crash, if (deviceQuery.currentDevice is CameraDevice device) was false. I had seen this issue happen once before. It resulted in a lock up in the app as the picture wasn't taken, and the rest of the code awaited for the picture.

As I didn't find how to kill the created "MediaDeviceQuery" I just made a new one upon every failure... using : deviceQuery = new MediaDeviceQuery(MediaDeviceQuery.Criteria.GenericCameraDevice);
which I presume resulted in a slow down and the final crash, as every call to the TakePicture tested false for the equation.

The 'if (deviceQuery.currentDevice is CameraDevice device)' becomes false after the app went to background a couple of times. What to do when if (deviceQuery.currentDevice is CameraDevice device) become false ???

Any ideas ?

Thanks!

 

Hi, Lanre. Ok, I fixed the IL2CPP Compiler Error, but there are new problems. The following is the wrong record.
My Minimum API Level setting Android 5.1 or Android 6 test Android build apk