NatDevice - Media Device API

Hi @Lanre, I have tracked the issue with dark photos down to the following lines of code.

device.previewResolution = (1280, 960);

device.photoResolution = (1280, 960);

These two device variables represent two different "devices". If I comment out those two lines the photos are fine. Our goal for setting the resolution in the first place was to reduce the photo size and in most cases with phones made in the last few years the photos are quite large and we typically can get by with half or quarter resolution for our application.

What options do we have to reduce the size of the photos without making them dark?

 

Hi @Lanre , i try to use this plugin in android and works great, but ios(iphone 8 with ios 14) photo make a bluescreen photo, any idea? i see the minicam.cs :S

 

Hi again @Lanre

As I mentioned earlier, I am using your plugin in a time critical application and I have now run into another issue.

I have noticed that on Android I am missing frames at regular intervels when using preview resolution larger than 160x120. I have tracked the issue down to Texture2D.GetPixels32() taking >50 ms to execute which I think further boils down to the garbage collector kicking in and causing this.



I was wondering if this is a known issue and if the upcoming 1.0.3 release might have a fix for this?

 - Greatly reduced camera preview memory consumption on Android.

Best,
WIktor

 

Hi, could you please respond to my email about 1.0.3 version? I can't change preview resolution on iOS and its very necessary for my project

 

I don't understand what you mean by the fact that they are two separate devices. Are you trying to run more than one device simultaneously?

 

Make sure that your app is rendering with Metal, not OpenGL ES.

 

I recommend using GetRawTextureData. It doesn't create any allocations; you get a direct handle to the native pixel buffer. The next update will provide overloads of the `StartRunning` method that give you a handle to the pixel data directly from NatDevice's native camera device. This can be used to completely avoid having to upload the data to a Texture2D.

 

Sorry for the late response; I've been very behind on emails. I'll send it over tomorrow.

 

I understand that you are busy, but could you please answer my email? I can't push very necessary update!

 

I'm working on a minor fix on Android for a bug which I found before sending you the build. I'll have the build to you later today.

 

Hi @Lanre , I hope I get this explanation across correctly. I tried to follow your example in MiniCam where you use a ICameraDevice for the preview and then check to see if device is a CameraDevice to take the photo/save the photo

We setup an ICameraDevice to use for the preview (device = query.currentDevice as ICameraDevice . If we are on the editor this device is also our webcamera and we use it to generate the images for testing in editor.

If we are on a real mobile device we still use the same device but test to see if it is also a CameraDevice (query.currentDevice is CameraDevice device) before we take the actual image on the device camera. If it is not a Camera device (Ie webcamera)then we save the image a different way.

Are we doing this incorrectly? It seems to work fine for us except when we actually try to specify a specific resolution.

If we do not ask specific resolution the photos are not dark.

 

Actually I do also have a question about minicam.cs

Line 41

Code (CSharp):

1. var device = query.currentDevice as ICameraDevice;

creates a local variable device and it is of type ICameraDevice

on line 58

Code (CSharp):

1. if (query.currentDevice is CameraDevice device) {

this creates a second local variable of type CameraDevice and that is used to Capture the photo.

My problem is I want to access that CameraDevice device outside of that if block and I am not sure how to code that

 

Does anybody using this package have a solution for scaling down the images generated? It is not clear to me if the API is supposed to handle that or should I be looking for another solution to handle that aspect?

 

What you're doing is correct, and your explanation is very helpful. This is one of the three remaining issues for the 1.0.3 release. I'll keep you posted.

There isn't a guarantee that the `device` is a `CameraDevice` (because it could be a `WebCameraDevice` on non-mobile platforms). All you need to do is cast it, whether using an is-cast or an as-cast.

NatDevice doesn't handle image manipulation (or technically speaking, computer vision) operations. Options include using Unity (Blit to a smaller render texture); or use a plugin like OpenCVForUnity.

 

I am not able to reproduce the dark photos on my iPhone 12 Pro or iPhone 7+. Here's my code:

Code (CSharp):

1. // Start camera preview
2. var device = query.current as CameraDevice;
3. device.previewResolution = (1280, 960);
4. device.photoResolution = (1280, 960);
5. var previewTexture = await device.StartRunning();

The rest of the code is the exact same as MiniCam.

 

Hi @Lanre , any update on this OnSampleBuffer issue ?

 

I've been doing investigations on this issue. NatDevice uses a different audio API from that used by NatMic. Apparently, this new API (AVAudioEngine) ignores the buffer size that is requested by developers, and used a fixed size buffer. Given this constraint, NatDevice cannot force a smaller buffer size; instead the app itself can do so by requesting smaller audio IO buffers on the global audio session. NatDevice cannot handle this because it would interfere with Unity's DSP. Today I'm going to see if Unity's Audio Settings, specifically the DSP buffer size, has any effect on this.

I've found some API additions on iOS 13 that might help with this issue. I'll be spending today implementing and testing it. I'll keep you posted. NatDevice 1.1.0 is almost ready!

 

Hi @Lanre, I'm having some problems with with captured photos being a different resolution to what the device specs say they should be. I'm using an iPad Pro 12.9 inch (4th gen), it has 2 cameras & the specs say their resolutions are 4032x3024 & 3840x2160 (https://www.devicespecifications.com/en/model/637c532e).

I can access both cameras with NatDevice and photos are coming out at either 4224x2376 or 3840x2160. When I take photos in Apple's native camera app using either the 1.0x or 0.5x options (which I think accesses the 2 different cameras) both photos come out at 4032x3024, which is the 4:3 ratio of the device.

Do you know how I can make photos taken with NatDevice be 4032x3024 on iPad? I can't get my head around why photos taken with NatDevice are 16:9 (4224x2376) and not 4:3 on iPad.

Incase it's relevant I'm using the MiniCam example scene, I've got the allowed orientation set to landscape-left, and I've tried setting device.photoResolution = (9999, 9999) but that doesn't change things.

Thanks!

 

Thank You.

I was not clear that "is" also returns a casted variable if true I thought it was only doing a test.

 

On iOS, setting the photo resolution is more complicated than you'd expect. The reason is that media devices on iOS can only be configured in a discrete set of "formats". Each format has exactly one preview resolution and a few photo resolutions (typically about 3). So when you set the preview or photo resolution, NatDevice can't just set what you requested, because if it did you would find that the other resolution changed. As such, NatDevice gives precedence to the preview resolution. This means that when you set a photo resolution, NatDevice looks for a format that preserves the preview resolution that is currently set, and sets the photo resolution to the one of the supported ones which is closest to what you request.

There isn't a good way to set a specific photo resolution with NatDevice, and this is a consequence of the trade off above, along with the fact that over the years it simply isn't a priority for the overwhelming majority of devs. Fundamentally, your app shouldn't expect a specific aspect ratio for photos, as this isn't something that's guaranteed across different devices and cameras.

 

This is a new pattern-matching feature introduced in C# 7.

 

NatDevice 1.1.0 is ready for beta testing.

Please email me with your invoice number and I'll send you the build. Here's the changelog:

+ Greatly reduced camera preview memory consumption on Android.
+ Greatly improved camera preview latency on Android.
+ Added `MediaDeviceQuery.count` property.
+ Added `MediaDeviceQuery.Criteria.EchoCancellation` criterion for finding microphones that perform echo cancellation.
+ Added `IAudioDevice.StartRunning` overload with callback that is given the native sample buffer handle with no memory copy.
+ Added `ICameraDevice.StartRunning` overload with callback that is given the raw pixel buffer in managed memory.
+ Added `ICameraDevice.StartRunning` overload with callback that is given the native pixel buffer handle with no memory copy.
+ Added `CameraDevice.exposurePointSupported` for checking if camera supports setting exposure point.
+ Added `CameraDevice.focusPointSupported` for checking if camera supports setting focus point.
+ Changed `CameraDevice` behaviour to no longer pause and resume when app is suspended. You must handle this yourself.
+ Fixed flash not firing on some Android devices.
+ Fixed preview resolution setting being ignored on iOS.
+ Fixed hardware microphone format being ignored on macOS.
+ Fixed memory leak in photo capture on iOS.
+ Fixed memory leak when `WebCameraDevice` is stopped.
+ Fixed non-ASCII characters in `AudioDevice` name not being captured on Windows.
+ Fixed media device query permission task never completing on Windows.
+ Fixed permissions dialog not being presented on UWP.
+ Refactored `MediaDeviceQuery.Criteria` to `MediaDeviceCriteria`.
+ Refactored `MediaDeviceCriteria.RearFacing` to `MediaDeviceCriteria.RearCamera`.
+ Refactored `MediaDeviceCriteria.FrontFacing` to `MediaDeviceCriteria.FrontCamera`.
+ Refactored `MediaDeviceQuery.currentDevice` to `MediaDeviceQuery.current`.
+ Deprecated `MediaDeviceQuery.devices` property. The query now acts as a collection itself.
+ Deprecated `MixerDevice` reference implementation.
+ Minimum requirement on Android has been increased to API level 24.
+ Minimum requirement on iOS has been increased to iOS 13.

 

Hey, I'm testing the 1.1.0b and StopPreview is causing an app crash on my Samsung Galaxy S8:

Code (Boo):

1. 01-26 13:30:57.202  5061 24850 D ExynosCameraNode: [CAM_ID(0)][MCSC_CAPTURE_MAIN]-DEBUG(close[285]):close(fd:52)
2. 01-26 13:30:57.202  5061 24850 D ExynosCameraMCPipe: [CAM_ID(0)][PIPE_3AA_REPROCESSING]-DEBUG(destroy[136]):Main node(MCSC_CAPTURE_MAIN, sensorIds:18890000) closed
3. 01-26 13:30:57.202  5061 24850 D ExynosCameraNode: [CAM_ID(0)][MCSC_CAPTURE_YUV_2]-DEBUG(close[285]):close(fd:51)
4. 01-26 13:30:57.203  5061 24850 D ExynosCameraMCPipe: [CAM_ID(0)][PIPE_3AA_REPROCESSING]-DEBUG(destroy[136]):Main node(MCSC_CAPTURE_YUV_2, sensorIds:18890000) closed
5. 01-26 13:30:57.203  5061 24850 D ExynosCameraNode: [CAM_ID(0)][MCSC_CAPTURE_YUV_1]-DEBUG(close[285]):close(fd:50)
6. 01-26 13:30:57.203  5061 24850 D ExynosCameraMCPipe: [CAM_ID(0)][PIPE_3AA_REPROCESSING]-DEBUG(destroy[136]):Main node(MCSC_CAPTURE_YUV_1, sensorIds:18890000) closed
7. 01-26 13:30:57.203  5061 24850 D ExynosCameraNode: [CAM_ID(0)][MCSC_CAPTURE_YUV_0]-DEBUG(close[285]):close(fd:49)
8. 01-26 13:30:57.203  5061 24850 D ExynosCameraMCPipe: [CAM_ID(0)][PIPE_3AA_REPROCESSING]-DEBUG(destroy[136]):Main node(MCSC_CAPTURE_YUV_0, sensorIds:18890000) closed
9. 01-26 13:30:57.203  5061 24850 D ExynosCameraNode: [CAM_ID(0)][REPROCESSING_ISP_CAPTURE_M2M]-DEBUG(close[285]):close(fd:48)
10. 01-26 13:30:57.203  5061 24850 D ExynosCameraMCPipe: [CAM_ID(0)][PIPE_3AA_REPROCESSING]-DEBUG(destroy[136]):Main node(REPROCESSING_ISP_CAPTURE_M2M, sensorIds:18884624) closed
11. 01-26 13:30:57.203  5061 24850 D ExynosCameraNode: [CAM_ID(0)][REPROCESSING_3AA_OUTPUT]-DEBUG(close[285]):close(fd:47)
12. 01-26 13:30:57.210  5061 24850 D ExynosCameraMCPipe: [CAM_ID(0)][PIPE_3AA_REPROCESSING]-DEBUG(destroy[136]):Main node(REPROCESSING_3AA_OUTPUT, sensorIds:18902529) closed
13. 01-26 13:30:57.210  5061 24850 D ExynosCameraMCPipe: [CAM_ID(0)][PIPE_3AA_REPROCESSING]-DEBUG(destroy[154]):Node destroyed
14. 01-26 13:30:57.210  5061 24850 I ExynosCameraMCPipe: [CAM_ID(0)][PIPE_3AA_REPROCESSING]-INFO(destroy[166]):destroy() is succeed, Pipe(202)
15. 01-26 13:30:57.210  5061 24850 I ExynosCameraMCPipe: [CAM_ID(0)][PIPE_3AA_REPROCESSING]-INFO(destroy[100]):
16. 01-26 13:30:57.210  5061 24850 D ExynosCameraMCPipe: [CAM_ID(0)][PIPE_3AA_REPROCESSING]-DEBUG(destroy[154]):Node destroyed
17. 01-26 13:30:57.210  5061 24850 I ExynosCameraMCPipe: [CAM_ID(0)][PIPE_3AA_REPROCESSING]-INFO(destroy[166]):destroy() is succeed, Pipe(202)
18. 01-26 13:30:57.210  5061 24850 D ExynosCameraPipeFlite: [CAM_ID(0)][PIPE_3AA_REPROCESSING]-DEBUG(destroy[108]):
19. 01-26 13:30:57.210  5061 24850 I ExynosCameraPipeFlite: [CAM_ID(0)][PIPE_3AA_REPROCESSING]-INFO(destroy[143]):destroy() is succeed (202)
20. 01-26 13:30:57.210  5061 24850 D ExynosCamera3FrameFactory: [CAM_ID(0)][ReprocessingFactory]-DEBUG(destroy[68]):Pipe(2) destroyed
21. 01-26 13:30:57.210  5061 24850 I ExynosCameraMCPipe: [CAM_ID(0)][PIPE_ISP_REPROCESSING]-INFO(destroy[100]):
22. 01-26 13:30:57.210  5061 24850 D ExynosCameraMCPipe: [CAM_ID(0)][PIPE_ISP_REPROCESSING]-DEBUG(destroy[154]):Node destroyed
23. 01-26 13:30:57.210  5061 24850 I ExynosCameraMCPipe: [CAM_ID(0)][PIPE_ISP_REPROCESSING]-INFO(destroy[166]):destroy() is succeed, Pipe(205)
24. 01-26 13:30:57.210  5061 24850 I ExynosCameraMCPipe: [CAM_ID(0)][PIPE_ISP_REPROCESSING]-INFO(destroy[100]):
25. 01-26 13:30:57.210  5061 24850 D ExynosCameraMCPipe: [CAM_ID(0)][PIPE_ISP_REPROCESSING]-DEBUG(destroy[154]):Node destroyed
26. 01-26 13:30:57.210  5061 24850 I ExynosCameraMCPipe: [CAM_ID(0)][PIPE_ISP_REPROCESSING]-INFO(destroy[166]):destroy() is succeed, Pipe(205)
27. 01-26 13:30:57.210  5061 24850 D ExynosCameraPipeFlite: [CAM_ID(0)][PIPE_ISP_REPROCESSING]-DEBUG(destroy[108]):
28. 01-26 13:30:57.210  5061 24850 I ExynosCameraPipeFlite: [CAM_ID(0)][PIPE_ISP_REPROCESSING]-INFO(destroy[143]):destroy() is succeed (205)
29. 01-26 13:30:57.210  5061 24850 D ExynosCamera3FrameFactory: [CAM_ID(0)][ReprocessingFactory]-DEBUG(destroy[68]):Pipe(5) destroyed
30. 01-26 13:30:57.211  5061 24850 I ExynosCameraMCPipe: [CAM_ID(0)][PIPE_MCSC_REPROCESSING]-INFO(destroy[100]):
31. 01-26 13:30:57.211  5061 24850 D ExynosCameraMCPipe: [CAM_ID(0)][PIPE_MCSC_REPROCESSING]-DEBUG(destroy[154]):Node destroyed
32. 01-26 13:30:57.211  5061 24850 I ExynosCameraMCPipe: [CAM_ID(0)][PIPE_MCSC_REPROCESSING]-INFO(destroy[166]):destroy() is succeed, Pipe(210)
33. 01-26 13:30:57.211  5061 24850 I ExynosCameraMCPipe: [CAM_ID(0)][PIPE_MCSC_REPROCESSING]-INFO(destroy[100]):
34. 01-26 13:30:57.211  5061 24850 D ExynosCameraMCPipe: [CAM_ID(0)][PIPE_MCSC_REPROCESSING]-DEBUG(destroy[154]):Node destroyed
35. 01-26 13:30:57.211  5061 24850 I ExynosCameraMCPipe: [CAM_ID(0)][PIPE_MCSC_REPROCESSING]-INFO(destroy[166]):destroy() is succeed, Pipe(210)
36. 01-26 13:30:57.211  5061 24850 D ExynosCameraPipeFlite: [CAM_ID(0)][PIPE_MCSC_REPROCESSING]-DEBUG(destroy[108]):
37. 01-26 13:30:57.211  5061 24850 I ExynosCameraPipeFlite: [CAM_ID(0)][PIPE_MCSC_REPROCESSING]-INFO(destroy[143]):destroy() is succeed (210)
38. 01-26 13:30:57.211  5061 24850 D ExynosCamera3FrameFactory: [CAM_ID(0)][ReprocessingFactory]-DEBUG(destroy[68]):Pipe(10) destroyed
39. 01-26 13:30:57.211  5061 24850 E libcsc  : csc_deinit:: unsupported csc_hw_type
40. 01-26 13:30:57.211  5061 24850 I ExynosCameraSWPipe: [CAM_ID(0)][PIPE_GSC_REPROCESSING]-INFO(release[55]):destroy() is succeed (218)
41. 01-26 13:30:57.211  5061 24850 I ExynosCameraSWPipe: [CAM_ID(0)][PIPE_GSC_REPROCESSING]-INFO(release[55]):destroy() is succeed (218)
42. 01-26 13:30:57.211  5061 24850 D ExynosCamera3FrameFactory: [CAM_ID(0)][ReprocessingFactory]-DEBUG(destroy[68]):Pipe(18) destroyed
43. 01-26 13:30:57.211  5061 24850 I ExynosCamera3FrameFactory: [CAM_ID(0)][ReprocessingFactory]-INFO(destroy[49]):
44. 01-26 13:30:57.211  5061 24850 I ExynosCamera3FrameFactory: [CAM_ID(0)][ReprocessingFactory]-INFO(m_transitState[927]):Skip state transition. curState 0
45. 01-26 13:30:57.211  5061 24850 I ExynosCamera3FrameFactory: [CAM_ID(0)][ReprocessingFactory]-INFO(destroy[49]):
46. 01-26 13:30:57.211  5061 24850 I ExynosCamera3FrameFactory: [CAM_ID(0)][ReprocessingFactory]-INFO(m_transitState[927]):Skip state transition. curState 0
47. 01-26 13:30:57.211  5061 24850 D ExynosCamera3: [CAM_ID(0)][]-DEBUG(m_deinitFrameFactory[3976]):m_frameFactory[3] destroyed
48. 01-26 13:30:57.211  5061 24850 I ExynosCamera3: [CAM_ID(0)][]-INFO(m_deinitFrameFactory[3981]):-OUT-
49. 01-26 13:30:57.211  5061 24850 D ExynosCamera3: DEBUG:duration time(   26 msec):(m_deinitFrameFactory)
50. 01-26 13:30:57.211  5061 24850 I ExynosCamera3: [CAM_ID(0)][]-INFO(release[900]):-OUT-
51. 01-26 13:30:57.211  5061 24850 I ExynosCamera3Interface: INFO(HAL3_camera_device_close[203]):close camera(0)
52. 01-26 13:30:57.213  5061 24850 I ExynosCamera3Interface: INFO(HAL3_camera_device_close[212]):out =====
53. 01-26 13:30:57.213  5061 24850 D ExynosCamera3Interface: DEBUG:duration time(  564 msec):(HAL3_camera_device_close)
54. 01-26 13:30:57.218  5061 24850 I CameraLatencyHistogram: Stream 0 dequeueBuffer latency histogram (72) samples:
55. 01-26 13:30:57.218  5061 24850 I CameraLatencyHistogram:         5     10     15     20     25     30     35     40     45    inf (max ms)
56. 01-26 13:30:57.218  5061 24850 I CameraLatencyHistogram:      98.61   1.39   0.00   0.00   0.00   0.00   0.00   0.00   0.00   0.00 (%)
57. 01-26 13:30:57.218  5061 24850 I Camera3-Device: disconnect: X
58. 01-26 13:30:57.218  5061 24850 D Camera3-Device: disconnect: mInterfaceLock.unlock()
59. 01-26 13:30:57.218  5061 24850 D Camera3-Device: disconnect: mInterfaceLock.lock()
60. 01-26 13:30:57.218  5061 24850 I Camera3-Device: disconnect: E
61. 01-26 13:30:57.218  5061 24850 D Camera3-Device: disconnect: mInterfaceLock.unlock()
62. 01-26 13:30:57.221  5061 24850 I CameraService: finishCameraOps: Finish camera ops, package name = com.company.app, client UID = 10271
63. 01-26 13:30:57.223  5061 24850 I CameraService: onTorchStatusChangedLocked: Torch status changed for cameraId=0, newStatus=1
64. 01-26 13:30:57.227  5061 24850 I CameraFlashlight: deviceClosed: device 0 is closed(package=com.company.app)
65. 01-26 13:30:57.229  5061 24850 I CameraService: onTorchStatusChangedLocked: Torch status changed for cameraId=0, newStatus=1
66. 01-26 13:30:57.229  5061 24850 I CameraService: disconnect: Disconnected client for camera 0 for PID 19244
67. 01-26 13:30:57.230 19244 19268 V NatSuite: NatDevice: Camera 0 stopped preview
68. 01-26 13:30:57.714  5675  5843 D NetworkController.MobileSignalController(0/1): onSignalStrengthsChanged signalStrength=SignalStrength: 0 99 -3 -200 -3 -200 -1 0 -113 -17 -50 2 0 2147483647 0 7 -113 0x1000 P gsm|lte use_rsrp_and_rssnr_for_lte_level  [-128, -118, -108, -98] [-115, -105, -95, -85] level=1
69. 01-26 13:30:57.714  5675  5843 D NetworkController.MobileSignalController(0/1): getMobileIconGroup(): 13
70. 01-26 13:30:57.714  5675  5843 D NetworkController.MobileSignalController(0/1): updateORGMobileIconGroup(): 13
71. 01-26 13:30:57.753  5332  5517 I WifiVendorHal: getWifiLinkLayerStats(l.2973) before calling iface.getLinkLayerStats
72. 01-26 13:30:57.754  3296  3296 I android.hardware.wifi@1.0-service: enter getLinkLayerStats
73. 01-26 13:30:57.754  3296  3296 I android.hardware.wifi@1.0-service: enter getLinkLayerStatsInternal
74. 01-26 13:30:57.759  3296  3296 I android.hardware.wifi@1.0-service: Successfully getLinkLayerStats.
75. 01-26 13:30:57.759  5332  5517 I WifiVendorHal: getWifiLinkLayerStats(l.2973) after calling iface.getLinkLayerStats
76. 01-26 13:30:57.768  5332  5517 I WifiStateMachine: checkScoreBasedQuality -  mPreviousScore[0]:62 mPreviousScore[1]:61 mPreviousScore[2]:61 s2Score:60mPrevoiusScoreAverage:61
77. 01-26 13:30:57.820 19244 19268 D Unity   : Uploading Crash Report
78. 01-26 13:30:57.820 19244 19268 E Unity   : ArgumentException: GCHandle value belongs to a different domain
79. 01-26 13:30:57.820 19244 19268 E Unity   :   at System.Runtime.InteropServices.GCHandle.op_Explicit (System.IntPtr value) [0x00025] in <9577ac7a62ef43179789031239ba8798>:0
80. 01-26 13:30:57.820 19244 19268 E Unity   :   at NatSuite.Devices.Internal.NativeCameraDevice.OnPixelBuffer (System.IntPtr context, System.IntPtr pixelBuffer, System.Int32 width, System.Int32 height, System.Int64 timestamp) [0x00000] in /<path>/Devices/Internal/NativeCameraDevice.cs:203
81. 01-26 13:30:57.820 19244 19268 E Unity   : UnityEngine.DebugLogHandler:Internal_LogException(Exception, Object)
82. 01-26 13:30:57.820 19244 19268 E Unity   : UnityEngine.DebugLogHandler:LogException(Exception, Object)
83. 01-26 13:30:57.820 19244 19268 E Unity   : UnityEngine.Logger:LogException(Exception, Object)
84. 01-26 13:30:57.820 19244 19268 E Unity   : UnityEngine.Debug:LogException(Exception)
85. 01-26 13:30:57.820 19244 19268 E Unity   : UnityEngine.<>c:<RegisterUECatcher>b__0_0(Object, UnhandledExceptionEventArgs) (at /Users/bokken/buildslave/unity/build/Runtime/Export/Scripting/UnhandledExceptionHandler.bindings.cs:46)
86.  

also, after selecting a camera from MediaDeviceQuery and testing if preview is already started with:

Code (CSharp):

1. if (camera.running) {
2.     return;
3. }
4. camera.StartRunning(OnFrame)

im getting:

Code (Boo):

1. NullReferenceException: Object reference not set to an instance of an object
2. NatSuite.Devices.WebCameraDevice.get_running () (at <path_to>/NatDevice/Devices/WebCameraDevice.cs:47)
3. XXX.StartCapture () (at ...)
4.  

 

Thanks for bringing these to my attention. I'll check them out and get back to you!

 

So does this address the buffer delay issue?

Or do we need to call the

setPreferredIOBufferDuration

method as mentioned above? If so, in what way does this interfere with Unity's DSP and can this not be wrapped by NatDevice so we can call it easily?

Thanks

 

You'll still need to call

setPreferredIOBufferDuration

. I don't know how it interferes with Unity's DSP; that's an internal implementation detail with Unity. I recommend trying to set the DSP buffer size in Unity's Audio Settings as a start. If that doesn't give you low enough latency, then try setting the IO duration and see if it affects Unity's audio output.

 

I've just fixed both of these. Email me for the updated build.

 

Unity-iPhone.xcodeproj The linked library 'libNatDevice.a' is missing one or more architectures required by this target: armv7.

any ideas, it is a pod build if that makes a difference

 

None of the NatSuite API's--NatCorder, NatDevice, NatShare, NatReader--have armv7 (32-bit) slices on iOS. They only support arm64 because they are built for iOS 11+ (soon to be iOS 13+), which only supports arm64.

 

Apple won't let me update the app without a Arm7 version?

 

Might be something else

 

When building with a deployment target at or greater than iOS 11, Xcode will simply not build armv7 slices.

 

@jonjojo I had the issue two months ago when we started targeting iOS 11+.
The UIRequiredDeviceCapability "armv7" was automatically switched by Xcode to "arm64".

We were afraid that Apple would not approve the update but it went fine !

 

On IOS when using Mini Cam scene from NatCam Media Device pack, on IOS we get a crash when changing scenes after using the camera feed. This is on IOS 14 and Unity 2020.2.5f1 but not exclusive to that specific unity version.

Is there an OnDestroy method we need to implement?

 

Can you share the crash logs? I recommend stopping the camera before you switch scenes. NatDevice is completely oblivious to what is happening in Unity Engine, so it doesn't know about scene changes and doesn't handle it in any way.

 

Fixed it for you in MiniCam.cs, in my case I had the crash because I just copied the example content into my app, it is good practice to stop the camera or any textures using void OnDestroy I think

I did this and removed the var in front of 'device' where you declared it in Start function

Even if the user wanted to keep the device running when changing scene, it doesn't actually crash or anything in the Editor but crashes on IOS and Android in our tests so best to stop the device when everything gets destroyed and start it again after when needed!

Code (CSharp):

1. private ICameraDevice device;
2.  
3.         void OnDestroy()
4.         {
5.             device.StopRunning();
6.         }

 

@Lanre thanks for explaining, that makes sense now, I can find a way to work with this

 

Hi Lanre, I have been using an earlier version of NatDevice to stream audio and react to sounds in game for the last 6 months for a project. I have only just updated to NatDevice 1.0.2 because I was getting a bitcode error when trying to Archive in Xcode to upload to the App Store. Since updating to v1.0.2, all my Android builds work fine, however my iOS builds have a noticeable delay in reacting to the streaming data. It is almost half a second delay. Is there something that I am missing with the new version? Can I do anything to reduce this delay for iOS? Alternatively is there a way to make the older version Archive in Xcode without the bitcode error because it worked fine before.

 

Hey there sorry for the late response. I responded to your email. NatDevice 1.1.0 will greatly improve latency on iOS.

 

Hi Lanre,

Could you let me know if the audio problem I reported back in November will be addressed by NatDevice 1.1.0, and when we can expect to see this build released to the Asset Store?

Thanks!

 

Yes it has, NatDevice 1.1.0 will not touch the audio session category any longer. What it will do is set the mode to allow for things like recording from a bluetooth device. Release date is TBD but the build is ready (just doing some beta testing). If you'd like to test it out, email me with your invoice number.

 

Great updates. I would appreciate if you could add the depth image for IOS. It is a necessary data for photo apps to create portrait mode or blur and etc.

 

I'll look into this, but I have no guarantees that this will eventually be implemented (and it certainly wouldn't be in the next update).

 

Sounds great. Just to clarify, this is what I mean by depth map, not the LiDar texture map.

 

I've been using NatCam and NatSuite for a LONG time (NatSuite Feb 2020 and NatCam Dec 2019). I am working on a new project in Unity 2020.1.6f1 and can't seem to find the ExposureBias, Focus or Zoom properties for the ICameraDevice. What am I doing wrong here?

Thanks!

 

Hey there. ICameraDevice is the base interface for camera devices, of which there are two implementations: CameraDevice for mobile cameras; and WebCameraDevice for webcams on all other platforms. The methods you're asking about only exist on CameraDevice instances, because NatDevice only supports these on mobile platforms. So you simply have to check if the query's device is a CameraDevice, and cast it. See the docs.

 

Thanks for getting back to me. I followed the example in MiniCam.cs and that's how I ended up with ICameraDevice instead of CameraDevice. Problem solved! Keep up the good work!

 

Hello, I am wondering if its possible to change webcam settings such as exposure, white balance, zoom, ect. I know this is possible if its a camera device but is it possible for the webcam device as well?

 

Considering purchasing as I am getting terrible latency for live mic input on mobile, but 2 questions first:

1. I'm currently using AudioSource.GetSpectrumData from a live microphone. To replicate this with your asset, my first guess would be to route audio from your API into a Unity AudioSource so I can GetSpectrumData on it, but I'm assuming that would just reintroduce all the latency anyways... Otherwise I'm guessing I'd need to find another way to get the FFT from a live source?
2. Is there any possibility of a refund/trial? I am not sure if this asset will completely satisfy my needs, it would be nice if there was a trial so I can test it in action.

 

You can't set any of these properties on WebCam devices. This is because NatDevice falls back to WebCamTexture on macOS and Windows, and WebCamTexture doesn't support any of these.