NatDevice - Media Device API

Just loop over the sample buffer and calculate the mean (average).

Code (CSharp):

1. void OnAudioBuffer (AudioBuffer audioBuffer) {
2.     // Calculate average value of `audioBuffer.sampleBuffer`
3. }

 

Hi @Lanre .
I'm having an issue where only iPhoneX device is not starting preview. Other iOS devices seem fine (iPhone 8 iPhone 13 Pro Max).
I just build MiniCam scene.When I open app crash.

SIGABRT received (abort() call) -- WHOA! I did something nasty. Dumping the stack trace:
0 myapp 0x0000000100a8ee38 __sig_handler + 172
1 libsystem_platform.dylib 0x00000001db96bd48 F26B24BD-CA8F-353C-B031-C5DB16FC6715 + 11592
2 libsystem_kernel.dylib 0x00000001bb014be0 3D363709-8D2D-3D1E-8CAB-11D232E5D508 + 187360
3 libsystem_kernel.dylib 0x00000001bb014c10 3D363709-8D2D-3D1E-8CAB-11D232E5D508 + 187408
4 TCC 0x00000001d30d6708 136B9A47-21E9-343C-BD88-CC1BFB095041 + 30472
5 TCC 0x00000001d30d13d0 136B9A47-21E9-343C-BD88-CC1BFB095041 + 9168
6 TCC 0x00000001d30d3e94 136B9A47-21E9-343C-BD88-CC1BFB095041 + 20116
7 libxpc.dylib 0x00000001db9ae350 5148C0B9-5920-3361-9AB6-83448AA80146 + 103248
8 libxpc.dylib 0x00000001db9a26bc 5148C0B9-5920-3361-9AB6-83448AA80146 + 54972
9 libdispatch.dylib 0x00000001805d810c F14F0161-E0DE-3D9C-851E-AD12F95A3073 + 409868
10 libdispatch.dylib 0x00000001805ac8e4 F14F0161-E0DE-3D9C-851E-AD12F95A3073 + 231652
11 libdispatch.dylib 0x00000001805bd2e4 F14F0161-E0DE-3D9C-851E-AD12F95A3073 + 299748
12 libsystem_pthread.dylib 0x00000001db977e48 _pthread_wqthread + 340
13 libsystem_pthread.dylib 0x00000001db97793c start_wqthread + 8

 

Have you added a "Camera Usage Description" in Player Settings in Unity?

 

It works! Thanks!!

 

Is this package no longer available outside of the monthly subscription version starting with 1.2.2?

I'm working on a project for an org which purchased this package from the Asset Store. The last version we had installed was 1.1.0, and since 1.2.1 is the latest version published, with zero files in the package archive, it seems that we cannot integrate the updates from version 1.2.0 (which fixes at least one build issue with the 2021 LTS version of Unity).

 

That's correct. We've migrated to a new distribution model using subscriptions. Your organization can create an account on NatML Hub and upgrade to the Cloud plan to use the latest versions of NatDevice. You can grab NatDevice from GitHub.

 

Not too happy about this news.
Where can we download the latest version without a subscription when we paid on the Unity Asset Store?

 

Well there is an issue with the asset store. The 1.2.1 package is empty... And I don't think we can download previous version from the asset store. So, we're currently stuck.

Whats is the last version available to user that bought it on the Asset Store? I hope that we are able to at least have the few fixes from 1.2.1 and 1.2.2 versions without having to subscribe to the cloud plan.

 

NatDevice 1.2.0 is the last version that was on the Asset Store before we switched to the Cloud Plan subscription model. You can DM me with your invoice number and I'll share it with you. All updates after this can only be used through the NatML Cloud Plan.

 

The last version was 1.2.0. Unfortunately we aren't adding fixes to this old version; you need to upgrade to the Cloud plan to use the latest and greatest.

 

I can't justify the cost of the new subscription when all I need is microphone streaming. Hoping this last version will continue to work with new versions of Unity for a while, and that you might consider offering a lower cost NatSuite-only subscription level for those in my situation with no interest in machine learning features etc.

Looking at the changelog for 1.2.1+, should I be concerned about "high CPU usage when streaming from an AudioDevice on Android" not being included? How big an issue is that?

I should be able to continue to work around "bug where AudioDevice fails to stream after creating a second MediaDeviceQuery on iOS" but it's frustrating to be denied a bug fix.

 

We're executing on a strategy that requires one subscription for access to all of our core API's. If it helps, we're introducing a discounted yearly price for the Cloud plan. More details to come in our Discord community.

A user found that the `AudioDevice` was throttling the CPU, causing CPU usage to be very high (basically a thread running a hot loop). It will adversely affect performance of your app, especially if your app is doing other non-trivial things. Finally, it's gonna drain your users' battery, so I strongly recommend upgrading.

 

It is definitely a weird way to proceed.

I have no issue with the fact that you have to make money to keep maintaining your software. But to keep performance/bug fixes like these behind a paywall is definitely not something I agree with.

I might be fine with a low cost subscription that give access to NatDevice without the ML stuff I don't need. But this raises other questions.

How does the access key system work currently? What happen if I don't renew my subscription? Is the webcam functionnality disabled in previously built apps? How does it work if my apps don't have access to the internet?

 

No software is without bugs. That's why we need to keep investing in upgrades. It's hard to think of any product that offers free maintenance in perpetuity (think your car, fridge, iPhone).

We've put all of our API's under one subscription because we're going to be blurring the lines between them. We're only at the very beginning stages, but we plan to introduce solutions that use functionality from combinations of our core API's (e.g. we can make a video editing solution with NatCorder and NatML).

First off, thank you for asking these very relevant questions. I'm working on adding them to our documentation. The system is pretty simple: NatDevice and NatCorder need an app token in order to work. When you build from Unity, an app token is generated and embedded into the app. They last forever, so it doesn't matter whether you cancel your subscription. You can view the code that handles this.

Once your app has been built from Unity, the app token is valid forever. It doesn't matter whether you then cancel your subscription. But one thing to note is that you won't be able to use NatDevice in the Editor, because the editor also needs to generate an app token when Unity is opened.

Nope.

App tokens are validated entirely on device. The native library does not need or use internet access whatsoever--for any reason.

Let me know if you have any further questions, and thanks for bringing these up. I'll get these answers on the online docs ASAP.

 

Hey all,

NatDevice 1.2.3 is now available:

+ Added support for WebGL. NatDevice now brings all of its features to the web, with no changes in code.
+ Added `IMediaOutput<T>` interface for working with media device outputs.
+ Added `AudioSpectrumOutput` output for computing the audio spectrum of audio buffers using the Fast Fourier Transform.
+ Added `AudioBuffer` constructor that accepts a native `float*` buffer.
+ Added `CameraImage.rowStride` property for retrieving the row stride of interleaved camera images.
+ Added `TextureOutput.orientation` convenience property for getting and setting the texture orientation.
+ Fixed some `CameraDevice` instances not being discovered on low-end Android models.
+ Fixed bug where continuous autoexposure routine stopped after setting exposure point on iOS (#1).
+ Removed `TextureOutput.pixelBufferOutput` property. This is now an implementation detail that is subject to change.
+ Removed `TextureOutput` constructor that accepts `PixelBufferOutput`. The constructor is now parameterless.

We're most excited about WebGL support, and for audio spectrum analysis with the FFT:


Grab the update from GitHub.

 

Hi!
We are using version 1.2.0 and are encountering a strange audio recording error on some android devices. On Motorola Edge 20 Lite which I am currently testing on I get these error:



This is how I setup the microphone and start the audio record session:



We use this on both android and iOS. On iOS we have no issues at all. While on android some devices works fine while on others we encounter that issue where recording audio doesn't work at all.

Any ideas what the issue could be?

Edit: I went back to test older version of NatDevice (1.0.2) which we used before, on that version audio recording works. However, we still would like to use the 1.2.0 version.

 

Hey there, I recommend upgrading to NatDevice 1.2.4. We fixed an issue where audio devices failed to start on 1.2.1.

 

We are currently using unity version 2019.4.23f1. Is it possible to upgrade to NatDevice 1.2.4 without upgrading the unity version also?

Thanks for swift reply!

 

Hm I don't think so, because NatDevice (and all our other APIs) use C# 8. Unity 2021 is already in LTS, so I strongly recommend upgrading your Unity version if possible.

 

Ok got it, thank you!

 

The removed NatCam from the Assetstore ;(WHY?????

 

NatCam has been deprecated for years now. We've moved to a new distribution model where we have all of our releases on GitHub, and we now use subscriptions. You can still download the old versions of NatCorder and NatDevice from before we made this shift by going to the NatML Hub archive.

 

Using NatDevice 1.2.5 with OnePlus 9, game crashes with this error:

Right after the game launches (microphone usage and etc comes later in the game so no idea what could trigger it).

We used older versions of NatDevice (from AssetStore) and had to purchase Cloud subscription hoping to have the issue fixed. With older version of NatDevice, mic was just not working and that's it, but with 1.2.5 it's a straight off crash on the same device.

Other devices seem to work with no issues, at least from what I have. Is there any solution to this?

 

Hey Igor, does this crash only happen on the OnePlus 9? And are you able to reproduce it locally on one of your devices?

 

Happens only on OnePlus 9. Cannot reproduce on several Samsung tablets, Pixel 6 Pro and iPhone 13 Pro Max.

 

Okay that's good to know. Also, that screenshot looks like Firebase? Does it provide any more information than what you shared in that screenshot? Any and all information you have is helpful.

 

Managed to extract crashlog via Android Studio, link: https://pastebin.com/NM7TxcQW
Hope this is helpful.

 

Thank you! Here's the issue to track.

 

Hi, I've been trying desperately to use an external USB mic with my Android NVidia Shield for awhile without any success.

If I use Unity's Microphone class the external Microphone isn't listed. Instead I get 'Android Audio Input', 'Android Camcorder Input' and 'Android Voice Recognition Input' listed. No matter whether the USB mic is plugged in or not, the list is the same. If I enable 'Auto USB Audio Routing' in the Android Developer Options, when I plug the USB mic in, Unity seems to automatically pick up the mic stream but I want full control over which USB device is selected, not what was plugged in last.

Therefore I've tried using NatDevices and the following code:



I only get a device called 'SHIELD Android TV' - no matter whether external USB devices are plugged in or not. If I run this same code on Windows it successfully seems to pick up the USB Mic but I need a solution for Android.

Really can't understand why this is so hard on Android to be able to list and select a USB Mic. Any help hugely appreciated as been stuck on this for some time now.

Thanks, Jake

 

Hey Jake, what version of NatDevice are you using? We've changed how we handled device discovery between releases, so different versions typically produce different results.

 

Appreciate the fast response.

I'm using version 1.2.5 installed via package manager.

 

This issue would be from Android itself. This is the method that NatDevice uses to discover available audio input devices. If NatDevice doesn't discover it, then it's cos Android isn't reporting the microphone.

 

I see. So it seems to be an issue with the NVidia Shield then - perhaps obfuscating the mic devices and using its own solution so everything is funneled through this 'Shield Android TV' device.

 

Hey all, we just published NatDevice 1.3.0. First, the biggest change:

NatDevice can now be tried out for free!

We added a free trial with limited functionality (see the docs for more details). Here's the full changelog:

• NatDevice can now be tried without an active NatML MediaKit subscription!
• Greatly improved camera streaming performance by adding thread safety and multithreading to `CameraDevice`. This introduces a breaking change whereby the camera image callback is no longer called on the Unity main thread. Below is a 1080p camera preview benchmark on the Samsung Galaxy S21:

• Improved `AudioDevice` memory behaviour on Android by eliminating sample buffer allocations.
• Improved garbage collection behaviour in `PixelBufferOutput` and `RenderTextureOutput`.
• Added `capacity` parameter to `MediaDeviceQuery` constructor to limit number of devices to discover.
• Added `TextureOutput.onFrame` event for listening for new camera images in the texture output.
• Added `RenderTextureOutput.onFrame` event for listening for new camera images in the texture output.
• Fixed `CameraDevice.CapturePhoto` not working on WebGL.
• Fixed crash when `MediaDeviceQuery` is created on older Android devices (#12).
• Fixed sporadic crashes while running the camera preview on Android (#11, #13, #14).
• Fixed rare crash immediately app is loaded on Android (#8).
• Removed `CameraDevice.exposureLockSupported` property.
• Removed `CameraDevice.focusLockSupported` property.
• Removed `CameraDevice.whiteBalanceLockSupported` property.
• Removed `CameraDevice.exposureRange` property.
• Removed `CameraDevice.exposureLock` property.
• Removed `CameraDevice.focusLock` property.
• Removed `CameraDevice.whiteBalanceLock` property.
• Removed `CameraDevice.exposurePoint` property.
• Removed `CameraDevice.focusPoint` property.
• Removed `CameraDevice.torchEnabled` property.
• NatDevice now requires iOS 14+.

Grab NatDevice 1.3.0 from GitHub.

 

Great news, curious about the limitations

 

Working on docs right now! Should be up before EOD today.

 

I am trying to run NatDevice with active Cloud subscription yet trying to use it on device gives me such error:

Code (CSharp):

1. System.InvalidOperationException: NatDevice session token is invalid. Check your NatML access key and plan.

Am I missing something? The token has been rechecked multiple times, no extra spaces and etc. Everything is set in project settings correctly. The only difference is that I'm building the project on a separate build machine (NatMLHub.asset file which is generated upon adding token is there), is that the limitation?

Account:

i.boyko@chillgaming.net

 

Hey, what platform (iOS, Android, etc)? And is your bundle ID in Player Settings different from the bundle ID at runtime? Finally, can you share the logs from the device in a .txt attachment? Building on a separate machine should not have any effect.

 

Tested running on Pixel 6 Pro and OnePlus 9. Haven't reached the moment of testing iPhones yet.

Bundle ID is not changing at any point (from what I understand, as we're not running any "embedded player" stuff)

Log:

Code (CSharp):

1. 2022-11-02 15:32:11.082 23661-23724 Unity                   com.Redacted.Redacted             E  [Dissonance:Recording] (12:32:11.081) CapturePipelineManager: Unexpected exception encountered starting microphone capture; local voice transmission will be disabled: System.InvalidOperationException: NatDevice session token is invalid. Check your NatML access key and plan.
2.                                                                                                       at NatML.Devices.Internal.NatDevice.CheckStatus (NatML.Devices.Internal.NatDevice+Status status) [0x00000] in <00000000000000000000000000000000>:0
3.                                                                                                       at NatML.Devices.MediaDeviceQuery+<GetAudioDevices>d__14.MoveNext () [0x00000] in <00000000000000000000000000000000>:0
4.                                                                                                       at System.Collections.Generic.List`1[T].AddEnumerable (System.Collections.Generic.IEnumerable`1[T] enumerable) [0x00000] in <00000000000000000000000000000000>:0
5.                                                                                                       at System.Collections.Generic.List`1[T].InsertRange (System.Int32 index, System.Collections.Generic.IEnumerable`1[T] collection) [0x00000] in <00000000000000000000000000000000>:0
6.                                                                                                       at NatML.Devices.MediaDeviceQuery..ctor (System.Predicate`1[T] filter, System.Int32 capacity) [0x00000] in <00000000000000000000

 

Can you share the full and unfiltered logs from app start? There should be more info that is logged earlier.

 

Sent full log via personal messages, thanks.

 

Got it, responded!

 

Hi @Lanre, I have an issue with setting the FPS to 24. It keeps getting reset to the previous value instead. Is there a reason for this?

 

Hey there, the actual frame rate that is set depends on the camera device itself. The camera only supports a fixed set of frame rates, so when you set a value with NatDevice, we set a supported frame rate closest to what you request. You can check the logs as NatDevice will log the value that was actually set.

 

Hi @Lanre I'm also having this issue where I'm trying to set the phone to 4K and it immediately crashes the application on my iPhone XR. This is error statement: stop reason = EXC_RESOURCE RESOURCE_TYPE_MEMORY (limit=1850 MB, unused=0x0)

 

The error shows that your iPhone is running out of memory, so you can't use 4K. Try reducing your resolution.

 

So went to update a 2019 project that used NatCam and NatShare and found out it is deprecated and replaced with NatML. So looking at the website I'm not sure what product I need to subscribe to get that functionality back? Sorry if stated already in this thread but don't have time to look back through 50 odd pages.

Cheers.

 

Hey there, no worries. NatCam has been deprecated and was succeeded by our NatDevice product. NatDevice, along with our NatCorder video recording product, is now under our VideoKit umbrella, so you'll need a VideoKit Core subscription. VideoKit basically combines NatDevice, NatShare, and our other products to create a single solution for doing user-generated content. Check it out.

In any case, the VideoKit Core subscription gives you access to NatDevice, NatCorder, VideoKit, and any combination thereof.

 

I am trying out the new free trial. It works in the editor, but when building the same code for Android I get the "NatDevice session token is invalid. Check your NatML access key and plan." exception when making a MediaDeviceQuery. I can't find information on whether the trial is supposed to apply to Android builds or not.

 

Hey there, the trial doesn't restrict which platforms you can run on. Regarding the error you're facing, make sure that your app's bundle ID doesn't have any underscores, because that would cause an error when generating a session token. If you still face the error, feel free to PM me with the full logs from `logcat`.