NatDevice - Media Device API

Great! Thanks for letting me know.

 

Hi, @Lanre - Back on Aug 25, 2018 in this thread, somebody asked about separate ISO, ShutterSpeed, and Aperature control. I am not seeing anything in the version I purchased. My use case is that I need a solid 30 fps capture at 1920x1080. This works well in a well lit room, but I get much lower framerate in darker spaces. Are there any parameters that I can mess with in order to keep a faster / constant frame rate in darker spaces (even if it lowers the quality of the frames)? The camera will be on a tripod and not moving, so it doesn't need to be dynamic, but it will be deployed in various spaces with different lighting.

 

Hey there, NatDevice doesn't support manual exposure. I've been looking into adding it, but because of how little interest there is and how complicated it is to actually expose, it's ultra-low priority.

Try using the exposure lock.

It sounds like you're doing computational photography (multiple exposure fusion?). This is my area of research actually, so feel free to hit me up with any questions on it.

 

@Lanre, I'm using OpenCV to process images, and really need the solid frame rate. I did try using the exposure lock. If I started in a well-lit room and then moved into a dark area, it stayed at 30 fps. The problem is if I start in a dark area and lock it, then it stays at the low frame rate even when lighting improves. Thus, it doesn't seem to be a full solution for me. I tried messing with the exposureBias, setting it to the max and min values of the range listed, and it didn't seem to help. Anything else in this library that I might be able to mess with? Or can you give me better instructions on how to use the exposure lock (i.e. once it is locked, are there any manual parameters that I can play with)?

 

Unfortunately this is as much as NatDevice currently exposes. Until manual exposure is implemented, it doesn't sound like it'll be possible to do what you want.

 

@Lanre ,I'm using Natcorder Integrate with Natdevice but i have some errors like,

DllNotFoundException: NatDevice
NatSuite.Devices.MediaDeviceQuery+<AudioDevices>d__18.MoveNext () (at Assets/NatSuite/NatDevice/Runtime/MediaDeviceQuery.cs:77)
System.Collections.Generic.List`1[T].InsertRange (System.Int32 index, System.Collections.Generic.IEnumerable`1[T] collection) (at <eae584ce26bc40229c1b1aa476bfa589>:0)
System.Collections.Generic.List`1[T].AddRange (System.Collections.Generic.IEnumerable`1[T] collection) (at <eae584ce26bc40229c1b1aa476bfa589>:0)
NatSuite.Devices.MediaDeviceQuery..ctor (System.Predicate`1[T] criterion) (at Assets/NatSuite/NatDevice/Runtime/MediaDeviceQuery.cs:53)
NatSuite.Examples.MiniCam+<Start>d__7.MoveNext () (at Assets/NatSuite/Examples/MiniCam/MiniCam.cs:40)
--- End of stack trace from previous location where exception was thrown ---
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () (at <eae584ce26bc40229c1b1aa476bfa589>:0)
System.Runtime.CompilerServices.AsyncMethodBuilderCore+<>c.<ThrowAsync>b__6_0 (System.Object state) (at <eae584ce26bc40229c1b1aa476bfa589>:0)
UnityEngine.UnitySynchronizationContext+WorkRequest.Invoke () (at <42a5878ce129403083acccf18e43363f>:0)
UnityEngine.UnitySynchronizationContext:ExecuteTasks()


MyCode:



/*
* NatSuite Examples
* Copyright (c) 2020 Yusuf Olokoba
*/

namespace NatSuite.Examples {

using UnityEngine;
using UnityEngine.UI;
using Devices;
using Recorders;
using Recorders.Clocks;
using Recorders.Inputs;

public class ReplayCam : MonoBehaviour {

[Header(@"Recording")]
public int videoWidth = 1280;
public int videoHeight = 720;
public bool recordMicrophone;

[Header(@"UI")]
public RawImage rawImage;
public AspectRatioFitter aspectFitter;

private IMediaRecorder recorder;
private CameraInput cameraInput;
private AudioDevice audioDevice; // Used to record microphone

async void Start () {

if (!await MediaDeviceQuery.RequestPermissions<AudioDevice>()) {
Debug.LogError("User did not grant microphone permissions");
return;
}
// Create a media device query for audio devices
var query = new MediaDeviceQuery(MediaDeviceCriteria.AudioDevice);
// Get the device
audioDevice = query.current as AudioDevice;
Debug.Log($"{audioDevice}");
// Get a microphone

}

public void StartRecording () {
// Create recorder
var microphone = recordMicrophone ? audioDevice : null;
var clock = new RealtimeClock();
recorder = new MP4Recorder(videoWidth, videoHeight, 30, microphone?.sampleRate ?? 0, microphone?.channelCount ?? 0);
// Stream media samples to the recorder
cameraInput = new CameraInput(recorder, clock, Camera.main);
microphone?.StartRunning((sampleBuffer, timestamp) => recorder.CommitSamples(sampleBuffer, clock.timestamp));
}
private void OnReplay (string path) {
Debug.Log ("Saved recording to: " + path);
// Playback the video
#if UNITY_EDITOR
UnityEditor.EditorUtility.OpenWithDefaultApp (path);
#elif UNITY_IOS
Handheld.PlayFullScreenMovie ("file://" + path);
#elif UNITY_ANDROID
//Handheld.PlayFullScreenMovie (path);
using (var payload = new SharePayload (
completionHandler: () => {
Debug.Log ("User shared video!");
}
))
payload.AddMedia (path);
#endif
}

public async void StopRecording () {
// Stop committing media samples
if (audioDevice.running)
audioDevice.StopRunning();
cameraInput.Dispose();
// Finish writing video
var recordingPath = await recorder.FinishWriting();
Debug.Log($"Saved recording to: {recordingPath}");
// Playback recording
var prefix = Application.platform == RuntimePlatform.IPhonePlayer ? "file://" : "";
OnReplay (recordingPath);
//Handheld.PlayFullScreenMovie($"file://{recordingPath}");
}
}
}

 

Hey there, are you seeing this error on Windows? If so, make sure to install the latest Visual C++ redistributable. Also, please use code tags instead of pasting code in your question.

 

Hi @Lanre,

Slight update from this previous issue. We had to rollback to NatDevice 1.0.2 because the 1.1.0 version does not support iOS <13. I confirm that the 1.0.2 version is way way faster to load the camera on the same Android device. It only takes less than 1 second. In version 1.1.0 it can takes around 4 or 5 seconds.

Of course I don't know what changes were made to NatDevice and what drove these changes, I would hope for the loading time issue to be resolved in a future version. As for the iOS compatibility, as long as we need to support iOS <13, we'll stay on 1.0.2 I guess.

 

Interestingly enough I think I found one of the culprit that extend the loading time in 1.1.0.

The MediaDevicePermission script from 1.0.2 is not working properly on Android (it is not explicitely said in the changelog by the way - only for UWP) :

RequestPermissions<CameraDevice>()

does not return false if the user did not grant permissions. So I went ahead and copied the MediaDevicePermission code from 1.1.0, now the code "works" but there is additionnal delay.

Found this in the RequestAndroid function :

Code (CSharp):

1. if (!Permission.HasUserAuthorizedPermission(permission))
2.     Permission.RequestUserPermission(permission);
3. /**
4.  * Unity dooesn't provide a callback for completion, and doesn't provide an indeterminate state.
5.  * so instead we're gonna have to wait for an arbitrary amount of time then check again.
6.  */
7. yield return new WaitForSeconds(3f); // This should be enough for user to decide
8. var result = Permission.HasUserAuthorizedPermission(permission);

Unfortunately I don't know of an asynchronous alternative to Unity's Android.Permission, but the synchronous one from Yarsirkula's isn't bad.

 

Good catch. I had to add this delay because Unity's Android.Permission API is severely lacking. I don't want to inline Yasirkula's library (I try to keep dependencies to an absolute minimum), but I think you can swap out `MediaDeviceQuery.RequestPermissions` with his library.

 

That was indeed my plan, it works great this way. And I totally respect your (good) decision to keep dependencies to a minimum. It might be great to warn new developpers in the documentation that there is a fixed wait time when using your permission system for Android, I think this could avoid some trouble.

 

This is a good suggestion. I'll update the docs with this detail. Sorry for the inconvenience!

 

I'm facing this in 2021, how to use NatShare + NatDevice to grab Augmented Reality stuff?
The issue is exact the same from @brzzda.

Lanre, we need to submit it this weekend.
Could you help us?

 

NatDevice 1.1 fixed the memory leak in photo capture on iOS. Perhaps you are facing a different issue of some sort? Can you share more information, like screenshots from Xcode? What versions of Unity, ARF, NatDevice, iOS are you using?

 

hello,
have a issue with NatDevice and Unity 2021.2.01f
Assets/NatSuite/NatDevice/Runtime/Internal/NativeCameraDevice.cs(172,17): error CS0165:
Use of unassigned local variable 'handle'

line 172 :
handle.Free ();

how to fix that?
need a quick fix .... thx
M.

 

Just change

GCHandle handle;

to

GCHandle handle = default;

 

Hello,
many many thx for the quick fix!
wish the Dev would care more about his products...
but looks like he has no interest in fixing issues anymore...
so can not recommend this product...

I'm now on Unity 2021.2.3f1 ... and getting lot's of problems with this plugins...
non of them work ... :-(

so fresh project... only import NatDevice ... open demo project...
and voila ... error after error...

I'll fixed this
GCHandle handle;
to
GCHandle handle = default;

but other error popping up...
wish the Dev would care ... but no support as it looks...

and that's what I'll really dislike on Assetstore... so many great plugins and assets... but not many dev's do support...
they only want to make quick money...
very very disappointed from NAT as well ...
he only focusses on NatML ...

NatDevice is dead... don't buy ... doesn't work and dev does not care :-(

 

The interesting thing about this is that I can't reproduce this error on Unity 2019.4 or 2021.1, on either .NET Standard or .NET 4x. It must be something introduced in Unity 2021.2. In any case there'll be a fix in the NatDevice 1.2 update.

Not at all, there are some very exciting updates coming to NatDevice very soon.

As for the DllNotFound error when requesting permissions, are you running on macOS or Windows? And if macOS, are you on Intel or Apple Silicon?

 

hello,
many thx for quick reply...
yes I'm on Mac OS X ... Silicon & Intel same issue...
and yes it only happened with 2021 version...
but need to use it on my silicon machine...

also wondering ... who is to blame... Unity for introducing some new strange stuff...
found also another plugin PDF reader has a issue too...
all other plugins I'll use normally work as expected...

hope for a fix soon... if I'll can beta test... let me know... I'll really need this plugin for my work
M.

 

Join us on Discord! I plan to send out a NatDevice beta build later today.

 

Hello @Lanre ,

I bought NatCorder and I need to record audio from a microphone on iOS connected via the Lightning port. How do I solve this task? Could I use NatDevice for this and connect with NatCorder?

Thank you in advance!

 

Hey there, NatDevice supports streaming mic audio from internal and external audio devices, including bluetooth microphones.

 

That sounds great! But just to make sure, does NatDevice work together with NatCorder? Will the sound be in the recorded video created by NatCorder?

I think this code snippet from you shows how NatDevice works together with NatCorder if I'm not mistaken:
https://gist.github.com/olokobayusuf/df6672fb86908dccaa656c4db7e99e3a

 

Yes, NatCorder and NatDevice are built with tight integration in mind. See the docs.

 

Awesome, thank you!

Now one last question, is there anything particular I need to set up or take care of in NatCorder for long recordings (up to around 30 minutes) on iOS (iPhone 13 Pro)?

 

You'll just need to make sure you've got enough space on your device. But besides that, NatCorder has no recording duration restrictions beyond the implicit ones from your device (storage available).

 

@Lanre
I'm finally working on implementing your frameworks (targeting iOS) and I'm getting this error in Xcode when starting to record with microphone:

NatDevice Error: AudioDevice Built-In Microphone failed to start recording with error: Error Domain=com.apple.coreaudio.avfaudio Code=-10851 "(null)" UserInfo={failed call=err = AUGraphParser::InitializeActiveNodesInInputChain(ThisGraph, *GetInputNode())}

And this is how my code looks like:

Code (CSharp):

1.         private void Start()
2.         {
3.             _currentState = RecorderState.Ready;
4.             var query = new MediaDeviceQuery(MediaDeviceCriteria.AudioDevice);
5.             _audioDevice = query.current as AudioDevice;
6.         }
7.  
8.         public void StartRecording()
9.         {
10.             if (State != RecorderState.Ready)
11.             {
12.                 return;
13.             }
14.             State = RecorderState.Recording;
15.  
16.             // Create recorder
17.             var clock = new RealtimeClock();
18.             _recorder = new MP4Recorder(videoResolution.x, videoResolution.y, 30,
19.                 _audioDevice.sampleRate, _audioDevice.channelCount);
20.      
21.             // Setup media sampling
22.             _cameraInput = new CameraInput(_recorder, clock, videoCamera);
23.             _audioDevice.StartRunning((sampleBuffer, timestamp) =>
24.                 _recorder.CommitSamples(sampleBuffer, clock.timestamp));
25.         }
26.  
27.         public async void StopRecording()
28.         {
29.             if (State != RecorderState.Recording)
30.             {
31.                 return;
32.             }
33.             State = RecorderState.Storing;
34.      
35.             // Stop media sampling and recorder
36.             _cameraInput.Dispose();
37.             _cameraInput = null;
38.             _audioDevice.StopRunning();
39.      
40.             // Flush data to app persistent data folder
41.             var videoFilePath = await _recorder.FinishWriting();
42.             // Move video to camera roll
43.             MoveVideoToCameraRoll(videoFilePath);
44.         }

Permissions for camera and microphone are given.

Also another important issue is that I'm recording in AR with an iPhone 12 Pro and I use 1920x1080 as the recording resolution. Sadly the recording is stretched/squished which is a no go for me. Do you know how I can fix this issue? Is it possible to do a cropped recording for example? Unfortunately I have to record in FullHD and can't use the devices screen native resolution.

This is how it looks in the recording:


And ingame:

 

Make sure that "Prepare iOS for Recording" is enabled in Player Settings.

There isn't a way to crop currently. When recording an AR app (ARFoundation, Vuforia, and so on), you have to make sure that the aspect ratio of the recording is the same as that of the screen. It doesn't have to have the exact screen resolution.

 

Thank you, that fixed it!

Sadly it actually it has to be the same resolution as the active XRCameraConfiguration on the ARCameraManager or otherwise it will try to do some weird compensation with the FieldOfView and you end up with a squishy image, even when using the correct aspect ratio.

Ok so it's not a hard technical limitation but more of a time/priority constraint on your end to work on such feature, right? If so, please write me a DM and let's continue talking there, maybe we can find a solution

 

Can you share a video and a screenshot illustrating this? You'd be the only dev who has seen this (I haven't seen this myself).

I'll send you a DM.

 

hmm save to heic really ought to be included

 

This would have to go into NatCorder, but it's actually a pretty good feature request. We've added it to the roadmap. I'll update you on which release we plan to launch it in.

 

One of my tablet has the same behaviour, did you ever solved it?

 

This isn't an issue that NatDevice can solve. We have no way of detecting whether the preview data is stretched or by how much it is stretched. We simply pass along the data that is provided to us by the Android system.

 

I realize this may not be applicable, but I'm using NatDevice for green screen stuff, and crop the image using a shader.
It works in RawImage UI Elements and when displayed as a Material Override map. Our shader is highly specific so I can't share it here, but it's worth looking into as a simple solution!

 

Hey @Lanre thanks for the great asset!
I'm running into an issue that I'm not sure how to solve, hopefully you can help me?
Long story short, I'm getting the following two errors, and I'm not sure why. It happens seemingly arbitrarily, on play, stop, and some other things in my app.
Any ideas?

 

I had the same issue after updating NatDevice. I recommend to look at the MiniCam.cs example. I had to dispose any "TextureOutput" that has been created.

Code (CSharp):

1. void OnDestroy () {
2.     // Stop the camera preview
3.     if (cameraDevice?.running ?? false)
4.         cameraDevice.StopRunning();
5.     // Dispose the preview texture output
6.     previewTextureOutput?.Dispose();
7. }

 

Ah I think this happens if you don't stop the preview when the game exits play mode. The `MiniCam` scene has an `OnDestroy` method to dispose the preview output and stop the camera when exiting play mode. I recommend doing the same in your code.

 

In the next update we'll add a check to do this automatically. We already stop the camera device automatically when exiting play mode.

 

Hi!

We got an app crash on iOS due to using NatCoder microphone api. It works pretty fine in regular cases.
Crash case:
1. Run the app
2. Receive a call from another person(so, microphone is being activated from another app)
3. Keep call running, and switch back to the app
4. Launch the mic throw api IAudioDevice.StartRunning (SampleBufferDelegate @delegate);
5. You will get unhandled exception here

Seems our users where able to get app crash without activated mic by 3rd app(we are investigating that). But at least we found this one, which has pretty similar end of stack trace.

Stack trace of exc in xCode(crash report):

 

Hey there, are you able to reproduce this with NatDevice 1.2.0? We got a report of this from Unity, and we added checks to make sure that we caught the exception.

 

We are using NatDevice 1.0.2(get the info from Changelog file inside Plugins/Managed folder)
Will try updating the package! Thanks!

UPD: unfortunately we can't update, because latest version "Needs Unity upgrade to version 2020.3.26", but we are on 2019.4.23f1

 

You can import NatDevice 1.2.0 into Unity 2019.4. You'll only have to modify the MiniCam example code to no longer use a C# feature that was introduced in Unity 2020. Replace this line in the `CapturePhoto` method:

Code (CSharp):

1. using var photoTextureOutput = ...

with:

Code (CSharp):

1. var photoTextureOutput = ...
2. ...
3. // Add this at the end of the method
4. photoTextureOutput.Dispose();
5.  

 

Thanks @Lanre !

I have a question. Test case with mic does not work. The audio output is just silent. And that happens, because MediaDeviceQuery returns 3 devices in the list, and microphone is not selected as "current". So, I had manually to get deviceQuery[2] to get it working. I wonder why it happens(in example script you use deviceQuery.current, and it also worked for us with previous version, but not this). How can we pick current microphone, instead of just current audio device?

Code (CSharp):

1. var deviceQuery = new MediaDeviceQuery(MediaDeviceCriteria.AudioDevice);
2. _microphone = deviceQuery[2] as AudioDevice;

Thank you

UPD: here is a screenshot of query response:
UPD2: Another question. We don't have a crash now, but we have "null" result in clipOutput.ToClip api(and that happens because of Samples rate and Channel count are 0). How can we know that microphone can't record anything because user has a call running on background at the start of recording? not at the end

 

The query acts like a cursor, so `query.current` is simply the current device that is being pointed to.

In NatDevice, AudioDevice instances are always microphones. They are the same thing.

Can you share your recording code? This happens when the clip output hasn't been given any audio samples to record.

 

The code is pretty similar to your example code. The issue happens if we have a call running on the background. Looks like if background app is using the mic(call), we can't get any samples to record. But I would to know that before starting to record(to inform user), not in the end.

 

Can you share the logs? NatDevice should log something along these lines when it happens:

Code (CSharp):

1. NatDevice Error: AudioDevice ... failed to start running because audio input bus is unavailable

And `audioDevice.running` should return false after the call to `StartRunning`.

 

@Lanre is it possible to get the audio input amplitude (is this the correct term?) of the currently selected microphone?
I need to present some simple visual feedback that microphone input is coming in.

 

When the microphone is running, the amplitude is simply the value of an element in the `sampleBuffer` array. You can compute a reduction over it, like the mean, in order to display to the user.

 

I don't know much about audio engineering, could you please provide me with some pseudo code, so that I get an idea how to implement it? Thank you!