NatDevice - Media Device API

Try to capture the logs when you can. Without it I have no way of knowing exactly what the problem is.

If this doesn't work, I'm inclined to think this is a device-specific bug. The `StopRunning` call tears down everything.

 

I am encountering similar problems with audio output on iOS devices as those mentioned by dri_richard, although my use case for NatDevice would appear to be far simpler.

I use NatDevice purely to get the camera preview feed which is then fed into the ZXing library to scan for barcodes. At present I have no need for audio capture, and this is unlikely to change. My app currently also doesn't play any sound effects or music and this too is unlikely to change any time soon.

I have had several users mention that if they are listening to music on their device through Bluetooth headphones/speakers then when my app first starts running the camera, their audio stops briefly then resumes through the headphones but at a much lower audio quality, as if it's suddenly being played through a distortion filter. This continues whilst the app remains running, even when the camera feed is stopped. On quitting the app however, the audio will again stop briefly and then restart at the original quality level.

Having read through the comments between yourself and dri_richard I'm thinking this is caused by NatDevice changing audio routing setting when the camera is started. Is there anyway the NatDevice API could be extended to prevent this happening so that apps like mine that have no requirement for audio recording can bypass any audio routing changes that take place and thus stop this weird behaviour? Assuming of course that's what the problem is.

 

hi ,
What is NatRender?
Is it a native rendering?

 

Hi @Lanre,

Two questions:

1. Whats the ETA on 1.0.3 release? The feature im looking for is:

Code (bash):

1.  - Added ICameraDevice.StartRunning overloads that take in a callback which is invoked on each new camera frame.

In particular, im looking to receive the timestamp of when the frame was captured (i need the ms/us accuracy in my application, so low level timestamp from native code is crucial).
I have modified your scripts to expose that myself (as you mentioned in one of your earlier posts), but obviously I'd prefer to stay on the 'main' branch rather than rebase every time there is an update

2. in my application it would be most beneficial to be able to set / lock the shutter speed at the very least and ideally also set and lock the ISO value rather than relying on exposure lock / compensation.
I have seen you mention earlier in the thread that you would be looking into that and I was wondering if there is any ETA on this feature? I can see that the minimal android and ios versions for your library are enough to support those features for both systems.
If this is not a priority for upcoming releases, could you please pm me with the process and cost of access to the native source, so i can implement that myself?

Best,
Wiktor

 

Hi @Lanre,

We're using NatDevice v1.0.2 mainly for recording voice tracks on iOS. All works fine except for one thing: the voice is stuttering at the very beginning of the recording (around 0.2-0.3 ms). Then it quickly stabilizes to normal.

This seems to be the iOS-only issue, at least in the editor on Mac OS and Windows all seem to work well.

I've attached an example video to this post, you can notice slight distortions on "A", "One" and "Odin" parts of the video. Those are three voice tracks recorded independently and combined in one video rendered after that.

We would appreciate any help or guidance on how we could eliminate that distortion because it is really annoying for our users. Thank you!

 

This happens because NatDevice currently updates the audio session's category when a device query is created, instead of when an audio device is started. It will be fixed in the next update.

 

I'm still not set on an ETA. The reason is that I've come across a potential way to increase the performance and stability of the camera preview on Android (removing the need for native rendering, belaboring the GPU, and eating up precious frame time). I really want to get this implemented and shipped, because it would be a critical piece to the larger NatSuite Framework strategy I'll be pushing for in 2021Q1. That being said, you can always email me for my current build. Just send me your invoice number.

NatDevice currently only supports automatic exposure programs. Adding support for manual exposure (shutter speed + ISO) isn't currently planned, but I can look into it.

 

I don't know of any way to go around this. What happens is that usually on the very first time you start an audio device, iOS has to do some system audio route changes. These will cause a hiccup, which is likely what results in the stutter. The easiest way to work around this is to start the microphone before you actually need it (while respecting user privacy of course). For example, in the ReplayCam example that is included in the NatCorder recording API, the microphone is started when the scene is opened, not when recording is started (which is when it is actually needed).

 

Thanks, that would be really helpful. Is this in your current 1.0.3 build? (in which case I'll email you to request a copy!)

This would be great if possible, keep me posted!

Actually, since my last post, I realised that the MediaQuery doesn't actually seem to require the Microphone permission, so I can do it at startup and then start the microphone in a thread. If you're moving the route change to the StartRunning method (the slow part?) will that be able to run in a thread still? In which case will that eliminate the stall?

 

Not yet, I'm thinking to implement this slightly differently.

Sure thing. Also make sure to check the online changelog; it's a living document.

Yes, I plan to make sure that media devices are thread-safe. I'll make an entry in the changelog if I can do this.

 

Good morning,
I went back to the basics ... I imported the Assets in an empty project.

Version 1.0.2

I setup the MiniCam test and run the app on my Samsung S6 .. when the app runs, press the "off/on" button and the phone sleeps. Then wake it up and 1/3 to 1/5 times the camera gets stuck. Same on the S9.

APK here : https://we.tl/t-pM9BpMx3Sy

If you want the project. Just ask and I can send you the empty project.

Please help. My current project is locked up on this now ... and the boss doesn't want to release with such a lock up.

Thanks !

 

I'll look into this soon; I'm in the middle of a major rewrite of the Android backend right now.

 

Hello.
How can I remove android's `RECORD_AUDIO` permission?

 

This is not possible. NatDevice specifies this permission along with the camera permission.

 

I am using NatDevice API 1.0.2 version.
The preview works fine. But when I take a picture, the LG V50 and LG G7 get too dark.
Is there any way to solve it?

 

I'm not sure exactly what could be going on, but email me with your invoice number and I'll share the current beta build which might fix this issue.

 

I sent an email.
Please reply.

 

I thought I'd announce an improvement I'm pretty excited about coming in the next update:

On Android, NatDevice will no longer use the GPU for preparing and providing the camera preview data. So if your app is GPU-bound, you should see a nice performance increase.

All processing will be done on a worker thread on the CPU, having almost no effect on your app's performance.

 

Hi Lanre,

Wanted to get your feedback on how I am implementing NatDevice with NatCorder. I am currently getting some microphone issues in the video playback. In my app, users will record themselves speaking into their phones, so I need to record the main camera as well as microphone input into a single MP4 file.

Here is my code:

Code (CSharp):

1.  public void StartRecording()
2.         {
3.             recordingPath = "";
4.  
5.             var newClock = new RealtimeClock();
6.             var frameRate = 30;
7.             sampleRate = AudioSettings.outputSampleRate;
8.             channelCount = (int)AudioSettings.speakerMode;
9.  
10.             // Set up recorder for mp4
11.             recorder = new MP4Recorder(720, 1280, frameRate, sampleRate, channelCount, 500000);
12.  
13.             // Stream media samples
14.             cameraInput = new CameraInput(recorder, newClock, Camera.main);
15.             audioDevice?.StartRunning((sampleBuffer, _) => recorder.CommitSamples(sampleBuffer, newClock.timestamp));
16. }

The resulting MP4 file has the audio slightly pitched higher than it should be.

Also, potentially unrelated, but I've received feedback from a couple of iOS testers that randomly there will be times when the video playback has their audio out of sync with the video. The audio is greatly sped up, sounding like a chipmunk. Is there anything in the way I am handling the StartRecording that you could think would cause that? And it's not a consistent issues, but it has been reported a few times now.

Thanks!

 

If you are recording audio from an audio device, you should initialize the recorder with the audio device's format, not Unity's audio format. In other words, you should be getting the sample rate and channel count from the device, not from `AudioSettings`.

 

Hello, my Logitech C920 web camera is working in a standalone build but in the editor the devices always comes back null. Unity 2019.4.12f1. Is there something I can check to see why this is happening?

Thanks

 

How do you mean the devices always come back null? Do you mean that the device query's `currentDevice` is `null`? Can you share your code?

 

// Create a device query for device cameras
// Use `GenericCameraDevice` so we also capture WebCamTexture cameras
query = new MediaDeviceQuery(Criteria.GenericCameraDevice);

if (query.devices.Length == 0)
{
// No Camera
Debug.LogError("No cameras on this device");
return;
}



Thanks for the reply

 

@Lanre Any luck reproducing this issue? Not being able to use the Camera in the editor is really slowing me down as I have do do a build every time I want to test something.

 

@Lanre I did my initial testing on an older Motorola G4 and the photos are okay. On my iPad pro and Iphone 12 mini the photos are coming out very dark. Is there a setting i need to adjust on these newer devices?

 

Hey sorry for the late response. Can you open a new scene/project and checking the size of WebCamTexture.devices? This is pretty much what the GenericCameraDevice criterion does.

I haven't had any report of this. I'll check it out.

 

Hi @Lanre , after rebooting my machine my webcamera shows up now. When using the MiniCam sample app in a fresh project I can start the preview with my webcamera but I am not able to take a picture as

if (query.currentDevice is CameraDevice device) {

Does not return true?

 

This is because NatDevice only has "full" camera support on iOS and Android. On other platforms, it falls back to WebCamTexture. See the documentation.

 

@Lanre I've done some more tests, and I'm seeing OnSampleBuffer fired only every ~100ms on iOS. Is that what you'd expect? In the editor it's every 10ms which is adequate for my game, but 100ms is deal-breakingly slow. Please try and see if you can increase this - if not I'll have to change back to using Unity's microphone class which I was polling every frame. Which would be a shame as that suffers from other issues including random distortion etc, as you'd know!

Thanks.

 

@Lanre I'm having the same problem as @tgrayston is having.

Due to NatMic not being available on Asset Store and it's functionality being moved to NatDevice I thought it would be safe to upgrade my product to use NatDevice instead of NatMic. However that seems not to be the case as my Pitch Detection is completely broken. I can keep on using NatMic but it would be great to have NatDevice work in identical way with firing OnSampleBuffer more frequently than ~100 ms on iOS.

 

@Lanre, are you making any headway with the dark photos on iOS. Seems higher in the thread here there are issues with Android as well https://forum.unity.com/threads/natdevice-media-device-api.374690/page-52#post-6530507

Thanks
Glenn

 

I have this issue on the list of things to get to, and it's high priority. It will be conclusively fixed in the next update. Right now I'm reworking the camera code on Android, which is highest priority and is also the largest undertaking in the update.

 

Hey Glenn, issues are usually platform-specific, so an issue affecting iOS would typically be fully independent from one affecting Android. I've opened an issue for this but I haven't gotten to reproducing it yet; I've been working on higher-priority items.

 

Hey everyone, I’ve only _just_ started poking around with NatDevice’s NatMic, though I’ve had my eye on it for a long time, (thank you Black Friday Sale!!! <333 )

I’m trying to do something that to me seems like it should be a no-brainer (feed the output of NatMic into the AudioMixer chain so I can hear the mic in real-time, but with effects like reverb and stuff,) but I’m getting something strange right-off-the-bat:

The .Length of the data[] sent by NatMic to the delegate keeps changing, (1764 on one pass through the test-script, 2205 in the pass before it, and 1323 in the pass before that, etc.).

Has anyone else experienced this?

Any ideas on how to get the data from NatMic into Unity’s Audio Mixer with effects?

(I’m going to try just copying the first 1024 indices of NatMic’s data into the data in OnAudioFilterRead (which is always 1024 with my current dev system’s settings,) but that just doesn’t seem right on many levels.

I’m at a loss. :/
~Amadhia

 

Hey sorry for the late response on this. I just responded to your email. It looks like you use NatDevice after all (NatMic has been succeeded by NatDevice). The native API's make no guarantee for the buffer size, but in practice it should remain constant. What device and OS version are you seeing this behaviour on?

 

Thanks @Lanre ! Sorry for the late reply here too — the weekend and all.

Thanks for your reply email! I’ll answer here so others may also benefit from what we’re problem-solving

I’m developing on a PC, and the build is currently set to PC Standalone.

On Friday, I was able to hook the buffer from NatDevice’s microphone into 3Y3’s pitch-detection chain and it ran butter-smooth in my readouts while the audio from the mic that was being fed through Unity’s built-in audio-chain to the reverb and headphones glitched, froze, and stuttered away as I’ve been seeing since I’ve integrated the Oculus Integration.

The 3y3 pitch detection package actually a Circular Buffer in place (and all the code is in C#) to do it’s magic...

...So, theroetically... all I have to do is to figure-out how the Pitch Detector is using its Circular Buffer to keep spot-on with the incoming data, and then use that to populate the OnAudioBufferRead data and we should be working again!

But... I’m wondering... Is there a way to see all microphones attached to a system and then specify which mic that NatDevice will use, (like the Unity default audio chain can)?

In addition to the built-in microphones on my development system (PC Desktop,) I’ve got the Rift, and also a studio audio in/out box (Focusrite).

In order for NatMic to register any audio data, I had to go into the system’s Audio Devices and disable all other microphones, even though I had the one I was intending to test set as Default as both “default device” and “default communicationsdevice”

Anyway. NatDevice has made me feel hope again for making my apps that rely on mic input!!!

 

This sounds good, and is exactly what I'd expect.

The MediaDeviceQuery acts as a container for devices that it has found. The documentation doesn't mirror the current Asset Store version, so you'll probably using the `MediaDeviceQuery.devices` property. To use a specific device, simply call start running from that device. See the documentation for more info.

I'm glad to hear this.

 

When I follow the procedure listed in the documentation, I get the following error in VS:
For each statement cannot operate on variables of type ‘MediaDeviceQuery’ because ‘MediaDeviceQuery’ does not contain a public instance definition for ‘GetEnumerator’

The same error happens whether I leave the Criteria blank as in:
Var query = new NatSuite.Devices.MediaDeviceQuery();

Or if I leave the criteria as in the example script/scene:
Var deviceQuery = new NatSuite.Devices.MediaDeviceQuery(NatSuite.Devices.MediaDeviceQuery.Criteria.AudioDevice);


And would it be at all possible to provide a code-snippet for being able to pipe the output from the OnSampleBuffer callback into an audio-source/clip/whatever? I’ve been hammering at it for a few days now and only get garbled sound.

 

I mentioned this above. In your version of NatDevice, use `MediaDeviceQuery.devices`.

Such an implementation would require something like a concurrent circular buffer. NatDevice isn't designed for piping to Unity's audio engine, so I don't have any code snippets. Logically, given an implementation of a concurrent circular buffer, you'll add samples to the buffer from NatDevice, then fetch samples from the buffer from the AudioClip's PCMReaderCallback.

 

Awww, geez... I can’t believe I missed that! I’m sorry for being such a dolt sometimes!

Followed that and it worked like a charm!

///////////////////////////////////////////
(For any others who may also be not-so-bright programming-wise... you get a microphone device list like this:

var deviceQuery = new MediaDeviceQuery(MediaDeviceQuery.Criteria.AudioDevices)

foreach (device in deviceQuery)
{
...
}
/////////////////////////////////////////////

Thank you @Lanre ... and thanks for being patient with someone who’s not the brightest bulb in the chain!

Oh, hey! I actually stumbled upon that in my own kludgey way today!

Audio from the mic sounded great but the best lag I could get was ~1sec.

Knowing that that is indeed the direction to head, I’ll keep working with that tomorrow!

(My attempts using OnAudioFilterRead failed miserably.)

Thanks for the time and the tips!

 

Hey @Lanre, I’ve not been able to overcome my next stumbling-block with NatDevice and I’m hoping you can help me.

On Oculus Quest (1), after exactly 2 seconds of playing-and-holding a single tone on a flute, the input from NatDevice’s mic silences — the data going to the callback is so attenuated that it may as well be silence.

This doesn’t happen on Desktop when playing the Android targets project in Unity (and it doesn’t happen in a test build on the Quest itself I did using Unreal4, so, I know it’s not innate to the Quest hardware).

My original project was in 2019 LTS, and I’ve re-built a test from scratch in the most recent version of 2020 and the same thing happens there.

This happens both with, and without echoCancellation active (in my test, I can switch between on and off, and see a readout from device.echoCancellation to tell whether it is indeed on or not).



(The app is being built around learning to play the flute where players will be holding long tones... so the mic cutting-out after 2-seconds is going to not be good.)

 

Can you share your recording code? I'm not sure why this might happen.

 

Basically, in the OnSampleBuffer callback, I’m sending its data directly to be pitch-tracked, and also filling the end of my circular buffer for playback in the OnAudioRead callback.

Things work perfectly on the desktop system, (even in-editor set to build to Android, which is Quest’s platform).

In the main Update thread, I’ve got a dB meter processing the data directly copied to a global float[] from the OnSampleBuffer callback, and that’s showing that actual “silence” from the quest’s mic is -100 dB, and good clean tone from the flute is about -7dB.

The mic “muting” I’m seeing at 2 seconds on Quest (v1) reduces the input to around -64dB. (And bear in mind that this does not happen in my test built on Unreal 4.25... so if this is some kind of hardware thing to cancel feedback screech, it is overridable.)

Oh... the code for the dB metering is:

Code (CSharp):

1. float sum = 0f;
2. for (int i = 0; i < dataToProcess.Length; i++)
3.     {sum += dataToProcess * dataToProcess;}
4.  
5. float rmsValue = Mathf.Sqrt(sum / dataToProcess.Length);
6. dbValue = 20f * Mathf.Log10(rmsValue / 0.1f);

 

I'd still like to see your full code, so as not to run into the XY problem. I've never gotten any report of mic audio being attenuated after some time, and I would think that such a problem is unlikely given how strange it is. Echo cancellation doesn't "kick in"; on Android it's an immutable property on any given audio device.

You should be isolate NatDevice pretty easily: run the HotMic example on the Oculus. Record for more than 2 seconds, and when the audio clip gets played back after recording, see if the audio in fact becomes attenuated after 2 seconds.

 

I could build a new scene without the property things in it and .7z the whole thing to you, but I've spent the last little while exploring a couple other options, and they all have that exact same problem on Quest. My intuition is that it's something to stop feedback.

(if I hadn't built that test in Unreal4 where this _didn't_ happen, I'd have said that it must be just a part of Quest's special tweaking of Android.)

so no worries.

 

@Lanre We had a question about permissions. When we request permissions for the first time and the user grants them we are still not able to use the Camera unless the customer quits and restarts the app. Is this expected behaviour?

Is there any kind of work around available?

 

What device and platform does this happen on?

 

This is happening on my iPhone 7Plus. Will double check my android device. Also these dark photos are problem for use basically making the camera unusable. For example on my 7P the front camera is fine but the rear camera produces really dark photos.

 

I also have an iPhone 7+, but I haven't been able to reproduce it. Are you setting any exposure settings on the camera device?

 

Hi we are not modiying any defaults but we do try to request a reduced resolution photo. I can share the code we are using in a PM if that would help.