NatDevice - Media Device API

Hello
Two users have reported
The camera gets stuck on the first frame.
Mobiles : Huawei Honor 8C and Samsung Galaxy A10

 

I just open the natdevice camera, the amount of remaining RAM goes from 651 to 228 !!!!!
I have already sent you the report of this mobile phone, the same mobile phone that had the problem of collecting the output image

 

Sorry for the late response; Unity didn't send me any notifications. This sounds consistent with iOS' behaviour. If you start recording from a different device than the current device, iOS will first issue a system-wide route change before proceeding. Unlike say macOS, you can't have multiple parallel audio routes. To find AirPods, the device name should have the word "AirPod" in it. To find any external device, check for any device that doesn't have the word "built-in".

 

Do you have logs by any chance? What versions of Android are these devices running?

I'll try to reproduce this and get back to you.

 

We've figured this out - and can reproduce it without AirPods.
Our loop is:
- Create MediaDeviceQuery and choose a device
- Enable echo cancellation on the device
- Start running
- Stop running

We can eliminate the crash if we don't set echo cancellation.
It seems that NatDevice will crash if a new MediaDeviceQuery is created if a device has been modifed (e.g. by setting echoCancellation)

 

This wasn't our experience with NatMic. The audio played through AirPods right from the start of the game, and continued even when we started recording from the builtin mic.

Isn't it in fact related to our earlier discussion about AVAudioSession options?

I can keep the audio playing through AirPods by setting AVAudioSessionCategoryOptionAllowBluetoothA2DP after querying for the microphone, but it causes a break in audio output. Seems there's also a stutter when NatDevice sets the audio session category & options too. Would be great if we could avoid that by having more control over how (and if) NatDevice modifies this.

 

No, unfortunately I do not have a log ,

Huawei Honor 8C => android 8
Samsung Galaxy A10 => android 10


Thanks for the reply
I am waiting for your response .

 

And just to confirm, this happens with the beta build I sent you? If so, I'll create a new issue to look into this.

 

Starting recording shouldn't affect audio output. The only thing we do with system audio when recording is started is this:

Code (CSharp):

1. [AVAudioSession.sharedInstance setPreferredInput:port error:&error];

NatMic and NatDevice both set the audio category with the same options (I wrote the exact code earlier). And they both happen at the exact same: when you create a `MediaDeviceQuery`. Ideally, you should only create a device query once then use it throughout your app. It isn't as lightweight as it seems, because discovering devices can (in this case) modify the system configuration, and keeps strong references to native objects throughout its lifetime.

 

For the camera getting stuck I haven't been able to reproduce it, and I haven't gotten any similar reports. Without logs I have no clue what could be going on. I do have memory optimizations planned.

 

Yes it does.

 

Our QA team tests for users connecting & disconnecting headsets, and headsets losing battery power, so the query could become invalid.
We’re writing a native plugin to get notifications of audio routing changes (so we can pause the game if a headset is disconnected). We could re-run the query at that time.

 

I see; this makes sense. I'm going to try to reproduce this. I have a potential fix in mind, just gotta verify if it works.

 

Is it a known limitation that IAudioDevice.StartRunning() can't be run on a separate thread on Android? We do that on iOS as the function can take some time to execute, and we want to avoid framerate glitches.

This is the callstack when it crashes. Have tried the beta you sent @Lanre, same problem.

at libc.abort(abort:63)
at libart.art::Runtime::Abort(char const*)(Runtime:392)
at libart.android::base::LogMessage::~LogMessage()(base:452)
at libart.art::JavaVMExt::JniAbort(char const*, char const*)(JavaVMExt:1212)
at libart.art::JavaVMExt::JniAbortV(char const*, char const*, std::__va_list)(JavaVMExt:58)
at libart.art::CheckJNI::CallNonvirtualCharMethodV(_JNIEnv*, _jobject*, _jclass*, _jmethodID*, std::__va_list)(CheckJNI:58)
at libart.art::CheckJNI::CallNonvirtualObjectMethod(_JNIEnv*, _jobject*, _jclass*, _jmethodID*, ...)(CheckJNI:10)
at libart.art::CheckJNI::GetObjectClass(_JNIEnv*, _jobject*)(CheckJNI:212)
at libart.art::CheckJNI::CallStaticLongMethodV(_JNIEnv*, _jclass*, _jmethodID*, std::__va_list)(CheckJNI:88)
at libart.unix_file::FdFile::GetLength() const(unix_file:64)
at libNatDevice.00001bcf(Native Method)
at libNatDevice.NDAudioDeviceStartRunning(NDAudioDeviceStartRunning:30)
at libil2cpp.00cc3a98(Native Method)
at libil2cpp.00cc8060(Native Method)

 

Forgot to mention, this is just using a simple Task, e.g.

 Task<bool> startRecord = new Task<bool>(() => { (device as AudioDevice).StartRunning(recorder.CommitSamples); return true; });

 

The Android JNI isn't very amenable to multithreading, because you need to be on a thread with a Java class loader somewhere in the call stack in order to find Java classes from native code. In Unity, only the main thread has the class loader. Your call stack is a bit curious because it mentions GetObjectClass instead of FindClass. The former should be fine whereas the latter is what should break, either when you create a media query or start a device on a background thread. TL;DR: Don't start an audio device in a worker task on Android. It's an annoyance on Android that prevents true thread safety.

Your recording code is a bit incorrect. If you are recording to an MP4 or HEVC recorder, you should be synchronizing the audio track's timestamps with those of the video track, by using a shared clock. The timestamp that the audio device reports is the elapsed time in nanoseconds from Jan 1, 1970; whereas the recorder is expecting a timestamp from zero.

 

Minor update on this: I'm working on a way to 'fix' this issue. I have it implemented for NatCorder. NatDevice is up next.

 

Hello Lanre!

We're using Natcam 2.3.1 for a while. We have upgraded to Unity 2020.1.6f1 and we are dealing with a weird error now on Android:
Stopping the preview, freezes the app on the test device (moto g plus).
There's no log entry, the last line is NatCam: Camera 0 stopped preview.

Do you know of this issue? How can we deal with this?

Thanks and greetings,
Enzi

 

I recommend upgrading to NatDevice. NatCam has been deprecated, and I recall an issue like this coming up but it has been fixed in NatDevice.

 

hi ,

In natDevice, how do I know that the frame has been updated?
Something like "didUpdateThisFrame" this in webcamtexture

 

This isn't exposed in the frontend API. Is there a reason you need this information?

 

Yes, I want to record every time the camera frame is updated.

 

Or to process the input image, how do I know the frame has changed?

 

NatDevice doesn't expose a callback or property for knowing when the frame has changed. The reason is that no matter when the preview data gets updated natively, the Unity main thread would have to 'pick it up' so to speak. So you could never get a frame update happening more often than your app's frame rate. Like say you had a 120FPS camera preview running on Android, and your app was running at 30FPS, NatDevice would effectively be dropping three consecutive frames because Unity is only refreshing fast enough to display the fourth.

Naturally, there's the question of what happens if Unity is updating more frequently than the camera preview, like if the camera is running at 30FPS but your app is running at 60FPS. In this case, you could just record at the app's frame rate, meaning that in the video stream, camera frames will get 'duplicated'. This isn't much of a concern as the H.264 or H.265 codecs are optimized to compress close-to-redundant frames.

You are likely processing the image in order to display or record it, in which case the earlier discussion holds. You could modify NatDevice to tell you when the frame is updated (see NativeCameraDevice.cs, specifically the StartRunning method, and how the device is created in MediaDeviceQuery). But for the front-end API we designed, I currently don't have plans to expose the actual callback. I'll look into if it's feasible or useful, but don't hold your breath.

EDIT: On looking into this further, I've decided to implement overloads of ICameraDevice.StartRunning that accept delegate functions. The delegates can either receive a `byte[]` that contains the pixel data in managed memory, or an `IntPtr` handle to the native pixel buffer that is written to by the camera (this would provide the best memory behaviour when recording with NatCorder, as you get to avoid copying memory into and out of the .NET CLR).

 

Hi @Lanre

I'm doing my monthly checkup about our flash issue. Any news about it? Is there hope for this or should we start searching for alternate solution?

Thanks.

 

We have the same issue. After the change, we get

NatDevice Error: Sample buffer delegate raised exception: System.ArgumentException: Offset and length were out of bounds for the array or count is greater than the number of elements from index to the end of the source collection.

Could you fix this properly and release a fixed version? Digging through this single long for thread for solutions feels pretty 2005 btw...

 

We have a large list of videos made by users,
We use both natcorder and natdevice.
Unfortunately, some movies are even made at 2 frames per second.
The configuration of these devices is also convenient.
We set the camera to 30 frames with the 264 codec and rendered 60 frames (due to duplicate rendering) doubled the video size.
I realized that didUpdateThisFrame is a basic method and you do not have it.

 

Some videos are lagging because they are not rendered at the right time, although their frame rate is good.

 

Hey there, sorry for the late response on this. It's almost done; I'll send you a beta build either tomorrow or Monday.

 

The MixerDevice included with NatDevice is only a reference implementation; it isn't listed on the Asset Store page or the online documentation. It isn't meant to be used in production; properly mixing audio requires a dedicated DSP, and implementing one is out of scope (and out of my skillset as I'm not an audio engineer).

 

As I mentioned I've added the ability to provide a callback that will be invoked directly from native code, much like how audio devices work.

 

Well. if it is not working properly, then please remove it altogether. If you supply something as an example, it shouldn't be broken, that's misleading your users. That said, it used to work fine, just seems to have issues with newer versions of iOS.

 

It was removed but enough folks had asked for the reference so I added it back. I'm with you, as I haven't wanted to include something that didn't always work. I'll be removing it in the next update.

 

Hello
In the analysis, we found that the natcorder image is only corrupted on the Samsung j7. This problem can not be solved for this device?

 

I likely can't solve this issue on my end. If the native camera API is providing squished data, there is no way for NatDevice to know how squished the data is in order to correct it.

 

Thanks for the reply

 

These answers seem contradictory? I've been trying to get my iOS app to play audio on a bluetooth speaker while recording with the built-in microphone. The speaker only plays the audio if I record using its microphone from the device query. As soon as I start the built-in iPhone microphone the audio output switches back to the built-in speaker.

1. Is it possible to play audio to a bluetooth output while recording on the built-in microphone or is this an iOS limitation? (Are speaker and microphone grouped together on a single audio route?)

2. Would you expect this to work with bluetooth speakers/headphones that don't have a microphone? I'd like to support this configuration...

3. I would prefer to use whatever audio route the system is currently using rather than having to prompt the user to select a microphone and hope that an appropriate output goes along with it. Is there a way to detect the current audio route and use the corresponding microphone, or could you add something that solves this?

I'm on NatDevice 1.0.2 and Unity 2019.4.12f1.
Thanks!

 

The keyword here is "shouldn't". iOS determines the audio routes; applications and frameworks like NatDevice can only request different settings. When you start recording from an audio device, NatDevice sets the preferred input port (so you get audio from the device you expect), but iOS determines the final audio configuration.

As far as I've seen in my own tests, I don't think this is possible. I don't know enough about how iOS's audio session routing rules work, so you best look at the iOS documentation.

Just use the default microphone (this should always be the very first audio device reported by the media device query.

 

Ah, but if the system is playing output to a bluetooth speaker and I start my app, it initially plays audio to the speaker but then switches to the built-in speaker once the built-in microphone is started. My preferred behaviour would be to use whatever microphone maintains the same audio output. But I don't know how to tell which this is, or even where the output is currently being routed to. If this isn't possible I'll just have to add the option for the user to select a microphone from a list, but as I said, I'd prefer not to do this.

Additional question: NatMic used to provide samples (firing the callback function) more frequently than NatDevice now seems to. I'm showing a VU-style meter of mic input and more importantly, I'm using the samples for real-time pitch detection so response time is super important. Is it possible to bump this refresh rate up please, via a setting perhaps? It's definitely making my game feel less snappy.

And finally, MediaDeviceQuery() freezes the app for a second or so which isn't great UX. I can't do it early (ie. on startup) because I don't want to trigger the permission prompt until it's needed. Unity tells me this must be in the main thread, is this correct? Any way around this?

 

When are you going to solve this issue?

 

I'm going to add a boolean getter on audio devices that indicates whether the device is the default audio device. This should resolve your use case.

I'll see if this is possible. NatDevice and NatMic work entirely differently when it comes to audio devices. I might be able to make NatDevice report samples more often, but I can't guarantee that it will work (it's up to the native API's which NatDevice interfaces with).

Nope, no way around this. What happens is that NatDevice has to ask iOS for a route change (so that audio comes from the device), and that process takes time. I've updated it so that it only happens when you call `StartRunning` on an audio device, instead of when you create a query. But there's no way around it.

 

This has been fixed in the upcoming update.

 

This is surprising. I thought NatDevice and NatMic both reported data as soon as the native APIs made them available.

This would be a harmful change for us. We prepare a number of systems - NatDevice, NatRecorder, audio playback - and start them all on the same frame. We need them synchronised. If StartRunning caused a CPU spike, the synchronisation would be off. The freeze would be ugly too - it's better if we can hide that when we fade between UI and gameplay.

 

Yes, they both do (which is why we expose the sample buffer delegate). But the native API that NatDevice uses is different from that used by NatMic.

There isn't a CPU spike; it's more like a global thread suspend. You can to this when switching scenes of course. Have the microphone running in the background, then only send sample buffers to the recorder when you are recording.

 

Oh great.
Thank you.

 

Hi,
I am still having a black camera surface issue. On Android on BlackView devices (we mainly test these) our App and the camera works great. However, sometimes, after the Android went to sleep mode, when it wakes up the camera surface is black and it doesn't come back.
I detect the black camera happens and I kill or restart the App but its a very nasty solution.
Also because the app is used in combination with a Kiosk, which really dislikes the Application.Quit(); in the app.
Is there a way to RESET the camera ??? If we don't know what really happens but I can just reset and restart the camera it would help !

Thanks.

 

Hello,

I am facing a weird performance issue with NatDevice when using the rear facing camera feed as the background.
I am rendering 5 models as in the screenshot on top of the camera picture. When rendering 4 models everything is fine but when I am rendering 5 models the camera feed starts to lag, it sometimes gets stuck for ~500 milliseonds before updating. The app itself is running smoothly at 30 FPS so the phone's computation power doesn't seem to be the issue.

I also tested with different slower and faster devices and the issue always arose when I enable the fifth model. It also makes no difference which of the models I disable, with 4 models everything's running fine.

Do you have an idea what the issue could be?

 

Can you share the full, unfiltered logs from logcat in a .txt attachment? Instead of killing the app you should be able to stop and restart the preview.

 

This is weird indeed. My best guess is that the GPU is being oversubscribed. How complex are the models? And can you reproduce this behaviour on different Android devices?

 

As this is rare, I did not have the opportunity to read the logcat.

You mean to :
var device = query.currentDevice as ICameraDevice;
device.StopRunning();
previewTexture = await device.StartRunning();

This actually doesn't work... how do I perfectly stop the preview and restart it ?