NatMic - Native Microphone API

@Lanre Is there any way to reduce ambient noise from the microphone?

 

Using an audio device that performs echo cancellation is your best bet.

 

I'm doing...

Code (CSharp):

1.  AudioDevice audioDevice = AudioDevice.Devices[0];

The EchoCancellation property on this object is read only in the latest version of NatMic. Is there alternative?

Also, any progress on the audio sync issue with MixerDevice?

 

Yes, the property is relative. Now, echo cancellation is expressed as an intrinsic property of any given AudioDevice. You simply have to find a device in AudioDevice.Devices that shows EchoCancellation = true.

Not yet. I've been sick for a week now and am only just recovering and getting back to work. Sorry for the late email responses.

 

Hello @Lanre , just checking in if any progress has been made on the audio sync issue with MixerDevice?

 

Regarding EchoCancellation on iOS and Android. I'm still not following. It seems like all I can do is check to see if EchoCancellation is true or false on a given audioDevice. How is it enabled?

 

I sent you an email.

If a device has echo cancellation, then any audio data that it provides will have echo cancellation pre-applied. It is not a setting you enable or disable; it is an intrinsic property of the audio device that you record from. A simple way to think about it is: "does this AudioDevice perform echo cancellation?"

 

I've also ran into the "GCHandle value belongs to a different domain" issue when calling audioDevice.StopRecording() in-editor on Windows. I am doing a manual check for audioDevice.IsRecording before calling StopRecording but it still errors.

 

Still working on this. Should get it completed today or tomorrow.

 

Hello! While recording the sound from the microphone in the video (using NatMic and NatCorder), you can see a clear slowdown in the sound received from the microphone, how can I fix this?
watch the attached video
https://drive.google.com/file/d/1rky_XZjBElAPNrTPcunprBKU7fWLnCrX/view?usp=sharing

 

Can you share your recording code?

 

The sample rate and channel count that you are using to create an MP4Recorder are different from those you use to start recording from an AudioDevice. These must match.

 

I'm trying to mix game audio and mic input for natcorder, and I get type / namespace errors when I try to declare VirtualDevice or MixerDevice as indicated in the docs for this purpose.

ah, ok : these classes require "using NatMic.DSP"

 

How’s the audio sync between your game audio and your microphone audio? It’s really out of sync for me. Would love to here if you had better luck and what your setup looks like.

 

I follow this guide
https://medium.com/@olokobayusuf/natcorder-and-natmic-a-crash-course-221e58bc3525

when I call audioDevice.StopRecording() throw exception

AndroidJavaException:
java.lang.NullPointerException: Attempt to invoke virtual method 'void java.lang.Thread.interrupt()' on a null object reference

 

We see inconsistent results across platforms and across runs. The cause of this issue is that we don't have precise audio timing from Unity's audio DSP. As a result of this limitation, we are removing the MixerDevice from NatMic in the next update.

 

Fix will be live on the Asset Store early next week.

 

OK

I will try again

Thank you!

 

Hello,
I'm using natmic and I have a song in my scene, I realized that the volume of my voice in natmic is played very low, not to hear because of the music of the game, how to increase the volume of the audio capture?

 

Hello,
We encounter some problems with the last update for recording both the microphone and the game audio. In the Editor the script is working, but I'm receiving "ArgumentException: GCHandle value belongs to a different domain" error when I stop recording with NatMic, with AudioDevice or MixerDevice. On Android recording the mic works, but it's crashing when I try to mix the audio from the mic and the game. Is this a bug or I'm doing something wrong? The previous versions worked flawless.

Here is the logcat from Android:

Code (JavaScript):

1. 05-10 11:40:43.163: A/com.asa.wear(13806): runtime.cc:574] "UnityMain" prio=5 tid=13 Native
2. 05-10 11:40:43.436: E/CRASH(13806):     #15  pc 0000739c   ( (wrapper managed-to-native) UnityEngine.AndroidJNI:CallStaticObjectMethod (intptr,intptr,UnityEngine.jvalue[]) {0xbc823778} + 0x54 (0xe388a348 0xe388a418) [0xe6babf00 - Unity Root Domain]+29596)
3. 05-10 11:40:43.436: E/CRASH(13806):     #16  il 0000003f  at (wrapper managed-to-native) UnityEngine.AndroidJNI.CallStaticObjectMethod (intptr,intptr,UnityEngine.jvalue[]) <0x0003f>
4. 05-10 11:40:43.436: E/CRASH(13806):     #17  il 000000c7  at UnityEngine.AndroidJNISafe.CallStaticObjectMethod (intptr,intptr,UnityEngine.jvalue[]) [0x00005] in /Users/builduser/buildslave/unity/build/Runtime/Export/Android/AndroidJNISafe.cs:208
5. 05-10 11:40:43.436: E/CRASH(13806):     #18  il 000002bf  at UnityEngine.AndroidReflection.GetMethodMember (intptr,string,string,bool) [0x00057] in /Users/builduser/buildslave/unity/build/Runtime/Export/Android/AndroidJavaImpl.cs:728
6. 05-10 11:40:43.436: E/CRASH(13806):     #19  il 00000117  at UnityEngine._AndroidJNIHelper.GetMethodID (intptr,string,string,bool) [0x0000c] in /Users/builduser/buildslave/unity/build/Runtime/Export/Android/AndroidJavaImpl.cs:1209
7. 05-10 11:40:43.436: E/CRASH(13806):     #20  il 000000af  at UnityEngine.AndroidJNIHelper.GetMethodID (intptr,string,string,bool) [0x00005] in /Users/builduser/buildslave/unity/build/Runtime/Export/Android/AndroidJNI.bindings.cs:65
8. 05-10 11:40:43.436: E/CRASH(13806):     #21  il 000000e3  at UnityEngine._AndroidJNIHelper.GetMethodID<long> (intptr,string,object[],bool) [0x0000a] in /Users/builduser/buildslave/unity/build/Runtime/Export/Android/AndroidJavaImpl.cs:1175
9. 05-10 11:40:43.437: E/CRASH(13806):     #22  il 000000af  at UnityEngine.AndroidJNIHelper.GetMethodID<long> (intptr,string,object[],bool) [0x00005] in /Users/builduser/buildslave/unity/build/Runtime/Export/Android/AndroidJNI.bindings.cs:159
10. 05-10 11:40:43.437: E/CRASH(13806):     #23  il 00000187  at UnityEngine.AndroidJavaObject._CallStatic<long> (string,object[]) [0x0001d] in /Users/builduser/buildslave/unity/build/Runtime/Export/Android/AndroidJavaImpl.cs:455
11. 05-10 11:40:43.437: E/CRASH(13806):     #24  il 000000ab  at UnityEngine.AndroidJavaObject.CallStatic<long> (string,object[]) [0x00004] in /Users/builduser/buildslave/unity/build/Runtime/Export/Android/AndroidJava.bindings.cs:135
12. 05-10 11:40:43.437: E/CRASH(13806):     #25  il 0000014b  at NatMic.Platforms.AudioDeviceAndroid.get_CurrentTimestamp () [0x00019] in F:\_UnityProjects\weAR\Assets\_Assets\NatMic\Plugins\Managed\Platforms\AudioDeviceAndroid.cs:32
13. 05-10 11:40:43.437: E/CRASH(13806):     #26  il 000000f7  at NatMic.AudioDevice.get_CurrentTimestamp () [0x00027] in F:\_UnityProjects\weAR\Assets\_Assets\NatMic\Plugins\Managed\AudioDevice.cs:45
14. 05-10 11:40:43.437: E/CRASH(13806):     #27  il 0000006b  at NatMic.DSP.MixerDevice.<MixerLoop>b__10_0 (NatMic.IAudioDevice) [0x00001] in F:\_UnityProjects\weAR\Assets\_Assets\NatMic\Plugins\Managed\DSP\MixerDevice.cs:85
15. 05-10 11:40:43.438: E/CRASH(13806):     #31  il 0000023f  at NatMic.DSP.MixerDevice.MixerLoop () [0x0002f] in F:\_UnityProjects\weAR\Assets\_Assets\NatMic\Plugins\Managed\DSP\MixerDevice.cs:84
16. 05-10 11:40:44.425: W/ActivityManager(3937):   Force finishing activity com.asa.wear/com.unity3d.player.UnityPlayerActivity

Here is a snippet of my code where I use a switch case for different scenarios:

Code (CSharp):

1.  switch (audioType)
2.         {
3.             case AudioType.noAudio:            
4.                 break;
5.             case AudioType.inGameAudio:
6.                 // use NatCorder AudioInput to record AudioListener
7.                 audioInput = new AudioInput(videoRecorder, recordingClock, inGameAudioListener);
8.                 break;
9.             case AudioType.microphoneAudio:
10.                 // use NatMic AudioDevice for recording device microphone
11.                 audioDevice = AudioDevice.Devices[0];// NatMic AudioDevice for recording device microphone
12.                 audioDevice.StartRecording(AudioSettings.outputSampleRate, (int)AudioSettings.speakerMode, this);// start recording
13.                 break;
14.             case AudioType.all:
15.                 // use NatMic MixerDevice to record both the AudioListener and the device microphone
16.                 audioDevice = AudioDevice.Devices[0]; // NatMic AudioDevice for recording device microphone
17.                 gameAudio = new VirtualDevice(inGameAudioListener); // NatMic VirtualDevice for recording AudioListener
18.                 audioMixer = new MixerDevice(gameAudio,audioDevice); //NatMic MixerDevice to mix audio
19.                 audioMixer.StartRecording(AudioSettings.outputSampleRate, (int)AudioSettings.speakerMode, this); // start recording
20.                 break;
21.             default:
22.                 break;
23.         }  

 

Hey guys, could anyone help me with this:
I have purchased the NatMic package from Unity store. I want to use the laptop microphone to stream the real-time environment sounds into Unity game, Audio Source -> AudioClip, **during the runtime** of the Unity game.
I have gone through the package and the tutorial site, but only see approaches that record the audio in a wav file on the drive. Could anyone help me how to record the live audio to AudioClip? Thanks a lot!

 

Hi, Lanre, I've got the same problem with armv7. I checked the file with lipo. It looks like libNatmic.a that comes with the unity package is just of architecture arm 64. Since we still have some user cases involving old iOS device, could you please pass me access to an armv7 version. That would be a great help for my current development and I would really appreciate that.

 

Check that you are recording from an AudioDevice that does not have EchoCancellation. Echo cancellation usually performs gain control. NatMic itself doesn't touch the audio data from the microphone, so we don't do anything to the volume.

 

We have fixed this in the next update, 1.3.1, but we aren't releasing the update yet because we've had to remove the VirtualDevice and MixerDevice classes. We had to remove the mixer device because we aren't able to get proper synchronization with game audio given Unity's audio API's. We are currently asking Unity about this.

 

You can use this script:

Code (CSharp):

1. var sampleRate = ...;
2. var channelCount = ...;
3. var audioDevice = AudioDevice.Devices[0];
4. // Create a realtime clip
5. var microphoneClip = new RealtimeClip(sampleRate, channelCount);
6. // Start recording from the microphone
7. // The mic will stream audio into the realtime clip
8. audioDevice.StartRecording(sampleRate, channelCount, microphoneClip);
9. // You can then use the realtime clip like any other AudioClip in Unity
10. audioSource.clip = microphoneClip;
11. audioSource.Play();
12.  

 

We have added back the armv7 slice in the 1.3.1 update, but we've removed the MixerDevice and VirtualDevice scripts because of audio synchronization issues we are facing with Unity (we are asking Unity about it right now). If you don't use either of these scripts, PM me with your invoice number and I'll share the 1.3.1 build with you.

 

anyone have a fix or work around for the GCHandle issue?

 

https://forum.unity.com/threads/natmic-native-microphone-api.431677/page-5#post-4570153

 

I ran into the same issue. The latest AssetStore release (v 1.3.0) seems to only have the arm64 slice in the library (or my cache is somehow corrupted). Could you send an armv7 supported version to me too?

 

I replied your PM.

 

before integrate natmic , single use natcorder record is smooth，after integrate natmic，when I play record button it will stuck 1 seconds everytime. how can made it smooth ?

 

This is a bug we've fixed in the upcoming NatMic update.

 

Hi Lanre, did you see the emails I sent over to you? I have a couple of issues, both on iOS:

1. AirPods do not work, audio comes out of the headphones but the microphone does not pick anything up, instead the phones microphone continues to be used
   
2. A series of record and then playback will often end up with NatMic locking (waiting for a mutex I believe)

In my email, I linked to a sample project which shows this all, I can link the project on here too if my email did not get to you (at least one of my emails was rejected, I believe because I included the project as an attachment, but I repeated the email with the project linked to instead, but I wonder if that could have ended up in your Spam folder)?

 

Why does NatCorder's ReplayCam case show audio distortion in Windows?I haven't changed any code.

 

Hello, I used NatMicCorder-Demo-master and there was a problem.

 

Hi @Lanre
I found a bug in NatMic, that appears when I often Start Record and after 5-10 sec Stop Record. It just can freeze the app for a.. 3 seconds to infinity. Also, when I Stop microphone at the application pause(after minimizing the application to the home screen), it can just crash the app. I am using the latest Unity and build the app on iOS devices.

 

Hi @Lanre when I was using NatMicCorder-Demo-master, I had a microphone delay problem.I hope you can help me. It means a lot to me.

 

I wonder if this is the same issue that I mentioned above:

Hopefully it is, in which case Lanre maybe about to fix two issues with one bug fix.

 

Hello

I am not see any new version on asset store.
So I still have this problem.

when I call audioDevice.StopRecording() throw exception

AndroidJavaException:
java.lang.NullPointerException: Attempt to invoke virtual method 'void java.lang.Thread.interrupt()' on a null object reference

 

Maybe. It seems, @boxofgod8521 also have issues with stopping of recording, but on Android now.
My last emails also were ignored by Lanre, or.. maybe author on vacation, I don't know. Still waiting for an update.

 

Hey Lanre -

Just a heads up - I tried 1.3.1 and found a bug with iOS 11 Beta. If there's audio playing from some app A (spotify, audible, etc.) and you start to record from the mic in app B with NatMic - app A audio slowly fades out while app B freezes indefinitely. If you go to app A and hit resume, then pause, and switch back to app B - it unfreezes.

Didn't know if this was a bug in NatMic or iOS11 - but figured it was worth sharing.

 

Hey Steve. Sorry for the late response. I've fixed the AirPods issue (I'll email you in a few hours). As for restarting recording locking, I haven't reproduced yet. Will keep trying.

 

This question belongs on the NatCorder forum thread. Unity's Microphone API, which is what the ReplayCam example uses, produces glitchy audio. That was one of the original reasons why we began developing NatMic.

 

What device are you running on? This sounds like an audio setting mismatch (some devices cannot produce stereo audio, so they silently fall back to mono instead).

 

This has been fixed in the upcoming update.

In NatMic 1.2, recording is automatically paused when the app gets suspended. This has been removed from the next version so you won't run into this issue.

Sorry for the late response! NatCorder has had me completely swamped.

 

I'll check this out.

 

Thanks for sharing this. I'll try to reproduce it.

 

This has been fixed in the upcoming update. The update should be live on Monday or Tuesday. We've also reduced the minimum API requirement back to 18.

 

Hi,
Do you have a time estimate for when you will push the update for the audioDevice.StopRecording() exception. I Have a project release date coming up and that is one of our blockers.

Thanks