NatMic - Native Microphone API

You can't use both NatMic and Unity's Micorphone API at the same time. You can use a custom IAudioProcessor to get the audio data being streamed from the microphone (this would be the equivalent of calling GetData), then do your volume detection.

 

I recommend adding your voice to get Unity to upgrade to Xcode 11. We've been compiling with Xcode 11 for quite some time now.

 

Yes it is.

We don't share the native sources of our API's, except if they are open source (like NatShare).

 

will do, thanks

 

Is is support FFT now?

 

We don't plan on adding FFT support any longer.

 

I have the same issue on iOS. Do you have a solution for that @Lanre ? Currently it is NOT possible to play the recorded video with the standard unity video player...

Does it something have to do with the player settings 'Prepare iOS for recording' and 'Force iOS Speakers when recording' ?

 

I'm a bit confused. What recorded video are you playing? NatMic already sets the proper audio category when a microphone is started, so it is up to iOS to choose what category to actually use.

 

I've recorded a video with NatCorder in combination with NatMic. When you play that recorder video in the app, with for example Unity's video player, the sounds is really low, even if you set the volume of your iOS device to maximum. When you save that recorded video to your camera roll, and play it over there, the sound is normal.

From what I've been reading online, this has something to do with when your record the microphone, the normal speaker of the iOS will be disabled and only the call speaker is used. Correct me if I'm wrong. But that is the problem and after recording, it still only uses the call speaker instead of the normal speaker, which explains why the volume is really low.

 

NatMic already asks that iOS forward audio to the normal speakers instead of the call speakers. You probably also want to make sure that 'force sound to speakers' and 'prepare iOS for recording' are checked in player settings.

 

Thanks for the reply, I don't get it to work properly. Do you have a sample project where it should work? The combination of recording a video and playing it afterwards?

 

In the example code, we always use Handheld.PlayFullscreenMovie to playback a recorded video.

 

Yes, I’ve tried that as well and same result. I will make a clean project today with only you example, see what happens.

Can you confirm it works on your side on iOS? With 2019.2

 

I just tried with : https://github.com/natsuite/NatMicCorder-Demo
When playing the recorder video in the app with Handheld.PlayFullscreenMovie, the sound is really low volume (even when the volume is maximum on my device). When saving that video to the cameraroll (with NatShare), and play the video in the cameraroll the sound is normal.

Can you confirm it is a bug in your plugin? Or is it a bug in Unity?

Build with 2019.2.10f1, xCode 11.2.1 and had both 'force sound to speakers' and 'prepare iOS for recording' checked in player settings.

 

I have the same problem with Unity 2019.2.12f1

 

I sent you an updated iOS library that should keep the app volume the same when the mic is started. Let me know if you still face this issue.

 

I have the exact same issue about ios audio very low. Can you send me a fix lib too?

 

Hi,

on iOS when we set mic sample rate at 44100 we end up with a non power of 2 sampleBuffer length (470). We expect to have 512 (behaviour in editor). Any idea why this is happening? We do set the unity audio sample rate to 44100 in the audio settings of the project

 

I also can't use RealtimeClip because the compiler can't find AudioStream...where is this script coming from??

 

Lanre, you rock! It works, big thanks for the support. We can now publish our app .

 

Update coming to the Asset Store soon. Thanks Tim!

 

There is no requirement or constraint on the size of the sample buffer (and no relationship with that of Unity Editor). The microphone is free to generate sample buffers of any size. Do not expect a fixed buffer size in your code.

 

Here it is.

 

Hi i have NatCorder to record videos, and I just bought NatMic too, so i have both.

So i record a video and audio at the same time, it records ok, but the volume is really low when played back. Is there a way to increase the sensitivity when recording? The phone volume is on maximum, the recorded audio sounds clear, but i can hardly hear it.

I followed the steps from
https://medium.com/@olokobayusuf/natcorder-and-natmic-a-crash-course-221e58bc3525

Thanks

 

Yikes, just saw the change log for 1.3.3. Does that mean we can no longer record mic + in-game audio?

 

Are you on iOS? If so, the latest update on the Asset Store fixes the volume change.

 

I have the MixerDevice and VirtualDevice implementations on gist. The problem is that I can't guarantee proper synchronization due to coarseness in timestamps.

 

Ah gotcha, thank you! And thanks for including the thread link in the note on gist. What's the worst-case you've seen for audio syncing, are we talking a few ms or 1/2s+ offsets?

Edit (NatMic 1.3.1): dang, with some testing it seems audio sync is pretty unreliable, had one video work fine while another had a 1/4s delay. In-editor the game audio was sometimes stretched. Too too bad :/

 

Is there any way to get the "Volume change fix" with the Mixer and Virtual device still intact?

If i update to the newest version, and try to revert back the mixer and virtual device changes, i get an error when trying to build in xcode: "Undefined symbol: _NMCurrentTimestamp"

The issues with mixer and virtual device for our project were not fatal, so we really want to keep that and still get the volume change fix, since we want to launch on Android and iOS.

Thanks

 

Hello @Lanre
Thanks for your help in the Natcorder forum. I have some problems integrating Natcorder and Natmic. I was following your Github demo here:https://github.com/natsuite/NatMicCorder-Demo/blob/master/Assets/Scripts/NatMicCorder.cs

I have been able to record a texture. However, when I try to do it along with sound, the Unity Editor crashes. Even after installing the app on the phone, the app gets stuck at the same point. Both the version of the code is attached within. The app crashes when I try to end the recording. Here are the two snippets

Code (CSharp):

1. class XYZ: Monobehaviour {
2.  
3. Void Update()
4.   {
5.     if (isRecord)
6.         {
7.             Texture2D g = (Texture2D) some_texture.texture;
8.             pixelBuffer = g.GetPixels32();
9.             videoRecorder.CommitFrame(pixelBuffer, recordingClock.Timestamp);
10.         }
11.   }
12.  
13.  
14. private void StartRecording()
15.     {
16.         int recordingWidth = 2*640;
17.         int recordingHeight = 480;
18.         recordingClock = new RealtimeClock();
19.         videoRecorder = new MP4Recorder(recordingWidth, recordingHeight, 30, 0, 0, OnVideo);
20.         isRecord = true;
21.     }
22.  
23. private void StopRecording()
24.     {
25.         videoRecorder.Dispose();
26.         isRecord = false;
27.         videoRecorder = null;
28.     }
29.  
30. private void OnVideo(string path)
31.     {
32.         PlayerPrefs.SetString("path_video", path);
33.     }
34.  
35. }

The above snippet works fine for recording a texture. However, the following does not work.

Code (CSharp):

1. class XYZ: Monobehaviour, IAudioProcessor {
2.  
3. Void Update()
4.   {
5.     if (isRecord)
6.         {
7.             Texture2D g = (Texture2D) some_texture.texture;
8.             pixelBuffer = g.GetPixels32();
9.             videoRecorder.CommitFrame(pixelBuffer, recordingClock.Timestamp);
10.         }
11.   }
12.  
13. public void OnSampleBuffer(float[] sampleBuffer, int sampleRate, int channelCount, long timestamp)
14.     {
15.         videoRecorder.CommitSamples(sampleBuffer, recordingClock.Timestamp);
16.     }
17.  
18. private void StartRecording()
19.     {
20.         int recordingWidth = 2*640;
21.         int recordingHeight = 480;
22.         recordingClock = new RealtimeClock();
23.         videoRecorder = new MP4Recorder(recordingWidth, recordingHeight, 30, 0, 0, OnVideo);
24.         isRecord = true;
25.    
26.     //Modified for Audio as per integration demo
27.         var sampleRate = 44100;
28.         var channelCount = 1;
29.         audioDevice = AudioDevice.GetDevices()[0];
30.         audioDevice.StartRecording(sampleRate, channelCount, this);
31.     }
32.  
33. private void StopRecording()
34.     {
35.         videoRecorder.Dispose();
36.         isRecord = false;
37.         videoRecorder = null;
38.    
39.     //Modified for Audio as per integration demo
40.         audioDevice.StopRecording();
41.     }
42.  
43. private void OnVideo(string path)
44.     {
45.         PlayerPrefs.SetString("path_video", path);
46.     }
47.  
48. }

An observation: If I comment out the videoRecorder.commitsamples line, there is no crash.
Can you reproduce this error? And what can be done to solve this? Any help would be appreciated.

 

Hi! Is possible with NatMic to manage various internal microphones on iPad ( 2 mics in older iPads, 5 in newer )?

 

Hi @Lanre ! Is it possible with NatMic record a native sound/music from the smartphone? I mean, I want to open native music player on the device, play music, then launch my app and start to record background music that played from native / custom app music player. And add to it microphone , for example. Is it possible?

 

I have the same exception on Android, Please fix

 

Another bug with MixerDevice:

Code (CSharp):

1.                     var delta = initialTimestampA - initialTimestampB; // +ve if B has more samples cos started earlier
2.                     var deltaSamples = (int)((delta / 1e+9) * sampleRate * channelCount);
3.                     var aheadBuffer = deltaSamples > 0 ? sampleBufferB : sampleBufferA;
4.                     aheadBuffer.RemoveRange(0, Math.Min(deltaSamples, aheadBuffer.Count));
5.  

It is easy for "deltaSamples" to be negative which leads to negative RemoveRange call.
I assume you forgot to call Math.Abs, I fixed it with this change:

Code (CSharp):

1. aheadBuffer.RemoveRange(0, Math.Min(Math.Abs(deltaSamples), aheadBuffer.Count));

 

I used the latest version of the plug-in, and found a new problem. When the microphone is always on, and then playing the video, there will be a random noise problem at intervals. I switched back to the unity microphone and did not have this problem. Could you investigate what went wrong? thanks.

 

I know this question has been asked a while back but.... will this NatMic help reduce the real-time microphone to real-time playback latency issue on the Android platform? I just wondered if your latest updates can help resolve this issue.

To give you a bit of background I am currently developing an app for the Oculus Quest where-by the user speaks into the virtual mic in the scene and their voice is then amplified/altered through the virtual speakers in the scene. Think stadium/auditorium audio reverb.

I have it working through the Unity mic to audioSource components with optimum settings but as you suggested the latency is still pretty bad but surprising no feedback loop which at least is good.

So hoping your Asset could solve this issue some way, if not, do you have any suggestions?

 

Yup. These are the issues we simply were not able to get around. Unity's audio engine isn't as open as say the native graphics API's, so we're externally constrained in what we can do.

 

I'm working on an updated MixerDevice implementation. I'll add it to a gist and let you know when I do.

 

You are creating an MP4Recorder that does not expect any audio to be committed. The sample rate and channel count arguments are zero.

 

You probably will not be able to use multiple microphones simultaneously, but GetDevices() will return as many microphones as the OS reports.

 

I don't believe so, as this will likely violate privacy (like cloning songs from a music subscription service).

 

AudioDevice.StartRecording returns a boolean informing you of whether the microphone actually started recording. StartRecording can fail if the microphone doesn't support the requested parameters (a lot of Android microphones don't support stereo). Hence you must check if AudioDevice.IsRecording before calling StopRecording.

 

NatMic has as little latency as is possible since there is no DSP sitting between the native microphone API and C#. The only issue you will face is that you have to buffer the samples in order to feed then into an AudioClip that Unity then plays through an AudioSource. Since AudioClip uses a pull architecture for getting more samples, you'll have to write a buffering layer that minimizes latency. NatMic doesn't have code that does this; it simply provides the raw PCM samples to a delegate that you (the client) provide.

 

Hi @Lanre

I'm using NatMic to record from microphone to a Wav File and would like to also show an audio visualizer in realtime. The visualizer takes an AudioSource as input. I'm using this on iOS and (eventually) Android.

Is there a way to pipe the sound from NatMic to the visualizer. I see your comment above about using AudioClip, but not sure whether that is the right approach in my case.

 

Having read back on this forum some more, I tried to track down AudioStream - link above is dead - and found it here - https://gist.github.com/olokobayusuf/19ea5ac14a3aa848dcf8dab78fa524bc

However, I'm not sure this is the latest code as it refers to method ReadNegative (which I'm guessing should really be ReadBackward) and also the class VirtualDevice, which I can't find anywhere.

For my application, it's a cosmetic feedback wave so a little latency would not be terrible.

 

→ Aha! VirtualDevice is at that link above, just further down.

I'm currently trying to merge RealtimeClip and WAVRecorder into a single class that does both - I'm assuming you can't attach multiple recorders to a single AudioDevice?

 

Hey @Lanre, I noticed that my app mutes audio sources (like Spotify) on startup even when I request it not to. I know that there is a Unity bug concerning this, but it's supposed to have been fixed in 2018.3 and I'm on 2018.4. Just checking to see if this is coming from NatMic.

Thank you!

 

With the approach described above, I got it working - sort of.

Periodically, it hangs on device (iOS) for me, always on this call:

Code (CSharp):

1. _audioDevice.StopRecording();

I've stripped out the NatMic implementation and used Unity's Microphone instead. Seems to have less latency on the AudioClip and it's not crashing. I'm saving to wav file using this:

https://gist.githubusercontent.com/...13ae09e075c79826395f138f93777eac222/SavWav.cs

 

@Lanre Is there a better way to contact you for support of the asset? My original post was 2 days ago.

 

Hello everyone! I'm looking to buy natmic, but i don't know much about coding. My game has a virtual microphone (A microphone inside the game) where the player speaks to it and hears back his voice. I need low latency, can I have an example of the code that I need to use? Also can it calculate the distance between the player and the virtual microphone so that the player's voice sound lower the further he is from the (virtual) microphone?