NatMic - Native Microphone API

here the code I tried RealtimeClip but still could not hear.

var device = Device.Default;
var format = Format.Default;
RealtimeClip clip = new RealtimeClip(format);
NatMic.StartRecording(device, clip.Format, OnSampleBuffer);
audioSource.clip = clip;
audioSource.Play();

 

I meant that when I tried to manually inject more reasonable timestamps, the audio was still not coherent and played back about twice as fast (dropping every other sample). Which led me to suspect the audiomixer wasn't mixing stereo and mono samples properly. But this may be wrong and I just didn't get the fix right.

In any case I reverted back to the previous NatMc version and will be waiting for the next version to re-do the integration.

 

Your code is almost complete. When you create a RealtimeClip, you still have to commit samples from NatMic into it (it needs an audio source itself). So in your OnSampleBuffer function, call `clip.CommitSamples(sampleBuffer, timestamp)`.

 

So NatMic will always interleave audio if the native mic doesn't support providing audio at the requested channel count. This means that AudioMixer doesn't have to worry about mono or stereo audio; the audio coming to it will always be of the same channel count. I'll see if I can reproduce this and get back to you.

 

Hi, we've updated NatMic to 1.2.0 and we are experiencing some problems on iOS and Android. We are using NatMic with Natcorder to record video with mic audio and in game audio also, using the AudioListener. It worked flawlessly until the new update. We updated our code to the new changes and in Editor is working without issues, but on iOS it's not recording the audio and on Android it's crashing when we are trying to record, also the microphone permission is not requested on both devices, on Android the crash occurs only after manually granting microphone permission, else it's not recording like on iOS. We tried the VoiceRecorder example scene, modified to record the AudioListener and we are experiencing the same problems.

Here is the part of the code that we use for NatMic

Code (CSharp):

1. // this we used to start NatMic
2. NatMic.StartRecording(Device.Default, inGameAudioListener, OnSampleBuffer);
3.  
4.   void OnSampleBuffer(float[] sampleBuffer, long timestamp)
5.     {
6.         // Send sample buffers directly to NatCorder for recording
7.         if (NatCorder.IsRecording)
8.             NatCorder.CommitSamples(sampleBuffer, recordingClock.CurrentTimestamp);
9.     }

Edit:
This is the android logcat errors we found regarding NatMic

Code (CSharp):

1. 02-20 18:59:09.728: E/AndroidRuntime(31629): FATAL EXCEPTION: NatMic Microphone Thread
2. 02-20 18:59:09.728: E/AndroidRuntime(31629): Process: com.asa.selfpatterns, PID: 31629
3. 02-20 18:59:09.728: E/AndroidRuntime(31629): java.lang.ArrayIndexOutOfBoundsException: length=1920; index=1920
4. 02-20 18:59:09.728: E/AndroidRuntime(31629):     at com.yusufolokoba.natmic.NatMic.run(NatMic.java:91)
5. 02-20 18:59:09.728: E/AndroidRuntime(31629):     at java.lang.Thread.run(Thread.java:764)
6.  

Using Unity 2018.3.3f1
Tested on Android 8.0 and iOS 12.1.3

 

Hi,

We've been using NatMic along with NatCorder, and it was very easy to use and integrate (as with all your plugins!). We just have one slight problem which is that on iOS sometimes the microphone input will be very low, almost inaudible. I suspect this is due to the automatic gain control, and see that was mentioned here as something you would look into. Is that something that made it into a build? I don't see a way to control the gain in the API. If not, is it something you are planning to add? We have no real way to work around it so this is kind of a blocker for us.

thanks,
Chris Howe, 1st Playable Productions

 

Hello,

I'm integrating NatMic with my project instead of the Unity's Microphone API (which is horrid for this kind of thing). The API is nice and simple, but I found one serious issue - I can't record from multiple devices at the same time, which complicates the replacement.

Would it be possible to support this? If the StopRecording accepted the recording Device the same way Unity's Microphone API does it would be perfect.

 

A fix for this is on the way to the Asset Store. We've had a few reports of it in the past. And similarly, we've put in a fix for the iOS bug when mixing with game audio (due to timestamps being based on different clocks).

 

Hey Chris! You should be able to completely circumvent AGC/AEC by turning off `Device.EchoCancellation`. I think it's disabled by default on iOS actually (and enabled by default on Android). There isn't any way for us to modify the gain, so we can't expose a similar setting in the frontend API. Are you recording through the internal speakers, or external devices? It shouldn't just change behaviour sporadically.

 

I really really like this feature request. In some sense, we could make the entire API a bit more object-oriented. I'll look into implementing this next. Thanks!

 

Hi Lanre! It actually seems that the first time we record audio the volume is fine and then any time after that will be quiet. If we restart the app then again the first time it will be fine.

I noticed some lines in the log when I start recording for the second time regarding audio routing.

[B]AVAudioSessionPortImpl.mm:56:ValidateRequiredFields: Unknown selected data source for Port Speaker (type: Speaker)[/B]



[B]AVAudioSessionPortImpl.mm:56:ValidateRequiredFields: Unknown selected data source for Port Receiver (type: Receiver)[/B]



[B]AVAudioSessionPortImpl.mm:56:ValidateRequiredFields: Unknown selected data source for Port Receiver (type: Receiver)



AVAudioSessionPortImpl.mm:56:ValidateRequiredFields: Unknown selected data source for Port Speaker (type: Speaker)[/B]

I'm testing a fix right now that seems to fix the issue for us. Instead of using Device.Default each time we call NatMic.StartRecording we now store Device.Default the first time and make sure we are using the same device every time. Is it possible that Device.Default specifies a different device the second time around?

 

Caching Device.Default shouldn't have anything to do with this. When recording is started, iOS will reroute the audio to the call speaker (the thin strip speaker towards the top of the device that is used on a voice call). NatMic tries to override this by then re-re-routing audio back to the main speaker. I think this is somehow what is affecting you. Try enabling the 'Prepare iOS for recording' flag in Player Settings and see if that helps.

 

Hello @Lanre. I'm using NatCorder along with NatMic for an iPhone app and it is working perfect expect for when I stop both NatCorder and NatMic and then I want to start the recording again, no audio will be recorded. I DOES record audio the first time, though. This only happens on iPhone. In the Unity editor it records audio always. Any idea on why would this be happening?

 

Hello @Lanre I'm trying to use realtimeclip with multichannel input but the plugin NatMic doesn't have the class AudioFormat so I can just assign a number of channels on the recorder.

When I connect my headphones (with microphones) it still having the output on the phone (and probably the input too), I set up all the configurations at the readme and made this changes on the VoiceRecoder.cs based on some answers in this forum.

I'm using an iPhone X, iOS 12. Unity 2018.3.7f1

public class VoiceRecorder : MonoBehaviour {

      

        public AudioSource audioSource;

        private IRecorder recorder;

        RealtimeClip clip;



        public void ToggleRecording (Text buttonText) { // Invoked by UI

            if (!NatMic.IsRecording) {

                var format = Format.Default;

                // Create a WAV recorder to record the audio to a file

                recorder = new WAVRecorder(format, OnAudioFile);

                recorder.StartRecording();

                // Start recording

                var device = Device.Default;

                clip = new RealtimeClip(format);

                NatMic.StartRecording(device, format, OnSampleBuffer);

                audioSource.clip = clip;

                audioSource.Play();

                buttonText.text = @"Stop Recording "+device;

                Debug.Log("Starting audio input device: "+device);

            } else {

                // Stop recording the WAV file and dispose the recorder

                recorder.Dispose();

                // Stop recording

                NatMic.StopRecording();

                buttonText.text = @"Start Recording";

            }

        }



        private void OnSampleBuffer (float[] sampleBuffer, long timestamp) {

            // Commit the sample buffer to the WAV recorder

            clip.CommitSamples(sampleBuffer, timestamp);

            recorder.CommitSamples(sampleBuffer, timestamp);

        }



        private void OnAudioFile (string path) {

            // Log the path

            Debug.Log("Saved recording to: "+path);

            // Play the recording in the scene

            Application.OpenURL(path);

        }

    }

 

I responded to your email. This issue should be implicitly fixed in the next update.

 

I'm a bit confused. What do you mean by trying to use RealtimeClip with multichannel input? The `Format` is what it seems you want.

So this sounds like a bug. We manually route speaker output to the speaker because when recording starts, iOS tries to route output to the call speakers (the tiny speakers for voice calls). I'll look into this for the next update.

 

I know there was some information that its not possible to change the gain on the mic.

On Android, disabling the `Device.EchoCancellation` helped.

Is it possible to add some gain via software, processing the signal when its still in the buffer?

Thanks for another great product, btw.
Leo

 

NatMic doesn't have any DSP functionality. You can add a gain multiplier yourself (simply multiply every sample in the sample buffer by a constant).

 

Thanks!

I did some tests and some android models are recording higher volumes than others.
I have some videos to show this. does it help?

 

There isn't anything we can do to normalize the volume across devices; it's simply an artifact of the hardware from different devices and vendors.

 

Is there a way to use NetMic in 2018.2.7f1?

 

NatMc is compatible with Unity versions as old as 2017.1 and even older. Are you having problems using it with 2018.2.7?

 

2018.2.7 running with NatMic VoiceRecorder scene, when recording finish, OS X MacbookAir complaint that there is no available Apps can play the recorded wave file, and when I having Xcode to build for iPadPro and iPhone 6, it show the the above error message. Please help.

MacBookAir OS X 10.14.3
IOS 12.1.4

 

Hm this is weird. Email me with your invoice number and I'll send you a build of the upcoming NatMic 1.3.0. As for the crash on iOS, you need to add microphone permissions to the app (in Player Settings) and then grant the permissions at runtime.

 

Thanks for the help. Please check your email.

 

@Lanre using NatCorder - Video Recording API and NatMic - Microphone API recording video is ok but cant record voice , this my test code:

Code (CSharp):

1. using NatCorderU.Core;
2. using NatCorderU.Core.Clocks;
3. using NatCorderU.Core.Recorders;
4. using System;
5. using System.Collections;
6. using System.Collections.Generic;
7. using UnityEngine;
8. using UnityEngine.Events;
9.  
10. public class Record : MonoBehaviour {
11.  
12.     [Serializable]
13.     public class VideoRecordedEvent : UnityEvent<string>
14.     {
15.     }
16.  
17.     [Serializable]
18.     public enum SelectableAudioFormat
19.     {
20.         None,
21.         Unity,
22.     }
23.  
24.     [Header("Recording"), SerializeField]
25.     private Container container = Container.MP4;
26.     [SerializeField]
27.     private Camera recordingCamera;
28.  
29.     [Header("Audio Format"), SerializeField]
30.     private SelectableAudioFormat audioFormat = SelectableAudioFormat.Unity;
31.  
32.     [Header("videoRecorded"), SerializeField]
33.     private VideoRecordedEvent videoRecorded = new VideoRecordedEvent();
34.  
35.     private CameraRecorder videoRecorder;
36.     private AudioRecorder audioRecorder;
37.     private IClock recordingClock;
38.  
39.     public void StartRecording()
40.     {
41.      
42.         var width = Screen.width;
43.         var height = Screen.height;
44.         var framerate = container == Container.GIF ? 10 : 30;
45.         var videoFormat = new VideoFormat(width, (int)height, framerate);
46.         // Create a recording clock for generating timestamps
47.         recordingClock = new RealtimeClock();
48.         // Start recording
49.         var aFormat = audioFormat == SelectableAudioFormat.None ? AudioFormat.None : AudioFormat.Unity;
50.         NatCorder.StartRecording(container, videoFormat, aFormat, OnReplay);
51.  
52.  
53.         if (recordingCamera == null)
54.         {
55.             recordingCamera = Camera.main;
56.         }
57.         videoRecorder = CameraRecorder.Create(recordingCamera, recordingClock);
58.  
59.         //Commenting out this line here will make the video stutter really badly after 2 seconds if you've set it to use AudioFormat.Unity
60.         audioRecorder = AudioRecorder.Create(FindObjectOfType<AudioListener>(), recordingClock);
61.         // If recording GIF, skip a few frames to give a real GIF look
62.         if (container == Container.GIF)
63.             videoRecorder.recordEveryNthFrame = 5;
64.     }
65.  
66.     public void StopRecording()
67.     {
68.         // Stop the recording
69.         videoRecorder.Dispose();
70.  
71.         if (audioRecorder != null)
72.         {
73.             audioRecorder.Dispose();
74.         }
75.         NatCorder.StopRecording();
76.  
77.  
78.     }
79.  
80.     void OnReplay(string path)
81.     {
82.         Debug.Log("Saved recording to: " + path);
83.         videoRecorded.Invoke(path);
84.         Application.OpenURL(path);
85.     }
86. }
87.  

 

Is it possible to record only the game audio, without the microphone or at least adjust the mic volume to be muted?

 

Hi Lanre,

The microphone permissions checked, the error still occur, any update?

 

Your code is recording audio from an AudioListener. There is no microphone (NatMic or other) code. Also, upgrade your NatCorder build to 1.5 on the Asset Store.

 

This gets out of scope of NatMic, whose purpose is for recording the microphone audio.

I'm not sure what this means. The mic is audio input, not output.

 

2 questions
1. Can NatMic be used for realtime mic playback on OSX?
2. If so, is there a working sample script for this?

 

Why do you want to playback the microphone in realtime? You will be creating a feedback loop.

 

if i had headphones on why would there be a feedback loop? i need realtime low latency mic for use in game. i want to use it to make characters speak

 

Yes, with headphones there will be no feedback loop.

We removed the recorder that can double as a realtime AudioClip because it was practically impossible to guarantee low latency for playback. The reason for this is that Unity's AudioClip will request for more audio samples on its own time and uses a pull architecture. As a result, you have to buffer up audio samples until Unity requested them. This process introduces latency, as some time will inadvertently pass between when the microphone reports some piece of audio and when Unity asks for that piece of audio to playback in the speakers.

If you're curious, here's the RealtimeClip recorder. Simply create it, pass it to IAudioDevice.StartRecording, then use it as any other AudioClip. You will likely experience the latency I describe.

 

Hey Lanre,
I coincidentally nabbed NatMic on the day of the big API change. Is there an example I can look at for getting NatCorder and NatMic playing nicely together?

Thanks!
-Chris

 

Hi there. Check out this article. The NatMicCorder example hasn't been updated for NatMic 1.3.

 

I'm very close to getting this to work in my project. The first time i record it works great. (Unbelievable sound quality!)

If I try to record a second time within the same testing session, I get an error:

Code (CSharp):

1. NatMic Error: AudioDevice processor raised exception: System.NullReferenceException: Object reference not set to an instance of an object
2.  

It happens on this call of "CommitSamples":

Code (CSharp):

1.     // Invoked by NatMic audio device with new audio sample buffer
2.     public void OnSampleBuffer(float[] sampleBuffer, int sampleRate, int channelCount, long timestamp)
3.     {
4.         // Send sample buffers directly to the video recorder for recording
5.         videoRecorder.CommitSamples(sampleBuffer, recordingClock.Timestamp);
6.     }

Having debugged things like this before, I feel like I may need to purge some more data between recordings...but I have no idea how to go about that.

 

When starting recording, you need to first create the NatCorder recorder before starting the microphone. I have updated the NatMicCorder Demo with this fix.

 

Hey Lanre -
Ran into (what I assume is) a bug while running this in-editor on windows. Everytime I call audioDevice.StopRecording() I get an error in AudioDeviceiOS.OnSampleBuffer:
var deviceRef = (GCHandle)context;
>GCHandle value belongs to a different domain
I wrapped that in a try catch(and just return inside the catch) and then AudioDeviceiOS.StopRecording() throws at
self.Free();
with an error similar to "handle is not initialized". I wrapped that line in an if(self.isAllocated).

After doing that - everything runs great in-editor. Video files are being saved correctly with audio etc. etc. I won't begin to posit what the actual cause is, but figured it was worth pointing out.

Appreciate the plugin. It's saved me tons of time.

-Chris

 

Thanks for reporting this. The error can happen if you call StopRecording more than once (the device itself won't check if IsRecording is true, you are expected to do this).

 

I updated to the latest NatMic and NatCorder today. It works as expected on iOS after updating to the new API, but on android I get some errors and the recording does not have any Audio. Below is the error output. I'm using the example code in NatMicCorder. Do you know what the issue may be?

One change I noticed is that before I was able to call Device.Default, Format.Default, etc. but the example hard codes it to 44100 sample rate and 2 channels. I'm not sure if this could cause the issue.

2019-03-26 17:35:00.222 11794-11817/io.lightup.studioalphadev V/Unity: NatMic: Inspecting device 8 with type 18

2019-03-26 17:35:00.223 11794-11817/io.lightup.studioalphadev V/Unity: NatMic: Channel counts=[1, 2]

2019-03-26 17:35:00.223 11794-11817/io.lightup.studioalphadev V/Unity: NatMic: Sample rates=[8000, 11025, 12000, 16000, 22050, 24000, 32000, 44100, 48000]

2019-03-26 17:35:00.223 11794-11817/io.lightup.studioalphadev V/Unity: NatMic: Channel masks=[12, 16, 48, 16384, 32768]

2019-03-26 17:35:00.271 11794-12359/io.lightup.studioalphadev W/AudioRecord: dead IAudioRecord, creating a new one from start()

2019-03-26 17:35:00.286 11794-12359/io.lightup.studioalphadev E/AudioRecord: Could not get audio input for session 44705, record source 0, sample rate 44100, format 0x1, channel mask 0xc, flags 0

2019-03-26 17:35:00.286 11794-12359/io.lightup.studioalphadev W/AudioRecord: restoreRecord_l() failed status -22

2019-03-26 17:35:00.286 11794-12359/io.lightup.studioalphadev E/AudioRecord: start() status -22

2019-03-26 17:35:00.286 11794-12359/io.lightup.studioalphadev V/Unity: NatMic: Microphone 8 started recording with format: 2@44100Hz

2019-03-26 17:35:00.286 11794-12359/io.lightup.studioalphadev W/AudioRecord: dead IAudioRecord, creating a new one from obtainBuffer()

2019-03-26 17:35:00.298 11794-12359/io.lightup.studioalphadev E/AudioRecord: Could not get audio input for session 44705, record source 0, sample rate 44100, format 0x1, channel mask 0xc, flags 0

2019-03-26 17:35:00.298 11794-12359/io.lightup.studioalphadev W/AudioRecord: restoreRecord_l() failed status -22

2019-03-26 17:35:00.298 11794-12359/io.lightup.studioalphadev E/AudioRecord-JNI: Error -22 during AudioRecord native read

2019-03-26 17:35:00.298 11794-12359/io.lightup.studioalphadev E/Unity: NatMic Error: Failed to read microphone 8 samples with error: -2

2019-03-26 17:35:00.299 11794-12359/io.lightup.studioalphadev V/Unity: NatMic: Microphone 8 stopped recording

 

It's weird because the logs indicate that the device supports stereo audio, but it fails. Try requesting mono audio instead (channelCount = 1).

 

Thanks for the quick response. That works!

 

Hi

We’d like to record input from the AirPods microphone (while also routing game audio output to them). Is it possible or planned?

Thanks

 

If airpods are connected, then there will be an AudioDevice for them. You just have to find this AudioDevice (by enumerating AudioDevice.Devices) and start recording from it.

 

Hi, is armv7 missing from the library?

ld: warning: ignoring file /censored/Libraries/Plugins/NatMic/Plugins/iOS/libNatMic.a, file was built for archive which is not the architecture being linked (armv7): /censored/Libraries/Plugins/NatMic/Plugins/iOS/libNatMic.a



Undefined symbols for architecture armv7:

  "_NMStopRecording", referenced from:

      _AudioDeviceBridge_StopRecording_m05AA2D62A03940E80C60CB7C3D576E9D545DA59F in Bulk_Assembly-CSharp-firstpass_43.o

      _AudioDeviceiOS_StopRecording_m8D2D5045E42EEFE0601CAAA4DD4D9CBCFB982845 in Bulk_Assembly-CSharp-firstpass_43.o

  "_NMIsRecording", referenced from:

      _AudioDeviceBridge_IsRecording_mBF4987063DD9678B68E263E02A8C4653B75EAD82 in Bulk_Assembly-



      _AudioDeviceiOS_get_CurrentTimestamp_m9FD809141C5FEC8E430F36A7C61764D7B459BC07 in Bulk_Assembly-CSharp-firstpass_43.o



      _AudioDeviceBridge_CurrentTimestamp_m9CFC574F8C1449A1A0CB2ABCE43BCE045D1B21B8 in Bulk_Assembly-CSharp-firstpass_43.o

---- removed part -----

ld: symbol(s) not found for architecture armv7



clang: error: linker command failed with exit code 1 (use -v to see invocation)

I check using lipo -info command which states following: libNatMic.a is architecture: arm64
Will there be armv7 support?

 

That's weird, because my libNatMic.a has both armv7 and arm64 slices. I've emailed you the library. Also, you should note that Apple has dropped support for armv7 (32-bit), so you really shouldn't ever need to archive a 32-bit slice.

 

Thanks for sorting out the issue very quickly.

 

Hey Lanre,

I'm seeing a massive lag spike on the call:

Code (CSharp):

1.  
2. audioDevice = AudioDevice.Devices[0];
3. audioDevice.StartRecording(sampleRate, channelCount, this);

This is using the sample code provided by the NatMicCorder example on iOS.

The profiler shows upwards of 350ms. The first call seems to lag more although the profiler doesn't show it.

 

I've had another report of this issue. I'm looking into it.