NatMic - Native Microphone API

Get NatMic on the Asset Store.

Hi all! I'm very happy to introduce an API that the Unity ecosystem has needed for years. If you've ever tried to record audio with Unity's Microphone class, you have likely run into unfixable bugs, like stuttering and unbearable latency. Now, we have created an API from scratch so that you don't have to face these issues any longer.

NatMic is a lightweight cross-platform microphone API for Unity. NatMic has support for iOS, Android, macOS, and Windows.

NatMic Features include:
- Realtime low-latency recording, with microphone data being provided in a dedicated thread.
- Configurable microphone sample rate and channel count.
- Recorders. NatMic comes with a WAV recorder and AudioClip recorder.
- Cross platform support for iOS, Android, macOS, Windows.

__

Check Out a More Detailed Introduction Here.

 

Yes! It will make NatCam complete. You can separate it into a new product. I bet there are many people that are willing to pay for this. If you google "Unity Read Mic data", you will see many posts with no solution. Unity's Mic API is extremely slow.

I do not own NatCam yet but the videos you made about them are awesome. What I need is a fast camera + audio API so that that I can read them and send them over network.

What this can be used for:

Online Game Voice Chat
WebCam Apps
Video(Front Camera) and Audio live calls
Waveform Visualization
Record Audio from the System Sound(Not just sound playing from Unity).
Record phone call

I see that the Asset store is full of Voice Chat plugins but those are crappy. Only looking for low access API to read from the Mic hardware.

NatMic API Structure:

• API to read Mic data as byte array.
• API to play the read data byte array.
• API that reads the Mic and returns the data in a way that be used to construct a waveform.

The idea is to allocate memory once and re-use them. Usually the read function does not return an array but fills an array that is passed in then returns how many bytes there were filled in. This prevents memory allocation.

What I plan to do:

Read audio bytes with NatMic, send to network with TCP/UDP. Server. On my TCP/UDP client, read bytes from server then use NatMic to play received data. I will use other network protocol on WebGL. Hopefully that makes sense. It would be good if NatMic can be used in another Thread as this will improve performance over-all.

I plan to buy NatCam but since there is no audio plugin yet, I am worried that trying to implement my own audio plugin will conflict with NatCam on Android since I need to extend UnityPlayer. I will wait to see if NatMic will become a thing. Hopefully, it will.

If you have question, please don't hesitate to ask.

 

We would make it separate from NatCam, but completely compatible.

Interesting. I will look into this.

Only the microphone input will be supported. Unity's sounds will not at all be recorded.

This will be done using Unity's Audio API's.

We'll take care of optimizations. No worries here.

Playback will be done with Unity API's.

NatCam does not extend UnityPlayer, so you are completely safe here.

 

What we have in mind for the API is this:
- Record audio data for file saving (record to waveform directly from native layer)
- Record audio data for streaming (audio data will be sent to Unity and then propagated through a callback)
- A Waveform class handy for visual animations

Abstractions will be made for the purposes above to make the process much easier for developers.

 

That's nice.

Yes, That sums it up. Recording for file saving is also useful. It will be good if you keep me updated in this.

 

Sure will. I hope other devs also express interest in this. So far there are 3 developers on board (including you and me).

 

One of the projects I'm working on is an online multiplayer VR game for PC that will need voice chat over Unity native multiplayer networking (UNet). If this NatMic would have a plugin that would that would easily integrate with UNet and instantly give me voice chat, then hell yes I would be all over this. I have no clue how to integrate voice chat for a PC VR game.

For my game I would be getting to the networking part in probably two months.

 

I'll be taking a look at the UNet API; I haven't had to use it before.

I am still doing a feasibility study for this plugin. I'm eager to see what we conclude.

 

I really don't care about UNet. Only care about reading and writing Mic data as fast as possible. Since UNet support has been requested, it looks like it will help many others who don't know how to use TCP/UDP with another Thread.

One thing to know is that if you decide to support Unet is that there is LLAPI and HLAPI, use the Low level (transport layer) API. LLAPI deals with byte arrays and that makes it very fast. Although nothing beats direct TCP/UDP protocol in terms of performance so I won't be using LLAPI for this unless for WebGL.

 

Going off what the majority of developers need, not you alone.

Thanks for the tip!

 

Just chiming in again to say that if you did include voice chat support, you would be one of the only working voice chat plugins on the unity asset store which would make your plugin pretty popular.

I'm looking now for a plugin and all the ones I find, the reviews either say doesn't work in unity 5, not UNet compatible, or reviews saying it doesn't work period.

 

Not necessarily. Exclusivity is not equal to prominence. We are still waiting and watching to see if there is a large-enough volume of developers that would want such a plugin.

 

Any update on if you are doing this and what it might entail?

Thanks.

 

Due to the amount of work we have to do, we have decided not to develop the project at this time. This might change, but not anytime soon.

 

hello,
any progress on this?
looking for mic input waveform visualisation...
thx
M.

 

We decided not to move forward with the project.

 

Hi all. It's almost a year after my last post. We've restarted the NatMic project since it has become a huge roadblock in our progress with some of our projects. We needed a reliable microphone API, and since there are no good options, we have decided to create one ourselves.

I will edit my first post at the top of this thread, and publish an article describing what we have planned. But to keep you without the suspense, NatMic 1.0 will natively support iOS, Android, Windows, and macOS (all have already been implemented). In the next release, we will add support for WebGL and possibly WinRT. NatMic 1.0 will also support mixing microphone audio with game audio (from an AudioSource or AudioListener). The API will be really simple, consisting of one callback that is provided with PCM audio data. Finally, NatMic will come with `Recorders`, which when given this audio data, will create an audio file. NatMic 1.0 will come with one recorder for recording waveform (WAV) files. In subsequent releases, we'll be looking at adding recorders for M4A, OGG, and MP3.

Sorry it's taken over a year to get this done!

 

Hi @Lanre , do you have an ETA on when NatMic will be coming out?

 

Not yet. We are working on getting the iOS implementation finalized. Once we do this, we'll be offering free beta access for a few weeks before releasing it.

 

NatMic is now in open beta.

If you would like to test the API, say so here or PM me your email address so I can share the build with you.

 

AWESOME !!! Do you think it would work with unity 5.6.1?

 

Dude! This is amazing. 2 months ago I was pulling my hair out at the Unity implementation and today I decided to get back to it for mic over the local network. I actually found this post through google since I was searching for new strategies and was thoroughly surprised to see this came out yesterday! I would gladly beta test and debug for you as this would also help my implementation. I am excited and hopeful you will send me the beta!

Thanks!!!

 

I was running into issues with recording audio with NatCorder and came across your post about this on Medium. I would love access to this beta!

 

I too would love access to this (I've sent a PM). I've just released an app and the microphone-speaker workaround breaks bluetooth (NatCoder & NatShare work perfectly).

 

Absolutely!

 

PM me your email address and I'll share the beta build with you on Dropbox.

 

Integration with NatCorder is very very high on our list (it's actually the reason why we decided to restart this project). PM me your email address and I'll share the build with you on Dropbox.

 

Out of curiosity, I have been trying to find a extreme low latency stream of microphone samples for real time audio visualization. Do you think NatMic is able to do that, and would it be up for some testing with the new ECS system? I noticed you were on Open beta right now so if you'd like I have already pmed you my email

 

NatMic handles low-latency microphone input. The user-provided callback gets invoked immediately a sample buffer is available (directly on the microphone thread). NatMic currently doesn't provide any visualization or spectral analysis functionality.

 

Hi!! Ive made a test with natmic on android and didnt had any problem! Will try also with ios when do you think this will be on the asset store? and integrated with natcorder?

 

We are currently working on NatCorder integration tests. Once this is done, we'll start the Asset Store submission process.

 

Hi @Lanre ! Thats great!!
Im currently trying to make it work on iOS but its not working! Is it working for ios devices?

Im using an iPhone 6 with ios 11.4.1

Ive used unity 5.6.1p1 and unity 2017.3.1p1, an empty project where i imported your plugin and opened your example scene, i also created the permission to record on the player settings. It compiles all ok, and on the iphone it does record the wav file, but doesnt have any sound.

In Xcode, with the iphone connected to it, i get this error while recording dont know if its related

2018-09-08 00:48:44.079925+0200 TESTAUDIOREC2017[1637:467765] 55: EXCEPTION (-1): ""

sample rate: 24000 sample count: 1024

NatMicU.Examples.VoiceRecorder:OnSampleBuffer(AudioEvent, Single[], Int64, Format)



(Filename: /Users/builduser/buildslave/unity/build/artifacts/generated/common/runtime/DebugBindings.gen.cpp Line: 51)






What can be happening?

Thanks

 

The exception log in Xcode is innocuous; we are looking for ways to remove it. You can ignore it. Regarding the recording being completely silent, I got a similar report on macOS which I am investigating right now. When we tested initially, it worked, so it's strange. I'll update you when we resolve it.

 

Thanks it would be awesome if you could make it working on ios!!! I tried on my imac and it did work ok! If you want i can give you more info on the imac versions os etc

 

I'll let you know. Thank you!

 

Great!! I'll be waiting for any advancements on the issue
Thanks

 

NatMic Documentation is Live.

We are submitting the API to Asset Store review tomorrow. I'll keep you posted!

 

Nice Did you manage to reproduce the error on the iphone (no sound on the wav file)?

 

I am able to reproduce the error but as I mentioned before, it is innocuous as far as we can tell. We are able to hear the audio. The silence we got on macOS was actually for a different reason (we were working on integrating NatCorder and NatMic). Can you email me a recorded WAV and the full logs from Xcode? Thanks.

 

Hey,

Ok so I got NatMic running fine in the editor but as soon as I build to a device, the microphone is really distorted, the apps audio volumes get turned right down and this error appears repeatedly in the console of Xcode:



55: EXCEPTION (-1): ""


Any ideas on what the issue could be?

 

Are you running the VoiceRecorder example? Can you share your code? The exception log in Xcode is apparently a bug in iOS 11; as far as we can tell, it is innocuous.

 

No, but I used this code provided to me earlier. The audio distortion happens even in a blank scene with just a couple of UI elements:

Code (CSharp):

1. public class NatMicCorder : MonoBehaviour {
2.  
3.     #region --Op vars--
4.     public Camera recordingCamera;
5.     private CameraRecorder cameraRecorder;
6.     #endregion
7.  
8.  
9.     #region --Recording--
10.  
11.     public void StartRecording () {
12.         // Start the microphone with NatMic
13.         var microphoneFormat = Format.Default;
14.         NatMic.StartRecording(microphoneFormat, OnSampleBuffer);
15.         // Start recording with NatCorder
16.         var audioFormat = new AudioFormat(microphoneFormat.sampleRate, microphoneFormat.channelCount);
17.         NatCorder.StartRecording(Container.MP4, VideoFormat.Screen, audioFormat, OnRecording);
18.         // Create a camera recorder for the main cam
19.         cameraRecorder = CameraRecorder.Create(recordingCamera);
20.     }
21.  
22.     public void StopRecording () {
23.         // Stop the microphone
24.         NatMic.StopRecording();
25.         // Stop recording
26.         cameraRecorder.Dispose();
27.         NatCorder.StopRecording();
28.     }
29.     #endregion
30.  
31.  
32.     #region --Callbacks--
33.  
34.     // Invoked by NatMic on new microphone events
35.     private void OnSampleBuffer (AudioEvent audioEvent, float[] sampleBuffer, long timestamp, Format format) {
36.         // Send sample buffers directly to NatCorder for recording
37.         if (audioEvent == AudioEvent.OnSampleBuffer)
38.             NatCorder.CommitSamples(sampleBuffer, Frame.CurrentTimestamp); // Use NatCorder's timestamps
39.     }
40.  
41.     // Invoked by NatCorder once recording is finished
42.     private void OnRecording (string path) {
43.         // Play the recorded video
44.         #if UNITY_EDITOR
45.         UnityEditor.EditorUtility.OpenWithDefaultApp(path);
46.         #else
47.         Handheld.PlayFullScreenMovie(path);
48.         #endif
49.     }
50.     #endregion
51. }

 

There's a minor change that needs to be made. In `OnSampleBuffer`, you need to check that NatCorder is recording before calling CommitSamples. Check out this full project demonstrating NatMic and NatCorder integration.

 

Ok so I tried that project, and all the microphone picks up is crazy distortion. This happens in iOS 12 and 12.1(beta). On an iPhone X, iPhone XS Max, an iPhone 7 and an iPhone 5S. All exactly the same distortion

Unity version 2018.2.9f1

I recorded it so you can see:

 

Replace libNatMic.a in NatMic > Plugins > iOS with the one I've attached and let me know if there's still stuttering.

 

Awesome that seems to have fixed things. I'll do some more integration tests after dinner. You're a legend Lanre

 

I'd like to try this on my iPhone X, and some Android platforms. I wrote a spectrum analyzer I want to try with some clean mic input on these platforms.

Also, is there any ability to combine this audio data with an mp4 video in this, NatCam or NatCorder? (I think you do this when recording with NatCorder but I'd also like to be able to edit this to add/remove/adjust audio or video and then merge them back into one mp4 on iOS or Android)

 

Hi!
I'm really interested in NATmic. We have an option to record a video in our ARapp (iOS & Android). Since integrating NATrecorder we are getting super smooth video, but unfortunately a choppy mic recording, maybe NATmic will help?

 

We aren't sharing the beta anymore; we are working with Unity to get it released onto the Asset Store (they requested some things from us that we're taking care of)

I read two possible questions from this. If you're asking if there's a way to blend game audio with the microphone audio, then yes; NatMic's StartRecording function offers overloads that take in an AudioSource or an AudioListener, and NatMic will provide you with sample buffers with the mixed audio.

If you're asking whether NatMic offers 'editing' capabilities (any sort of processing after recording has happened), then the answer is no. NatCorder doesn't do any post-recording editing either.

 

Absolutely! NatMic will definitely help (NatCorder was the reason why we decided to resume this project). We are working with Unity to get it published on the Asset Store.