Search Unity

  1. Unity 2019.2 is now released.
    Dismiss Notice

NatMic - Native Microphone API

Discussion in 'Assets and Asset Store' started by Lanre, Sep 17, 2016.

  1. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,626

    Get NatMic on the Asset Store.

    Hi all! I'm very happy to introduce an API that the Unity ecosystem has needed for years. If you've ever tried to record audio with Unity's Microphone class, you have likely run into unfixable bugs, like stuttering and unbearable latency. Now, we have created an API from scratch so that you don't have to face these issues any longer.

    NatMic is a lightweight cross-platform microphone API for Unity. NatMic has support for iOS, Android, macOS, and Windows.

    NatMic Features include:
    - Realtime low-latency recording, with microphone data being provided in a dedicated thread.
    - Configurable microphone sample rate and channel count.
    - Recorders. NatMic comes with a WAV recorder and AudioClip recorder.
    - Cross platform support for iOS, Android, macOS, Windows.

    __

    Check Out a More Detailed Introduction Here.
     
    Last edited: Oct 29, 2019
    Ghosthowl and hangemhigh like this.
  2. hangemhigh

    hangemhigh

    Joined:
    Aug 2, 2014
    Posts:
    56
    Yes! It will make NatCam complete. You can separate it into a new product. I bet there are many people that are willing to pay for this. If you google "Unity Read Mic data", you will see many posts with no solution. Unity's Mic API is extremely slow.

    I do not own NatCam yet but the videos you made about them are awesome. What I need is a fast camera + audio API so that that I can read them and send them over network.

    What this can be used for:

    Online Game Voice Chat
    WebCam Apps
    Video(Front Camera) and Audio live calls
    Waveform Visualization
    Record Audio from the System Sound(Not just sound playing from Unity).
    Record phone call

    I see that the Asset store is full of Voice Chat plugins but those are crappy. Only looking for low access API to read from the Mic hardware.

    NatMic API Structure:

    • API to read Mic data as byte array.
    • API to play the read data byte array.
    • API that reads the Mic and returns the data in a way that be used to construct a waveform.

    The idea is to allocate memory once and re-use them. Usually the read function does not return an array but fills an array that is passed in then returns how many bytes there were filled in. This prevents memory allocation.

    What I plan to do:

    Read audio bytes with NatMic, send to network with TCP/UDP. Server. On my TCP/UDP client, read bytes from server then use NatMic to play received data. I will use other network protocol on WebGL. Hopefully that makes sense. It would be good if NatMic can be used in another Thread as this will improve performance over-all.

    I plan to buy NatCam but since there is no audio plugin yet, I am worried that trying to implement my own audio plugin will conflict with NatCam on Android since I need to extend UnityPlayer. I will wait to see if NatMic will become a thing. Hopefully, it will.

    If you have question, please don't hesitate to ask.
     
  3. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,626
    We would make it separate from NatCam, but completely compatible.
    Interesting. I will look into this.
    Only the microphone input will be supported. Unity's sounds will not at all be recorded.
    This will be done using Unity's Audio API's.
    We'll take care of optimizations. No worries here.
    Playback will be done with Unity API's.
    NatCam does not extend UnityPlayer, so you are completely safe here.
     
    hangemhigh likes this.
  4. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,626
    What we have in mind for the API is this:
    - Record audio data for file saving (record to waveform directly from native layer)
    - Record audio data for streaming (audio data will be sent to Unity and then propagated through a callback)
    - A Waveform class handy for visual animations

    Abstractions will be made for the purposes above to make the process much easier for developers.
     
  5. hangemhigh

    hangemhigh

    Joined:
    Aug 2, 2014
    Posts:
    56
    That's nice.


    Yes, That sums it up. Recording for file saving is also useful. It will be good if you keep me updated in this.
     
  6. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,626
    Sure will. I hope other devs also express interest in this. So far there are 3 developers on board (including you and me).
     
  7. dberroa

    dberroa

    Joined:
    Dec 16, 2012
    Posts:
    141
    One of the projects I'm working on is an online multiplayer VR game for PC that will need voice chat over Unity native multiplayer networking (UNet). If this NatMic would have a plugin that would that would easily integrate with UNet and instantly give me voice chat, then hell yes I would be all over this. I have no clue how to integrate voice chat for a PC VR game.

    For my game I would be getting to the networking part in probably two months.
     
  8. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,626
    I'll be taking a look at the UNet API; I haven't had to use it before.

    I am still doing a feasibility study for this plugin. I'm eager to see what we conclude.
     
  9. hangemhigh

    hangemhigh

    Joined:
    Aug 2, 2014
    Posts:
    56
    I really don't care about UNet. Only care about reading and writing Mic data as fast as possible. Since UNet support has been requested, it looks like it will help many others who don't know how to use TCP/UDP with another Thread.

    One thing to know is that if you decide to support Unet is that there is LLAPI and HLAPI, use the Low level (transport layer) API. LLAPI deals with byte arrays and that makes it very fast. Although nothing beats direct TCP/UDP protocol in terms of performance so I won't be using LLAPI for this unless for WebGL.
     
  10. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,626
    Going off what the majority of developers need, not you alone.
    Thanks for the tip!
     
  11. dberroa

    dberroa

    Joined:
    Dec 16, 2012
    Posts:
    141
    Just chiming in again to say that if you did include voice chat support, you would be one of the only working voice chat plugins on the unity asset store which would make your plugin pretty popular.

    I'm looking now for a plugin and all the ones I find, the reviews either say doesn't work in unity 5, not UNet compatible, or reviews saying it doesn't work period.
     
  12. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,626
    Not necessarily. Exclusivity is not equal to prominence. We are still waiting and watching to see if there is a large-enough volume of developers that would want such a plugin.
     
  13. dberroa

    dberroa

    Joined:
    Dec 16, 2012
    Posts:
    141
    Any update on if you are doing this and what it might entail?

    Thanks.
     
  14. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,626
    Due to the amount of work we have to do, we have decided not to develop the project at this time. This might change, but not anytime soon.
     
  15. Muckel

    Muckel

    Joined:
    Mar 26, 2009
    Posts:
    436
    hello,
    any progress on this?
    looking for mic input waveform visualisation...
    thx
    M.
     
  16. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,626
    We decided not to move forward with the project.
     
  17. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,626
    Hi all. It's almost a year after my last post. We've restarted the NatMic project since it has become a huge roadblock in our progress with some of our projects. We needed a reliable microphone API, and since there are no good options, we have decided to create one ourselves.

    I will edit my first post at the top of this thread, and publish an article describing what we have planned. But to keep you without the suspense, NatMic 1.0 will natively support iOS, Android, Windows, and macOS (all have already been implemented). In the next release, we will add support for WebGL and possibly WinRT. NatMic 1.0 will also support mixing microphone audio with game audio (from an AudioSource or AudioListener). The API will be really simple, consisting of one callback that is provided with PCM audio data. Finally, NatMic will come with `Recorders`, which when given this audio data, will create an audio file. NatMic 1.0 will come with one recorder for recording waveform (WAV) files. In subsequent releases, we'll be looking at adding recorders for M4A, OGG, and MP3.

    Sorry it's taken over a year to get this done!
     
  18. DerrickBarra

    DerrickBarra

    Joined:
    Nov 19, 2013
    Posts:
    103
    Hi @Lanre , do you have an ETA on when NatMic will be coming out?
     
  19. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,626
    Not yet. We are working on getting the iOS implementation finalized. Once we do this, we'll be offering free beta access for a few weeks before releasing it.
     
  20. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,626
    NatMic is now in open beta.

    If you would like to test the API, say so here or PM me your email address so I can share the build with you.
     
    Archviz3d likes this.
  21. Archviz3d

    Archviz3d

    Joined:
    Apr 4, 2016
    Posts:
    92
    AWESOME :) !!! Do you think it would work with unity 5.6.1?
     
  22. TJD269

    TJD269

    Joined:
    Oct 17, 2017
    Posts:
    17
    Dude! This is amazing. 2 months ago I was pulling my hair out at the Unity implementation and today I decided to get back to it for mic over the local network. I actually found this post through google since I was searching for new strategies and was thoroughly surprised to see this came out yesterday! I would gladly beta test and debug for you as this would also help my implementation. I am excited and hopeful you will send me the beta!

    Thanks!!!
     
    Lanre likes this.
  23. yetidevops

    yetidevops

    Joined:
    May 12, 2015
    Posts:
    2
    I was running into issues with recording audio with NatCorder and came across your post about this on Medium. I would love access to this beta!
     
    Lanre likes this.
  24. Rich0

    Rich0

    Joined:
    Aug 9, 2013
    Posts:
    43
    I too would love access to this (I've sent a PM). I've just released an app and the microphone-speaker workaround breaks bluetooth (NatCoder & NatShare work perfectly).
     
    Lanre likes this.
  25. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,626
    Absolutely!
     
    Archviz3d likes this.
  26. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,626
    PM me your email address and I'll share the beta build with you on Dropbox.
     
  27. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,626
    Integration with NatCorder is very very high on our list (it's actually the reason why we decided to restart this project). PM me your email address and I'll share the build with you on Dropbox.
     
  28. Soaryn

    Soaryn

    Joined:
    Apr 17, 2015
    Posts:
    213
    Out of curiosity, I have been trying to find a extreme low latency stream of microphone samples for real time audio visualization. Do you think NatMic is able to do that, and would it be up for some testing with the new ECS system? I noticed you were on Open beta right now so if you'd like I have already pmed you my email :)
     
  29. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,626
    NatMic handles low-latency microphone input. The user-provided callback gets invoked immediately a sample buffer is available (directly on the microphone thread). NatMic currently doesn't provide any visualization or spectral analysis functionality.
     
  30. Archviz3d

    Archviz3d

    Joined:
    Apr 4, 2016
    Posts:
    92
    Hi!! Ive made a test with natmic on android and didnt had any problem! Will try also with ios :) when do you think this will be on the asset store? and integrated with natcorder?
     
    Lanre likes this.
  31. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,626
    We are currently working on NatCorder integration tests. Once this is done, we'll start the Asset Store submission process.
     
  32. Archviz3d

    Archviz3d

    Joined:
    Apr 4, 2016
    Posts:
    92
    Hi @Lanre ! Thats great!!
    Im currently trying to make it work on iOS but its not working! Is it working for ios devices?

    Im using an iPhone 6 with ios 11.4.1

    Ive used unity 5.6.1p1 and unity 2017.3.1p1, an empty project where i imported your plugin and opened your example scene, i also created the permission to record on the player settings. It compiles all ok, and on the iphone it does record the wav file, but doesnt have any sound.

    In Xcode, with the iphone connected to it, i get this error while recording dont know if its related

    2018-09-08 00:48:44.079925+0200 TESTAUDIOREC2017[1637:467765] 55: EXCEPTION (-1): ""

    sample rate: 24000 sample count: 1024

    NatMicU.Examples.VoiceRecorder:OnSampleBuffer(AudioEvent, Single[], Int64, Format)



    (Filename: /Users/builduser/buildslave/unity/build/artifacts/generated/common/runtime/DebugBindings.gen.cpp Line: 51)






    What can be happening?

    Thanks
     
    Lanre likes this.
  33. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,626
    The exception log in Xcode is innocuous; we are looking for ways to remove it. You can ignore it. Regarding the recording being completely silent, I got a similar report on macOS which I am investigating right now. When we tested initially, it worked, so it's strange. I'll update you when we resolve it.
     
    Archviz3d likes this.
  34. Archviz3d

    Archviz3d

    Joined:
    Apr 4, 2016
    Posts:
    92
    Thanks :) it would be awesome if you could make it working on ios!!! I tried on my imac and it did work ok! If you want i can give you more info on the imac versions os etc
     
    Lanre likes this.
  35. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,626
    I'll let you know. Thank you!
     
    Archviz3d likes this.
  36. Archviz3d

    Archviz3d

    Joined:
    Apr 4, 2016
    Posts:
    92
    Great!! I'll be waiting for any advancements on the issue :)
    Thanks
     
  37. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,626
    Archviz3d likes this.
  38. Archviz3d

    Archviz3d

    Joined:
    Apr 4, 2016
    Posts:
    92
    Nice :) Did you manage to reproduce the error on the iphone (no sound on the wav file)?
     
  39. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,626
    I am able to reproduce the error but as I mentioned before, it is innocuous as far as we can tell. We are able to hear the audio. The silence we got on macOS was actually for a different reason (we were working on integrating NatCorder and NatMic). Can you email me a recorded WAV and the full logs from Xcode? Thanks.
     
  40. TillmaniaLtd

    TillmaniaLtd

    Joined:
    Jan 29, 2013
    Posts:
    44
    Hey,

    Ok so I got NatMic running fine in the editor but as soon as I build to a device, the microphone is really distorted, the apps audio volumes get turned right down and this error appears repeatedly in the console of Xcode:



    55: EXCEPTION (-1): ""


    Any ideas on what the issue could be?
     
  41. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,626
    Are you running the VoiceRecorder example? Can you share your code? The exception log in Xcode is apparently a bug in iOS 11; as far as we can tell, it is innocuous.
     
  42. TillmaniaLtd

    TillmaniaLtd

    Joined:
    Jan 29, 2013
    Posts:
    44
    No, but I used this code provided to me earlier. The audio distortion happens even in a blank scene with just a couple of UI elements:

    Code (CSharp):
    1. public class NatMicCorder : MonoBehaviour {
    2.  
    3.     #region --Op vars--
    4.     public Camera recordingCamera;
    5.     private CameraRecorder cameraRecorder;
    6.     #endregion
    7.  
    8.  
    9.     #region --Recording--
    10.  
    11.     public void StartRecording () {
    12.         // Start the microphone with NatMic
    13.         var microphoneFormat = Format.Default;
    14.         NatMic.StartRecording(microphoneFormat, OnSampleBuffer);
    15.         // Start recording with NatCorder
    16.         var audioFormat = new AudioFormat(microphoneFormat.sampleRate, microphoneFormat.channelCount);
    17.         NatCorder.StartRecording(Container.MP4, VideoFormat.Screen, audioFormat, OnRecording);
    18.         // Create a camera recorder for the main cam
    19.         cameraRecorder = CameraRecorder.Create(recordingCamera);
    20.     }
    21.  
    22.     public void StopRecording () {
    23.         // Stop the microphone
    24.         NatMic.StopRecording();
    25.         // Stop recording
    26.         cameraRecorder.Dispose();
    27.         NatCorder.StopRecording();
    28.     }
    29.     #endregion
    30.  
    31.  
    32.     #region --Callbacks--
    33.  
    34.     // Invoked by NatMic on new microphone events
    35.     private void OnSampleBuffer (AudioEvent audioEvent, float[] sampleBuffer, long timestamp, Format format) {
    36.         // Send sample buffers directly to NatCorder for recording
    37.         if (audioEvent == AudioEvent.OnSampleBuffer)
    38.             NatCorder.CommitSamples(sampleBuffer, Frame.CurrentTimestamp); // Use NatCorder's timestamps
    39.     }
    40.  
    41.     // Invoked by NatCorder once recording is finished
    42.     private void OnRecording (string path) {
    43.         // Play the recorded video
    44.         #if UNITY_EDITOR
    45.         UnityEditor.EditorUtility.OpenWithDefaultApp(path);
    46.         #else
    47.         Handheld.PlayFullScreenMovie(path);
    48.         #endif
    49.     }
    50.     #endregion
    51. }
     
  43. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,626
    There's a minor change that needs to be made. In `OnSampleBuffer`, you need to check that NatCorder is recording before calling CommitSamples. Check out this full project demonstrating NatMic and NatCorder integration.
     
  44. TillmaniaLtd

    TillmaniaLtd

    Joined:
    Jan 29, 2013
    Posts:
    44
    Ok so I tried that project, and all the microphone picks up is crazy distortion. This happens in iOS 12 and 12.1(beta). On an iPhone X, iPhone XS Max, an iPhone 7 and an iPhone 5S. All exactly the same distortion

    Unity version 2018.2.9f1

    I recorded it so you can see:

     
  45. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,626
    Replace libNatMic.a in NatMic > Plugins > iOS with the one I've attached and let me know if there's still stuttering.
     

    Attached Files:

  46. TillmaniaLtd

    TillmaniaLtd

    Joined:
    Jan 29, 2013
    Posts:
    44
    Awesome that seems to have fixed things. I'll do some more integration tests after dinner. You're a legend Lanre :)
     
    Lanre likes this.
  47. AlienFreak

    AlienFreak

    Joined:
    Jul 2, 2012
    Posts:
    13
    I'd like to try this on my iPhone X, and some Android platforms. I wrote a spectrum analyzer I want to try with some clean mic input on these platforms.

    Also, is there any ability to combine this audio data with an mp4 video in this, NatCam or NatCorder? (I think you do this when recording with NatCorder but I'd also like to be able to edit this to add/remove/adjust audio or video and then merge them back into one mp4 on iOS or Android)
     
    Last edited: Oct 3, 2018
  48. ThomQ

    ThomQ

    Joined:
    Nov 23, 2010
    Posts:
    303
    Hi!
    I'm really interested in NATmic. We have an option to record a video in our ARapp (iOS & Android). Since integrating NATrecorder we are getting super smooth video, but unfortunately a choppy mic recording, maybe NATmic will help?
     
  49. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,626
    We aren't sharing the beta anymore; we are working with Unity to get it released onto the Asset Store (they requested some things from us that we're taking care of)
    I read two possible questions from this. If you're asking if there's a way to blend game audio with the microphone audio, then yes; NatMic's StartRecording function offers overloads that take in an AudioSource or an AudioListener, and NatMic will provide you with sample buffers with the mixed audio.

    If you're asking whether NatMic offers 'editing' capabilities (any sort of processing after recording has happened), then the answer is no. NatCorder doesn't do any post-recording editing either.
     
  50. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,626
    Absolutely! NatMic will definitely help (NatCorder was the reason why we decided to resume this project). We are working with Unity to get it published on the Asset Store.