Search Unity

NatMic - Native Microphone API

Discussion in 'Assets and Asset Store' started by Lanre, Sep 17, 2016.

  1. yanoty3d

    yanoty3d

    Joined:
    Oct 2, 2014
    Posts:
    2
    I am also getting a choppy mic recording when I record video and audio in Oculus Go. It is so unpredictable and it is driving me crazy. Could I use NatMic open beta for my app? Many thanks in advance!
     
  2. EbowUK

    EbowUK

    Joined:
    Dec 19, 2012
    Posts:
    2
    Hi Lanre,

    Any idea when NatMic will be available on the asset store? Sounds like it might be a good solution to some audio input issues I'm having, so i'm very eager to try it out.
     
    furmank likes this.
  3. mathias_unity633

    mathias_unity633

    Joined:
    Jul 17, 2018
    Posts:
    35
    Hi. I tried following your NatMicCorder.cs example.
    When I record e.g. a 5 second video, the video turns out to be 10 seconds, with the video itself only lasting for 5, freezing the last frame for 5 seconds and the audio running at half speed for the entire 10 second duration.

    What did I mess up?


    Code (CSharp):
    1.     private void BeginRecording()
    2.     {
    3.         _recorder = CameraRecorder.Create(MainARController.Instance.MainCamera);
    4.         _isRecording = true;
    5.  
    6.         var format = NatMicU.Core.Format.Default;
    7.         var audioListener = Camera.main.GetComponent<AudioListener>();
    8.         NatMic.StartRecording(audioListener, format, OnSampleBuffer);
    9.  
    10.         var audioFormat = new AudioFormat(format.sampleRate, format.channelCount);
    11.         NatCorder.StartRecording(   Container.MP4,
    12.                                     new VideoFormat(Screen.width / 2, Screen.height / 2, VideoFormat.Screen.framerate, VideoFormat.Screen.bitrate, VideoFormat.Screen.keyframeInterval),
    13.                                     audioFormat,
    14.                                     OnVideoFinish);
    15.     }
    16.  
    17.     private void OnSampleBuffer(AudioEvent audioEvent, float[] sampleBuffer, long timestamp, Format format)
    18.     {
    19.         // Send sample buffers directly to NatCorder for recording
    20.         if (audioEvent == AudioEvent.OnSampleBuffer && NatCorder.IsRecording)
    21.             NatCorder.CommitSamples(sampleBuffer, Frame.CurrentTimestamp); // Use NatCorder's timestamps instead
    22.     }
    23.  
    24.     private void StopRecording()
    25.     {
    26.         if (_isRecording)
    27.         {
    28.             NatMic.StopRecording();
    29.             NatCorder.StopRecording();
    30.             _recorder.Dispose();
    31.             _isRecording = false;
    32.         }
    33.     }
    34.  
    35.     private void OnVideoFinish(string path)
    36.     {
    37.         Debug.Log($"Finished recording video at path {path}");
    38.         var fileName = Path.GetFileName(path);
    39.         var galleryPath = $"{Application.persistentDataPath}/Gallery";
    40.         string targetPath = $"{galleryPath}/{fileName}";
    41.         if (!Directory.Exists(galleryPath))
    42.         {
    43.             Directory.CreateDirectory(galleryPath);
    44.         }
    45.         File.Copy(path, targetPath);
    46.         File.Delete(path);
    47.     }
    48.  
    49.  
    Tested in Editor on macOS
     
  4. mathias_unity633

    mathias_unity633

    Joined:
    Jul 17, 2018
    Posts:
    35
    Actually I just noticed I haven't updated NatCorder for a couple of weeks, disregard above post for now.
     
    Lanre likes this.
  5. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,542
    NatMic is now available on the Asset Store

    Sorry for this taking an eternity; Unity had us jump through a few more hoops before they'd approve the package. I will be publishing an article on my Medium on integrating NatMic and NatCorder later today. I'm trying to move much faster with NatMic as we have big plans for integrating NatMic, NatCorder, and NatCam.
     
  6. SniperED007

    SniperED007

    Joined:
    Sep 29, 2013
    Posts:
    134
    Glad to see this has been released, sounds very promising.
    Not sure why you would want to integrate NatCam with NatMic?
    This really feels like it should be bundled as part of NatCorder or at least be offered as a discount to those who own NatCorder as the current Mic support in NatCorder is pretty unusable in production (with what I've tested on Android).
     
  7. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,542
    This is for recording mouth movement, and ensuring that the microphone audio perfectly syncs with it. Check out our NatMicCorder Demo where we demonstrate this.
    There's three things I want to touch on. Firstly, NatMic is an independent API for good reason. There are a lot of use cases for the microphone which don't have anything to do with recording. Spectrum analysis is a good example of this. Though NatMic currently doesn't support this, we will be adding FFT in a subsequent release.

    Secondly, we can't offer discounts on the Asset Store. Unity has no provisions for this. Lastly, regarding NatCorder's 'mic support' being unusable. This has been entirely because of Unity's microphone API (which is why we introduced NatMic to begin with). NatCorder doesn't have any special microphone support; it simply records the audio data you provide to it. As has been mentioned, Unity's microphone API is pretty much unusable with its constant glitches. This is what NatMic drastically improves on.
     
  8. furmank

    furmank

    Joined:
    May 24, 2017
    Posts:
    4
    Hi Lenre,

    I'm happy that the API is finally out!
    I've got a question / problem when implementing NatMic with NatCorder in the line
    Code (CSharp):
    1. NatCorder.CommitSamples(sampleBuffer, Frame.CurrentTimestamp); // Use NatCorder's timestamps instead
    where Frame does not exist in the current context. I am using
    Code (CSharp):
    1. using NatCorderU.Core;
    2. using NatCorderU.Examples;
    3. using NatCorderU.Core.Recorders;
    4. using NatShareU;
    5. using NatMicU.Core;
    6. using NatMicU.Core.Recorders;
    What did I miss? How should be Frame implemented there / how to you implement the "Use NatCorder's timestamps instead"?

    Thanks!

    EDIT: maybe a NatCorder question, sorry for that
     
  9. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,542
    You're totally fine. I realized I forgot to update the NatMicCorder demo to use NatCorder 1.3f2. Here's the fix. Sorry about that!
     
  10. SniperED007

    SniperED007

    Joined:
    Sep 29, 2013
    Posts:
    134
    I'm still confused... isn't NatCam for stills (i.e. photos) only?

    Agreed, and I think there is room to have the NatMic as a separate item, but I do think NatCorder and NatMic should be bundled. I honestly thought Unity Asset Store had the ability to give discounts on products based on previous items you have bought.

    I feel your pain, I've logged many bugs with Unity around their Microphone support as we learned the hard way with our game Comedy Night when we started getting negative reviews for mic support not working most Razer Microphones and a few other brands. Never really had any major issues with the glitches which is what surprises me with NatCorder recorded audio being so bad, but maybe it's a mobile vs PC thing.

    Does NatMic work on Linux as well (I know your store page doesn't mention it but is that because you haven't tested it, or doesn't it actually work on Linux)?
     
  11. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,542
    Nope. NatCam entirely replaces WebCamTexture on iOS and Android, giving you full control of device cameras including things like focus, exposure, flash, zoom, and so on.
    This is entirely up to Unity; this isn't exposed to publishers.
    Unfortunately, NatMic doesn't support Linux.

    EDIT: We just might add Linux support soon, and possibly WebGL too.
     
  12. SniperED007

    SniperED007

    Joined:
    Sep 29, 2013
    Posts:
    134
    I understand that... and I'm sorry to continue questioning this but I still don't understand why do you need NatMic integration with NatCam?
     
  13. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,542
    During NatMic development, we had to take special care (in NatCam too) to make sure that the camera preview matched up with the microphone audio. This is a crucial factor when recording the camera preview. A good example is if we are recording a user talking or singing, we want the timestamps on the microphone sample buffers to match up exactly with presentation timestamps of camera preview buffers.
     
    SniperED007 likes this.
  14. mathias_unity633

    mathias_unity633

    Joined:
    Jul 17, 2018
    Posts:
    35
    I'm getting this every frame when trying to record.


    10-22 11:17:47.742 18485 18767 E Unity : Exception: No such proxy method: NatMicU.Core.Platforms.NatMicAndroid.onSampleBuffer(System.Int32,UnityEngine.AndroidJavaObject,System.Int32,System.Int32)
    10-22 11:17:47.742 18485 18767 E Unity : at UnityEngine.AndroidJavaProxy.Invoke (System.String methodName, System.Object[] args) [0x00000] in <00000000000000000000000000000000>:0
    10-22 11:17:47.742 18485 18767 E Unity : at UnityEngine._AndroidJNIHelper.InvokeJavaProxyMethod (UnityEngine.AndroidJavaProxy proxy, System.IntPtr jmethodName, System.IntPtr jargs) [0x00000] in <00000000000000000000000000000000>:0


    On Android, OnePlus 6.

    I'm pretty much following the NatMicCorder example, works fine in editor.

    Btw, you should correct 'audio source' to 'audio listener' in line 94 of NatMic.cs
    Also is there any information in the readme about having to add permission checks for iOS and Android? I would probably add that :)
     
    Last edited: Oct 22, 2018
  15. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,542
  16. furmank

    furmank

    Joined:
    May 24, 2017
    Posts:
    4
    Hi Lenre,

    I have yet another question: my app is crashing due to sample buffer and i can't figure out what i'm doing wrong even after reading the crash course.

    my code:
    Code (CSharp):
    1.     #region --Recording--
    2.  
    3.     public void StartRecording()
    4.     {
    5.         Debug.Log("Start recording");
    6.         SetUiElements(false);
    7.         sliderAnim.Play();
    8.  
    9.         int recordingWidth = isPortrait ? Screen.width : Screen.height;
    10.         int recordingHeight = isPortrait ? Screen.height : Screen.width;
    11.         if (recordingHeight > 1920)
    12.         {
    13.             float tempratio = 1920f / recordingHeight;
    14.             recordingWidth = Mathf.RoundToInt(tempratio * recordingWidth);
    15.             recordingHeight = 1920;
    16.         }
    17.  
    18.         var videoFormat = new VideoFormat(recordingWidth, recordingHeight, 30);
    19.  
    20.         // Start microphone
    21.         var microphoneFormat = Format.Default;
    22.         NatMic.StartRecording(microphoneFormat, OnSampleBuffer);
    23.         // Start recording
    24.         recordingClock = new RealtimeClock();
    25.         var audioFormat = new AudioFormat(microphoneFormat.sampleRate, microphoneFormat.channelCount);
    26.         NatCorder.StartRecording(container, videoFormat, audioFormat, OnReplay);
    27.         // Create a camera recorder for the main cam
    28.         videoRecorder = CameraRecorder.Create(Camera.main, recordingClock);
    29.  
    30.  
    31.         StartCoroutine(TakeSS());
    32.     }
    33.  
    34.  
    35.     public void StopRecording()
    36.     {
    37.         NatMic.StopRecording();
    38.         videoRecorder.Dispose();
    39.         NatCorder.StopRecording();
    40.  
    41.     }
    42.  
    43.     private void OnSampleBuffer(AudioEvent audioEvent, float[] sampleBuffer, long timestamp, Format format)
    44.     {
    45.         // Send sample buffers directly to NatCorder for recording
    46.         if (audioEvent == AudioEvent.OnSampleBuffer && NatCorder.IsRecording)
    47.             NatCorder.CommitSamples(sampleBuffer, recordingClock.CurrentTimestamp); // Use NatCorder's timestamps instead
    48.      
    49.     }
    50. #endregion
    and the log:

    Code (CSharp):
    1. 10-22 18:54:59.263: D/Unity(22068): NatCorder: Preparing video encoder with format: {color-format=2130708361, i-frame-interval=3, mime=video/avc, width=1080, bitrate=5909759, frame-rate=30, height=1776}
    2. 10-22 18:54:59.277: D/Unity(22068): NatMic: Microphone started recording
    3. 10-22 18:54:59.577: D/Unity(22068): NatCorder: Preparing audio encoder with format: {channel-mask=16, sample-rate=44100, mime=audio/mp4a-latm, channel-count=1, aac-profile=2, bitrate=64000}
    4. 10-22 18:54:59.643: E/Unity(22068): NatMic Error: Sample buffer callback raised exception: UnityEngine.AndroidJavaException: java.nio.BufferOverflowException
    5. 10-22 18:54:59.643: E/Unity(22068): java.nio.BufferOverflowException
    6. 10-22 18:54:59.643: E/Unity(22068):     at java.nio.ByteBuffer.put(ByteBuffer.java:544)
    7. 10-22 18:54:59.643: E/Unity(22068):     at com.yusufolokoba.natcorder.mp4.Encoder.encodeData(Encoder.java:60)
    8. 10-22 18:54:59.643: E/Unity(22068):     at com.yusufolokoba.natcorder.mp4.AudioEncoder.encodeSamples(AudioEncoder.java:67)
    9. 10-22 18:54:59.643: E/Unity(22068):     at com.yusufolokoba.natcorder.mp4.MP4Recorder.encodeSamples(MP4Recorder.java:86)
    10. 10-22 18:54:59.643: E/Unity(22068):     at com.yusufolokoba.natcorder.NatCorder.encodeSamples(NatCorder.java:86)
    11. 10-22 18:54:59.643: E/Unity(22068):     at com.unity3d.player.ReflectionHelper.nativeProxyInvoke(Native Method)
    12. 10-22 18:54:59.643: E/Unity(22068):     at com.unity3d.player.ReflectionHelper.a(Unknown Source:0)
    13. 10-22 18:54:59.643: E/Unity(22068):     at com.unity3d.player.ReflectionHelper$1.invoke(Unknown Source:6)
    14. 10-22 18:54:59.643: E/Unity(22068):     at java.lang.reflect.Proxy.invoke(Proxy.java:913)
    15. 10-22 18:54:59.643: E/Unity(22068):     at $Proxy9.onSampleBuffer(Unknown Source)
    16. 10-22 18:54:59.643: E/Unity(22068):     at com.yusufolokoba.natmic.NatMic.run(NatMic.java:86)
    17. 10-22 18:54:59.643: E/Unity(22068):     at java.lang.Thread.run(Thread.java:764)
    18. 10-22 18:54:59.643: E/Unity(22068):   at UnityEngine.AndroidJNISafe.CheckException () [0x00000] in <filename unknown>:0
    19. 10-22 18:54:59.643: E/Unity(22068):   at UnityEngine.AndroidJNISafe.
    20. ...

    EDIT: i think i figured out the problem. chaning microhpone format to DefaultForMixing and assigning audiolistener stopped my problems
     
    Last edited: Oct 22, 2018
  17. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,542
    This is really weird. What version of Unity are you using? And thanks for the note on the typo.
     
  18. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,542
    Buffer overflow exceptions in NatCorder usually means that the audio format passed into `NatCorder.StartRecording` doesn't match the actual data being sent to `NatCorder.CommitSamples`. It looks like you found the culprit to be the wrong format.
     
  19. mathias_unity633

    mathias_unity633

    Joined:
    Jul 17, 2018
    Posts:
    35
    I'm on 2018.1.2f1 You're welcome :)
     
  20. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,542
    Can you email me the full logs from logcat? Don't filter for Unity tags, and send it in a .txt attachment. Thanks.
     
  21. shin_unity197

    shin_unity197

    Joined:
    Oct 30, 2017
    Posts:
    18
    Does Natmic not work well with AudioRecorder in Natcorder? I'm trying to get audio from game and also record with mic but the audio is really messed up with sharp pitches.


    Code (CSharp):
    1.                 m_recordMicrophoneAudio &= m_container == Container.MP4;
    2.                 // Create recording configurations // Clamp video width to 720
    3.                 var width = 720;
    4.                 var height = width * Screen.safeArea.height / Screen.safeArea.width;
    5.                 var framerate = m_container == Container.GIF ? 10 : 30;
    6.                 var videoFormat = new VideoFormat(width, (int)height, framerate);
    7.  
    8.  
    9.                 var microphoneFormat = Format.Default;
    10.                 if(m_recordMicrophoneAudio)
    11.                 {
    12.                     NatMic.StartRecording(microphoneFormat, OnSampleBuffer);
    13.                 }
    14.                 m_recordingClock = new RealtimeClock();
    15.                 var audioFormat = m_recordMicrophoneAudio ? new AudioFormat(microphoneFormat.sampleRate, microphoneFormat.channelCount) : AudioFormat.None;
    16.  
    17.  
    18.                 // Start recording
    19.                 NatCorder.StartRecording(m_container, videoFormat, audioFormat, OnReplay);
    20.                 m_videoRecorder = CameraRecorder.Create(Camera.main, m_recordingClock);
    21.                 // If recording GIF, skip a few frames to give a real GIF look
    22.                 if (m_container == Container.GIF)
    23.                     m_videoRecorder.recordEveryNthFrame = 5;
    24.                 // Start microphone and create audio recorder
    25.                 AudioListener audioListener = Camera.main.GetComponent<AudioListener>();
    26.                 if (m_recordMicrophoneAudio && audioListener != null) {
    27.                     //StartMicrophone();
    28.                     // m_audioRecorder = AudioRecorder.Create(audioListener, m_audioSource, true, m_recordingClock);
    29.                     m_audioRecorder = AudioRecorder.Create(audioListener, m_recordingClock);
    30.                 }
    Also when I just record normally removing the AudioRecorder.Create(audiolistener, m_recordingClock). There is no recording whatsoever. Below is my log cat when I begin recording.
    upload_2018-10-25_15-49-43.png
     
    Last edited: Oct 25, 2018
  22. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,542
    If you are recording from the microphone, then don't create an AudioRecorder, use only NatMic. Regarding the error you are seeing, this is because IL2CPP is stripping away some methods. Change your scripting backend to Mono in Player Settings to fix this.
     
  23. mathias_unity633

    mathias_unity633

    Joined:
    Jul 17, 2018
    Posts:
    35
    Done
     
  24. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,542
    IL2CPP seems to be stripping out the C# sample buffer delegate during compilation. Change your app's scripting backend to Mono to resolve this. I have a fix coming for IL2CPP in a minor update sometime next week.
     
  25. shin_unity197

    shin_unity197

    Joined:
    Oct 30, 2017
    Posts:
    18
    @Lanre If I can't create Audiorecorder, then how do I record both my ingame audio and mic audio at the same time?
     
  26. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,542
    NatMic has an overload for StartRecording that takes in an AudioListener. If you use this overload, NatMic will automatically mix the game audio with the microphone audio in the sample buffer that gets passed into your callback. You can then send this sample buffer to NatCorder for encoding.
     
  27. FK22

    FK22

    Joined:
    Jul 7, 2017
    Posts:
    28
    Hi,

    I wanted to ask you: -Does this program supports audio record from computer microphone and webcam record from computer webcam camera? -Also is this possible in WebGL?

    I am creating this application in the web (Using Unity and WebGL) as a research project and I want to record and save some microphone input and webcam input. Is this possible with your program?

    What would be the best to get: Nat Cam, Nat Mic or Nat Corder? I want this for windows computer.

    Thanks
     
  28. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,542
    NatMic currently doesn't support WebGL. We are looking into the possibility of adding this, but I can't provide any definitive timelines at this point.
     
  29. mathias_unity633

    mathias_unity633

    Joined:
    Jul 17, 2018
    Posts:
    35
    I need IL2CPP for other dependencies so I'll wait for the fix :)
     
  30. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,542
    You can actually add the fix yourself. Open NatMicAndroid.cs and do two things:
    1. Add this:
    Code (CSharp):
    1. using UnityEngine.Scripting;
    at the top of the file, with all the other `using` statements.
    2. Scroll down to the `onSampleBuffer` function, and add the `Preserve` attribute:
    Code (CSharp):
    1. [Preserve]
    2. private void onSampleBuffer (...) {
    3.     ...
    4. }
     
  31. shin_unity197

    shin_unity197

    Joined:
    Oct 30, 2017
    Posts:
    18
    @Lanre I need IL2CPP for other dependencies as well. The following changes above did not change anything. I still get the Exception: No such proxy method error.
     
  32. ErickEiben

    ErickEiben

    Joined:
    Oct 26, 2018
    Posts:
    1
    Hey, so I am currently doing research into possible plugins that will help me with real-time audio in an app. The idea is you speak into the microphone and it plays back your voice with a modulation effect on it, do you think this plugin will be useful for real-time playback with a minimal delay?
     
  33. DerrickBarra

    DerrickBarra

    Joined:
    Nov 19, 2013
    Posts:
    98
    @Lanre I am also getting errors with NatMic. In Unity Editor no audio is recorded and on my Android Galaxy S8 running Android 8.0.0 we are receiving an Illegal State Exception error while recording (which normally ends in an Out of Memory crash error).

    In Windows 10, on Unity 2018.2.14f1, using NatMic 1.0f1, NatCorder 1.3f2, NatCam 2.0f4, NatShare 1.1f2.

    For a reproducible test project, I am using the NatMicCorder demo on GitHub.

    Code (CSharp):
    1. AndroidJavaException: java.lang.IllegalStateException
    2. 11-01 15:04:41.294 26463 26531 E Unity   : java.lang.IllegalStateException
    3. 11-01 15:04:41.294 26463 26531 E Unity   :      at android.media.MediaCodec.native_dequeueInputBuffer(Native Method)
    4. 11-01 15:04:41.294 26463 26531 E Unity   :      at android.media.MediaCodec.dequeueInputBuffer(MediaCodec.java:2559)
    5. 11-01 15:04:41.294 26463 26531 E Unity   :      at com.yusufolokoba.natcorder.mp4.Encoder.encodeData(Encoder.java:55)
    6. 11-01 15:04:41.294 26463 26531 E Unity   :      at com.yusufolokoba.natcorder.mp4.AudioEncoder.encodeSamples(AudioEncoder.java:57)
    7. 11-01 15:04:41.294 26463 26531 E Unity   :      at com.yusufolokoba.natcorder.mp4.MP4Recorder.stopRecording(MP4Recorder.java:59)
    8. 11-01 15:04:41.294 26463 26531 E Unity   :      at com.yusufolokoba.natcorder.NatCorder.stopRecording(NatCorder.java:60)
    9. 11-01 15:04:41.294 26463 26531 E Unity   :      at com.unity3d.player.UnityPlayer.nativeRender(Native Method)
    10. 11-01 15:04:41.294 26463 26531 E Unity   :      at com.unity3d.player.UnityPlayer.c(Unknown Source:0)
    11. 11-01 15:04:41.294 26463 26531 E Unity   :      at com.unity3d.player.UnityPlayer$e$2.queueIdle(Unknown Source:72)
    12. 11-01 15:04:41.294 26463 26531 E Unity   :      at android.os.MessageQueue.next(MessageQueue.java:394)
    13. 11-01 15:04:41.294 26463 26531 E Unity   :      at android.os.Looper.loop(Looper.java:142)
    14. 11-01 15:04:41.294 26463 26531 E Unity   :      at com.unity3d.player.UnityPlayer$e.run(Unknown Source:33)
    15. 11-01 15:04:41.294 26463 26531 E Unity   :   at UnityEngine.AndroidJNISafe.CheckException () [0x0008c] in /Users/builduser/buildslave/unity/build/Runtime/Export/AndroidJNISafe.cs:25
     
  34. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,542
    That's weird. I tested the fix on my end and it worked. I'm using IL2CPP and have the C++ compiler running in Release mode (so I'd expect the most aggressive stripping). Can you email me your invoice number? I'd like to share my build with you to test.
     
  35. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,542
    NatMic doesn't provide functionality for audio effects. Also, NatMic currently doesn't provide any means for realtime playback. I am looking into adding this because I'm working on a VoIP demo.
     
  36. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,542
    Can you email me with the full logs from logcat (no filtering)? That should shed light on what is going on.
     
  37. mathias_unity633

    mathias_unity633

    Joined:
    Jul 17, 2018
    Posts:
    35
    Any progress on the IL2CPP + Android = No such proxy method error? :)
     
  38. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,542
    I still haven't been able to reproduce this. Can you email me? I'd like to have to test my current build to see if we can pin-point the source.
     
  39. mathias_unity633

    mathias_unity633

    Joined:
    Jul 17, 2018
    Posts:
    35
    I can, but what do you want me to e-mail you? :)
    I sent a log earlier to your e-mail.
    It was on Unity 2018.1.2f1, Android version 9 on a OnePlus 6 (Build number ONEPLUS A6003_22_181026) with IL2CPP.
     
  40. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,542
    Send me your invoice number; I'll send you my current build of the upcoming NatMic 1.1 to test.
     
  41. ryancookyeti

    ryancookyeti

    Joined:
    Mar 6, 2017
    Posts:
    5
    @Lanre
    Do you know how to commit NatMic's samples to a Unity AudioClip? Im in a predicament where we are using NatMic to record microphone during gameplay but there is a scene that uses Unity's Microphone API to do some audio visualization stuff by listening to an AudioSource that is playing the AudioClip created by UnityEngine.Microphone.Start

    The target platform is Android and it does not support having two resources use the microphone at the same time. So i would rather retro-fit the scene to use NatMic's audio samples and just encode those to an audio clip in a similar way to Unity's Microphone class. I've created the class below to handle the conversion of NatMic's audio samples to an AudioClip but it seems to only kind of work. Playback of the clip has a really slowed down, low pitched, demon voice quality to it.

    Code (CSharp):
    1.  
    2. public class MicrophoneAudioSource : MonoBehaviour, IMicrophoneListener
    3. {
    4.     private float _clipLength = 5.0f;
    5.     private int _clipFrequencey = 24000;
    6.     private AudioSource _audioSource;
    7.     private List<float[]> _samples = new List<float[]>();
    8.     private int _curOffsetSamples;
    9.  
    10.     private void Start()
    11.     {
    12.         _audioSource = GetComponent<AudioSource>();
    13.      
    14.         // Create audio clip for mic samples to be set to.
    15.         int sampleLength = (int)(_clipFrequencey * _clipLength);
    16.         var channels = MyMicrophone.MicrophoneFormat.channelCount;
    17.         _audioSource.clip = AudioClip.Create("MyMicrophone Clip", sampleLength, channels, _clipFrequencey, false, false);
    18.  
    19.         // Start listening for microphone samples.
    20.         // MyMicrophone is a simple wrapper for NatMic that just allows for more than one
    21.         // object to listen for mic audio samples.
    22.         MyMicrophone.AddListener(this);
    23.         MyMicrophone.StartRecording();
    24.  
    25.         // Hookup the AudioMixerGroup to the AudioSource
    26.         _audioSource.outputAudioMixerGroup = mixer;
    27.      
    28.         // Start playing back the audio streamed in from the mic.
    29.         // We won't hear the mic played back, because it's muted through the AudioMixerGroup
    30.         _audioSource.Play();
    31.     }
    32.  
    33.     private void Update()
    34.     {
    35.         if (_samples.Count > 0)
    36.         {
    37.             var samplesToTransfer = new List<float[]>(_samples);
    38.  
    39.             for (int i = 0; i < samplesToTransfer.Count; i++)
    40.             {
    41.                 _audioSource.clip.SetData(samplesToTransfer[i], _curOffsetSamples);
    42.                 _curOffsetSamples += samplesToTransfer[i].Length;
    43.  
    44.                 if (_curOffsetSamples >= _audioSource.clip.samples)
    45.                     _curOffsetSamples = 0;
    46.             }
    47.  
    48.             _samples.RemoveRange(0, samplesToTransfer.Count);
    49.         }
    50.     }
    51.  
    52.     public void OnMicrophoneSamples(float[] samples)
    53.     {
    54.         // Store the samples until the next update call is recived where we will set them on the audio clip.
    55.         // We need to do this because the callback from NatMic is happening on a seperate thread.
    56.         _samples.Add(samples);
    57.     }
    58. }
    59.  
     
    Last edited: Nov 8, 2018
  42. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,542
    We have a ClipRecorder coming in NatMic 1.1, but this isn't meant for realtime playback; the clip is created once recording is finished.
    I'd highly recommend re-engineering your code to use raw PCM samples instead (so that you can use not only NatMic, but any other audio DSP implementation).
    You should use the PCMReaderCallback in AudioClip.Create instead of using SetData. The PCMReaderCallback is suited for streaming, which is what you intend to do. Also, use a circular buffer (look at NatMic's utility scripts) instead of using a List<float[]>. But again, I highly recommend re-engineering your audio to use raw PCM samples instead of Unity's AudioSource/AudioClip infrastructure.
    Lmao.
     
  43. mathias_unity633

    mathias_unity633

    Joined:
    Jul 17, 2018
    Posts:
    35
    Hiya, just checking in to say that the 1.1 build fixed the no proxy method error on Android for me, thanks!
     
    Lanre likes this.
  44. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,542
    Thank you!
     
  45. niv_onehamsa

    niv_onehamsa

    Joined:
    Oct 30, 2016
    Posts:
    11
    Hi,

    Can NatMic be used on 2017.4.14f1 (the LTS version). It is flagged as 2018.2.14 for minimum version ...
     
  46. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,542
    Yes it can. The Asset Store says 2018.2 because that was the version used to submit the package.
     
  47. bastianmeyerbm3

    bastianmeyerbm3

    Joined:
    May 22, 2018
    Posts:
    15
    Okay, first i am totally sorry that i dont get the readme files.

    But how can i integrate NatMic into the NatCorder ReplayCam.cs?

    Code (CSharp):
    1. /*
    2. *   NatCorder
    3. *   Copyright (c) 2018 Yusuf Olokoba
    4. */
    5.  
    6. namespace NatCorderU.Examples {
    7.  
    8.     using UnityEngine;
    9.     using UnityEngine.UI;
    10.     using System.Collections;
    11.     using Core;
    12.     using Core.Recorders;
    13.     using UnityEngine.SceneManagement;
    14.     using System.IO;
    15.  
    16.  
    17.     public class ReplayCam : MonoBehaviour {
    18.  
    19.         /**
    20.         * ReplayCam Example
    21.         * -----------------
    22.         * This example records the screen using a `CameraRecorder`.
    23.         * When we want mic audio, we play the mic to an AudioSource and record the audio source using an `AudioRecorder`
    24.         * -----------------
    25.         * Note that UI canvases in Overlay mode cannot be recorded, so we use a different mode (this is a Unity issue)
    26.         */
    27.  
    28.         [Header("Recording")]
    29.         public Container container = Container.MP4;
    30.  
    31.         [Header("Microphone")]
    32.         public bool recordMicrophone;
    33.         public AudioSource microphoneSource;
    34.  
    35.         private CameraRecorder videoRecorder;
    36.         private AudioRecorder audioRecorder;
    37.  
    38.         public void StartRecording () {
    39.             // First make sure recording microphone is only on MP4
    40.             recordMicrophone &= container == Container.MP4;
    41.             // Create recording configurations // Clamp video width to 720
    42.             var width = 720;
    43.             var height = width * Screen.height / Screen.width;
    44.             var framerate = container == Container.GIF ? 10 : 30;
    45.             var videoFormat = new VideoFormat(width, (int)height, framerate);
    46.             var audioFormat = recordMicrophone ? AudioFormat.Unity: AudioFormat.None;
    47.             // Start recording
    48.             NatCorder.StartRecording(container, videoFormat, audioFormat, OnReplay);
    49.             videoRecorder = CameraRecorder.Create(Camera.main);
    50.             // If recording GIF, skip a few frames to give a real GIF look
    51.             if (container == Container.GIF)
    52.                 videoRecorder.recordEveryNthFrame = 5;
    53.             // Start microphone and create audio recorder
    54.             if (recordMicrophone) {
    55.                 StartMicrophone();
    56.                 audioRecorder = AudioRecorder.Create(microphoneSource, true);
    57.             }
    58.         }
    59.  
    60.         private void StartMicrophone () {
    61.             #if !UNITY_WEBGL || UNITY_EDITOR // No `Microphone` API on WebGL :(
    62.             // Create a microphone clip
    63.             microphoneSource.clip = Microphone.Start(null, true, 60, 48000);
    64.             while (Microphone.GetPosition(null) <= 0) ;
    65.             // Play through audio source
    66.             microphoneSource.timeSamples = Microphone.GetPosition(null);
    67.             microphoneSource.loop = true;
    68.             microphoneSource.Play();
    69.             #endif
    70.         }
    71.  
    72.         public void StopRecording () {
    73.             // Stop the microphone if we used it for recording
    74.             if (recordMicrophone) {
    75.                 Microphone.End(null);
    76.                 microphoneSource.Stop();
    77.                 audioRecorder.Dispose();
    78.             }
    79.             // Stop the recording
    80.             videoRecorder.Dispose();
    81.             NatCorder.StopRecording();
    82.         }
    83.  
    84.         void OnReplay (string path) {
    85.             Debug.Log("Saved recording to: "+path);
    86.          
    87.             //do something
    88.         }
    89.     }
    90. }

    i am really struggling to get this work :confused: Thank you a lot, i really appreciate the support!
     
  48. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,542
    Okay here's some pseudocode to guide you.
     
    bastianmeyerbm3 likes this.
  49. bastianmeyerbm3

    bastianmeyerbm3

    Joined:
    May 22, 2018
    Posts:
    15
    Lanre likes this.
  50. zolrath

    zolrath

    Joined:
    Dec 14, 2017
    Posts:
    1
    Do you have the ability to disable Automatic Gain Control in NatMic? When using the default Unity Microphone implementation there doesn't seem to be a way to disable this feature on Android/iOS.

    Thanks!u