Search Unity

  1. Unity 2019.2 is now released.
    Dismiss Notice

NatMic - Native Microphone API

Discussion in 'Assets and Asset Store' started by Lanre, Sep 17, 2016.

  1. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,533
    No there currently isn't a way to do so. We will look into it.
     
  2. adam_unity235

    adam_unity235

    Joined:
    Dec 5, 2018
    Posts:
    2
    Is it possible to choose one of multiple mic's available on the system? (Running in the editor on macOS).
     
  3. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,533
    That's currently not possible.
     
  4. adam_unity235

    adam_unity235

    Joined:
    Dec 5, 2018
    Posts:
    2
    I've bought the plugin; is the code made available such that I could create a PR to add this as a feature? I'm going to have to build it anyway, your product might as well benefit from it.
     
  5. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,533
    I'll be adding this feature in the next version.
     
  6. rroylance

    rroylance

    Joined:
    May 1, 2018
    Posts:
    11
    Hi @Lanre, I can't get the NatMic plugin working with NatCorder, I followed the demo project but am just getting the following errors;

    Code (CSharp):
    1. // When starting the recording
    2. NatMic Error: Sample buffer callback raised exception: UnityEngine.AndroidJavaException: java.nio.BufferOverflowException
    3. uid=10211() NatMic Micropho identical 2 lines
    4. NatMic Error: Sample buffer callback raised exception: UnityEngine.AndroidJavaException: java.nio.BufferOverflowException
    5. RecordThread: buffer overflow
    6.  
    7. // When Stopping the recording
    8. AndroidJavaException: java.lang.IllegalStateException
    9.  
    Code (CSharp):
    1.     public void SetRecordingVideo(bool startRecording) {
    2.         if (startRecording) {
    3.             // Start the microphone
    4.             Format microphoneFormat = Format.Default;
    5.             NatMic.StartRecording(microphoneFormat, OnSampleBuffer);
    6.             // Start recording
    7.             recordingClock = new RealtimeClock();
    8.             AudioFormat audioFormat = new AudioFormat(microphoneFormat.sampleRate, microphoneFormat.channelCount);
    9.             NatCorder.StartRecording(Container.MP4, VideoFormat.Screen, audioFormat, OnRecording);
    10.             // Create a camera recorder for the main cam
    11.             cameraRecorder = CameraRecorder.Create(recordingCamera, recordingClock);
    12.         } else {
    13.             // Stop the microphone
    14.             NatMic.StopRecording();
    15.             // Stop recording
    16.             cameraRecorder.Dispose();
    17.             NatCorder.StopRecording();
    18.         }
    19.     }
    20.  
    21.     // Invoked by NatMic on new microphone events
    22.     private void OnSampleBuffer (AudioEvent audioEvent, float[] sampleBuffer, long timestamp, Format format) {
    23.         // Send sample buffers directly to NatCorder for recording
    24.         if (audioEvent == AudioEvent.OnSampleBuffer && NatCorder.IsRecording) {
    25.             NatCorder.CommitSamples(sampleBuffer, recordingClock.CurrentTimestamp);
    26.         }
    27.     }
    28.  
    I get even more errors when trying to record the game audio as well.

    Code (CSharp):
    1.     public void SetRecordingVideo(bool startRecording) {
    2.         if (startRecording) {
    3.             // Start the microphone
    4.             Format microphoneFormat = Format.DefaultForMixing;
    5.             NatMic.StartRecording(microphoneFormat, OnSampleBuffer);
    6.             // Start recording
    7.             recordingClock = new RealtimeClock();
    8.             AudioFormat audioFormat = new AudioFormat(microphoneFormat.sampleRate, microphoneFormat.channelCount);
    9.             NatCorder.StartRecording(Container.MP4, VideoFormat.Screen, audioFormat, OnRecording);
    10.             // Create a camera recorder for the main cam
    11.             cameraRecorder = CameraRecorder.Create(recordingCamera, recordingClock);
    12.             // Create an audio recorder to record game audio
    13.             audioRecorder = AudioRecorder.Create(audioListener, recordingClock);
    14.         } else {
    15.             // Stop the microphone
    16.             NatMic.StopRecording();
    17.             // Stop recording
    18.             cameraRecorder.Dispose();
    19.             audioRecorder.Dispose();
    20.             NatCorder.StopRecording();
    21.         }
    22.     }
    23.  
    24.     // Invoked by NatMic on new microphone events
    25.     private void OnSampleBuffer (AudioEvent audioEvent, float[] sampleBuffer, long timestamp, Format format) {
    26.         // Send sample buffers directly to NatCorder for recording
    27.         if (audioEvent == AudioEvent.OnSampleBuffer && NatCorder.IsRecording) {
    28.             NatCorder.CommitSamples(sampleBuffer, recordingClock.CurrentTimestamp);
    29.         }
    30.     }
    31.  
     
    Last edited: Dec 11, 2018
  7. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,533
    Your code looks good. The BufferOverflowException is caused by NatCorder. We have a fix for it coming in the next update. The IllegalStateException is as a result of the buffer overflow. But you shouldn't be facing errors when mixing mic audio with the audio listener audio. Can you email me those logs?
     
    rroylance likes this.
  8. rroylance

    rroylance

    Joined:
    May 1, 2018
    Posts:
    11
    Hi. Thanks for the quick response. Good to know the BufferOverflow will be fixed soon.

    I've emailed you the logs for when I'm attempting to mix the game and mic audio as well, it spits out a tonne of errors and just locks up the app within a couple seconds.
     
  9. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,533
    Thanks I got it. I just noticed something interesting. When recording just mic audio, the channel count is 2 (stereo). This shouldn't be the case when using Format.Default, since Format.Default is hard-coded to have 1 channel. I assumed that the first logs you sent was from recording just mic audio with no mixing. Is this correct? If so, did you make any mods to NatCorder such that the audio format being passed to StartRecording would have 2 channels instead of 1 when using the default mic format?
     
    rroylance likes this.
  10. rroylance

    rroylance

    Joined:
    May 1, 2018
    Posts:
    11
    I may have accidentally left the Format as DefaultForMixing… my bad. I was playing around with things trying to get it working. I've sent you an updated log that is for no mixing using Format.Default and having 1 channel as expected. Sorry about that.

    That wasn't the cause of my issues however, as I initially got the issue and started playing around before trying to add game audio into the mix. *and as you noted, the BufferOverflow issues is in NatCorder and will be fixed soon, I assume that will resolve my first issue that I send you the original logs for.
     
    Last edited: Dec 11, 2018
  11. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,533
    Looks good, thanks! I just wanted to make sure that there wasn't any detail I was skipping. It's the same buffer overflow exception which will be in the next NatCorder update (already fixed).
     
  12. kanpanstar

    kanpanstar

    Joined:
    Sep 16, 2018
    Posts:
    3
    I Same issue too. with natcorder in Android.

    Code (CSharp):
    1. NatCorder: Preparing video encoder with format: {color-format=2130708361, i-frame-interval=3, mime=video/avc, width=1440, bitrate=5909759, frame-rate=30, height=2560}
    2. NatMic: Microphone started recording with format: 1@48000Hz
    3. NatCorder: Preparing audio encoder with format: {channel-mask=12, sample-rate=48000, mime=audio/mp4a-latm, channel-count=2, aac-profile=2, bitrate=64000}
    4. NatMic Error: Sample buffer callback raised exception: UnityEngine.AndroidJavaException: java.nio.BufferOverflowException
    5. java.nio.BufferOverflowException
    6.     at java.nio.ByteBuffer.put(ByteBuffer.java:615)
    7.     at com.yusufolokoba.natcorder.mp4.Encoder.encodeData(Encoder.java:60)
    8.     at com.yusufolokoba.natcorder.mp4.AudioEncoder.encodeSamples(AudioEncoder.java:67)
    9.     at com.yusufolokoba.natcorder.mp4.MP4Recorder.encodeSamples(MP4Recorder.java:86)
    10.     at com.yusufolokoba.natcorder.NatCorder.encodeSamples(NatCorder.java:86)
    11.     at com.unity3d.player.ReflectionHelper.nativeProxyInvoke(Native Method)
    12.     at com.unity3d.player.ReflectionHelper.a(Unknown Source:0)
    13.     at com.unity3d.player.ReflectionHelper$1.invoke(Unknown Source:59)
    14.     at java.lang.reflect.Proxy.invoke(Proxy.java:1006)
    15.     at $Proxy9.onSampleBuffer(Unknown Source)
    16.     at com.yusufolokoba.natmic.NatMic.run(NatMic.java:83)
    17.     at java.lang.Thread.run(Thread.java:764)
    18.   at UnityEngine.AndroidJNISafe.CheckException () [0x00000] in <filename unknown>:0
    19.   at UnityEngine.AndroidJNISaf
    20. NatMic Error: Sample buffer callback raised exception: UnityEngine.AndroidJavaException: java.nio.BufferOverflowException
    21. java.nio.BufferOverflowException
    22.     at java.nio.ByteBuffer.put(ByteBuffer.java:615)
    23.     at com.yusufolokoba.natcorder.mp4.Encoder.encodeData(Encoder.java:60)
    24.     at com.yusufolokoba.natcorder.mp4.AudioEncoder.encodeSamples(AudioEncoder.java:67)
    25.     at com.yusufolokoba.natcorder.mp4.MP4Recorder.encodeSamples(MP4Recorder.java:86)
    26.     at com.yusufolokoba.natcorder.NatCorder.encodeSamples(NatCorder.java:86)
    27.     at com.unity3d.player.ReflectionHelper.nativeProxyInvoke(Native Method)
    28.     at com.unity3d.player.ReflectionHelper.a(Unknown Source:0)
    29.     at com.unity3d.player.ReflectionHelper$1.invoke(Unknown Source:59)
    30.     at java.lang.reflect.Proxy.invoke(Proxy.java:1006)
    31.     at $Proxy9.onSampleBuffer(Unknown Source)
    32.     at com.yusufolokoba.natmic.NatMic.run(NatMic.java:83)
    33.     at java.lang.Thread.run(Thread.java:764)
    34.   at UnityEngine.AndroidJNISafe.CheckException () [0x00000] in <filename unknown>:0
    35.   at UnityEngine.AndroidJNISaf
    36. NatMic Error: Sample buffer callback raised exception: UnityEngine.AndroidJavaException: java.nio.BufferOverflowException
    37. java.nio.BufferOverflowException
    38.     at java.nio.ByteBuffer.put(ByteBuffer.java:615)
    39.     at com.yusufolokoba.natcorder.mp4.Encoder.encodeData(Encoder.java:60)
    40.     at com.yusufolokoba.natcorder.mp4.AudioEncoder.encodeSamples(AudioEncoder.java:67)
    41.     at com.yusufolokoba.natcorder.mp4.MP4Recorder.encodeSamples(MP4Recorder.java:86)
    42.     at com.yusufolokoba.natcorder.NatCorder.encodeSamples(NatCorder.java:86)
    43.     at com.unity3d.player.ReflectionHelper.nativeProxyInvoke(Native Method)
    44.     at com.unity3d.player.ReflectionHelper.a(Unknown Source:0)
    45.     at com.unity3d.player.ReflectionHelper$1.invoke(Unknown Source:59)
    46.     at java.lang.reflect.Proxy.invoke(Proxy.java:1006)
    47.     at $Proxy9.onSampleBuffer(Unknown Source)
    48.     at com.yusufolokoba.natmic.NatMic.run(NatMic.java:83)
    49.     at java.lang.Thread.run(Thread.java:764)
    50.   at UnityEngine.AndroidJNISafe.CheckException () [0x00000] in <filename unknown>:0
    51.   at UnityEngine.AndroidJNISaf
    52. NatCam Rendering: Created ES3 GLRenderContext
    53. AndroidJavaException: java.lang.IllegalStateException
    54. java.lang.IllegalStateException
    55.     at android.media.MediaCodec.native_dequeueInputBuffer(Native Method)
    56.     at android.media.MediaCodec.dequeueInputBuffer(MediaCodec.java:2635)
    57.     at com.yusufolokoba.natcorder.mp4.Encoder.encodeData(Encoder.java:55)
    58.     at com.yusufolokoba.natcorder.mp4.AudioEncoder.encodeSamples(AudioEncoder.java:57)
    59.     at com.yusufolokoba.natcorder.mp4.MP4Recorder.stopRecording(MP4Recorder.java:59)
    60.     at com.yusufolokoba.natcorder.NatCorder.stopRecording(NatCorder.java:60)
    61.     at com.unity3d.player.UnityPlayer.nativeRender(Native Method)
    62.     at com.unity3d.player.UnityPlayer.c(Unknown Source:0)
    63.     at com.unity3d.player.UnityPlayer$e$2.queueIdle(Unknown Source:72)
    64.     at android.os.MessageQueue.next(MessageQueue.java:395)
    65.     at android.os.Looper.loop(Looper.java:160)
    66.     at com.unity3d.player.UnityPlayer$e.run(Unknown Source:32)
    67.   at UnityEngine.AndroidJNISafe.CheckException () [0x00000] in <filename unknown>:0
    68.   at UnityEngine.AndroidJNISafe.CallVoidMethod (IntPtr
    69.  
    Code (CSharp):
    1. private void StartRecording() {
    2.     // Start the microphone with NatMic
    3.     var microphoneFormat = Format.Default;
    4.     NatMic.StartRecording(microphoneFormat, OnSampleBuffer);
    5.     // Start recording with NatCorder
    6.     recordingClock = new RealtimeClock();
    7.     var audioFormat = new AudioFormat(microphoneFormat.sampleRate, microphoneFormat.channelCount);
    8.     // Start recording
    9.     NatCorder.StartRecording(Container.MP4, VideoFormat.Screen, audioFormat, OnRecording);
    10.     cameraRecorder = CameraRecorder.Create(Camera.main, recordingClock);
    11. }
    12.  
    13. private void StopRecording() {
    14.     // Stop the microphone
    15.     NatMic.StopRecording();
    16.     // Stop the recording
    17.     cameraRecorder.Dispose();
    18.     NatCorder.StopRecording();
    19. }
    20.  
    21. // Invoked by NatMic on new microphone events
    22. private void OnSampleBuffer(AudioEvent audioEvent, float[] sampleBuffer, long timestamp, Format format)
    23. {
    24.     // Send sample buffers directly to NatCorder for recording
    25.     if (audioEvent == AudioEvent.OnSampleBuffer && NatCorder.IsRecording)
    26.         NatCorder.CommitSamples(sampleBuffer, recordingClock.CurrentTimestamp);
    27. }
     
  13. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,533
    I'll let you know when the NatCorder update is live.
     
  14. kanpanstar

    kanpanstar

    Joined:
    Sep 16, 2018
    Posts:
    3
    Thank's. can you give me when the Natcorder roughly update schedule is?
     
  15. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,533
    Instead of publishing the update, we are opening it to current users for testing. We're doing this for a few reasons, before we actually submit it to the Asset Store. Email me your NatCorder invoice number and I'll send the build.
     
  16. RadiantSilvergun

    RadiantSilvergun

    Joined:
    Dec 27, 2017
    Posts:
    11
    Hi Lanre,

    Firstly, great product! I have purchased NatCorder, NatMic and NatShare and find them extremely useful. I have built an app that performs some 3d graphics operations while recording with NatMic. The first time the user hits record, the app slows down and the 3d graphic's frame rate drops for about a second and is very noticeable. Now the second time the user records there is no problem and everything runs smoothly. So this issue seems to be related to initialising NatMic, is there a way to initialise everything without actually recording anything? If so, can I complete this operation on app start up, so the user won't experience any slowdown on their first recording?

    Thanks for your time,

    Travis
     
  17. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,533
    Hey there Travis. What platform does this happen on?
     
  18. RadiantSilvergun

    RadiantSilvergun

    Joined:
    Dec 27, 2017
    Posts:
    11
  19. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,533
    I can't tell what could be causing the noticeable hiccup when you start the microphone. Is it only NatMic.StartRecording that is called when the hiccup happens? Are you doing anything else? You should be able to hide the hiccup by calling NatMic.StartRecording on a background thread. NatMic's API is thread safe.
     
  20. RadiantSilvergun

    RadiantSilvergun

    Joined:
    Dec 27, 2017
    Posts:
    11
    @Lanre It is definitely initialising Natmic that is causing the slowdown, as no slowdown occurs when I don't run NatMic.StartRecording(). I am aware using separate threads would solve it but it shouldn't slow down so much when first run. It would be better if I could preload the initialisation of NatMic. The fact that it only happens on the first recording and not subsequent times also conforms it is not any other part of the code and it is only to do with initialising NatMic. Is it possible to initialise partially without actually recording something?

    I am not very well up on multi-threading in unity, the last time I checked it was a no go haha. Do you have any small code snippets that would start the recording on a separate thread, if that is the only solution?
     
  21. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,533
    You can probably just call NatMic.StartRecording in a Start function so that the mic starts anyway, then keep the mic on till you want to use it. I'm looking into the lag to see if it can be mitigated.
    Simply create a new thread, passing in a lambda or function to start NatMic. Here's code on how to start a thread.
     
    RadiantSilvergun likes this.
  22. shallwaycn

    shallwaycn

    Joined:
    Apr 3, 2012
    Posts:
    198
    Hi Lanre
    Thanks for the great tool.
    I'm using NatCorder and NatMic in iOS.
    After stop recording, then check and play it in a native UIImagePickerController, 100% crash.
    Without NatMic it works fine.
    Can you take a look at this? here is the log.
    Code (CSharp):
    1. Application Specific Information:
    2. abort() called
    3.  
    4. Thread 0 name:  Dispatch queue: com.apple.main-thread
    5. Thread 0 Crashed:
    6. 0   libsystem_kernel.dylib            0x000000021ac1f104 __pthread_kill + 8
    7. 1   libsystem_pthread.dylib           0x000000021ac9b020 pthread_kill$VARIANT$mp + 380
    8. 2   libsystem_c.dylib                 0x000000021ab76d78 abort + 140
    9. 3   libsystem_malloc.dylib            0x000000021ac73768 _malloc_put + 0
    10. 4   libsystem_malloc.dylib            0x000000021ac73924 malloc_report + 64
    11. 5   libsystem_malloc.dylib            0x000000021ac662d4 free + 376
    12. 6   camerax                           0x0000000102a8cf84 0x102790000 + 3133316
    13. 7   libFontParser.dylib               0x000000021ec87de8 TParserDictionary::~TParserDictionary+ 277992 () + 60
    14. 8   AudioToolbox                      0x000000021f2c831c AudioOutputUnitStop + 28
    15. 9   camerax                           0x0000000103200678 0x102790000 + 10946168
    16. 10  camerax                           0x000000010320082c 0x102790000 + 10946604
    17. 11  camerax                           0x0000000103200374 0x102790000 + 10945396
    18. 12  Foundation                        0x000000021ba31dcc -[__NSObserver _doit:] + 316
    19. 13  CoreFoundation                    0x000000021aff794c __CFNOTIFICATIONCENTER_IS_CALLING_OUT_TO_AN_OBSERVER__ + 20
    20. 14  CoreFoundation                    0x000000021aff7918 ___CFXRegistrationPost_block_invoke + 64
    21. 15  CoreFoundation                    0x000000021aff6e08 _CFXRegistrationPost + 392
    22. 16  CoreFoundation                    0x000000021aff6ab4 ___CFXNotificationPost_block_invoke + 96
    23. 17  CoreFoundation                    0x000000021af6ec90 -[_CFXNotificationRegistrar find:object:observer:enumerator:] + 1404
    24. 18  CoreFoundation                    0x000000021aff6540 _CFXNotificationPost + 696
    25. 19  Foundation                        0x000000021ba07ca0 -[NSNotificationCenter postNotificationName:object:userInfo:] + 68
    26. 20  AVFAudio                          0x0000000221030660 -[AVAudioSession privateInterruptionWithInfo:] + 1112
    27. 21  AVFAudio                          0x00000002210288d8 (anonymous namespace)::HandlePropertyListenerCallback+ 596184 (unsigned int, objc_selector*, NSObject*) + 72
    28. 22  libdispatch.dylib                 0x000000021aac16c8 _dispatch_call_block_and_release + 24
    29. 23  libdispatch.dylib                 0x000000021aac2484 _dispatch_client_callout + 16
    30. 24  libdispatch.dylib                 0x000000021aa6e9a4 _dispatch_main_queue_callback_4CF$VARIANT$mp + 1068
    31. 25  CoreFoundation                    0x000000021b018df4 __CFRUNLOOP_IS_SERVICING_THE_MAIN_DISPATCH_QUEUE__ + 12
    32. 26  CoreFoundation                    0x000000021b013cbc __CFRunLoopRun + 1964
    33. 27  CoreFoundation                    0x000000021b0131f0 CFRunLoopRunSpecific + 436
    34. 28  GraphicsServices                  0x000000021d28c584 GSEventRunModal + 100
    35. 29  UIKitCore                         0x000000024836ad40 UIApplicationMain + 212
    36. 30  camerax                           0x0000000102797880 0x102790000 + 30848
    37. 31  libdyld.dylib                     0x000000021aad2bb4 start + 4
    seems something in AudioToolbox is conflicted
     
  23. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,533
    I'm aware of this bug and already have a fix coming. I'll publish the update in the next few days.
     
    shallwaycn likes this.
  24. niv_onehamsa

    niv_onehamsa

    Joined:
    Oct 30, 2016
    Posts:
    11
    Hi again Lanre,

    I own both NatCorder and NatMic and use them together. I noticed some weird behavior since the integration with NatMic. You seem to be override a default behavior of iOS to mute the speakers when microphone is recording (because that always sounds bad, unless headphones are used in which case no muting occurs). That was easy enough to work around on my own (since I can mute the sounds I make).

    However, it seems that after the first recording the mute button is completely ignored and sounds plays out of the speakers regardless of the phones mute state. any thoughts?
     
  25. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,533
    Actually, NatMic overrides the default behaviour to play out of the speakers always. So if the speakers are muted on the very first recording, then it's a bug.
     
  26. RadiantSilvergun

    RadiantSilvergun

    Joined:
    Dec 27, 2017
    Posts:
    11
    Hi Lanre, thank you very much for your help!
     
    Lanre likes this.
  27. niv_onehamsa

    niv_onehamsa

    Joined:
    Oct 30, 2016
    Posts:
    11
    Ok, ETA on the fix?
     
  28. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,533
    I'm not entirely clear what's going on here since I'm unable to reproduce it. Are you using any special audio plugins besides NatMic? I have a new NatMic build that should be ready tomorrow or Friday. I'll let you know when it's live.
     
  29. asyranamin

    asyranamin

    Joined:
    Nov 27, 2017
    Posts:
    3
    Hello there.
    I recently bought your NatMic API in hopes that it would fix the audio skipping/glitchyness when using native unity microphone. But upon implementation, i get the java buffer flow exception. Even when using your demo scene where you implement both natcorder and natmic.

    Natmic API works fine on itself, but when paired with Natcorder, i cant seem to get it to record both video and audio.
    Any news on the fix?
     
  30. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,533
    We'll be submitting the NatCorder update with the fix today or tomorrow.
     
  31. cpamarco001

    cpamarco001

    Joined:
    Jan 10, 2019
    Posts:
    1
    I have the same buffer overflow problem when using NatMic with NatCorder, may I know the update schedule as it is not fixed yet currently?
     
  32. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,533
    We've had more issues come up with the update, so we've been sharing the build with devs who need it. Once we tidy up, we'll push it onto the Asset Store. If you'd like to get the build, email me with your invoice number.
     
  33. unityCZH

    unityCZH

    Joined:
    Nov 12, 2018
    Posts:
    16
    Hello:
    When I combine NatCorder and NatMic, the recorded video on the IOS device will end, and the audio in video will appear discontinuous, I don't know why.Here's my code

    public void StartRecording (Action<string, Texture2D> callBack) {
    this.callBack = callBack;
    var format = Format.Default;
    NatMic.StartRecording(format, OnSampleBuffer);
    var framerate = container == Container.GIF ? 10 : 30;
    const float DownscaleFactor = 1f / 3;
    var size = rect.rect;
    var videoFormat = new VideoFormat((int)(size.width * DownscaleFactor), (int)(size.height * DownscaleFactor), framerate);
    var audioFormat = new AudioFormat(format.sampleRate, format.channelCount);
    recordingClock = new RealtimeClock();
    NatCorder.StartRecording(container, videoFormat, audioFormat, OnReplay);
    videoRecorder = CameraRecorder.Create(cameraMain, videoFormat, rect, recordingClock);
    if (container == Container.GIF)
    videoRecorder.recordEveryNthFrame = 5;
    }
    public void OnSampleBuffer(AudioEvent audioEvent, float[] sampleBuffer, long timestamp, Format format)
    {
    if (audioEvent == AudioEvent.OnSampleBuffer && NatCorder.IsRecording)
    NatCorder.CommitSamples(sampleBuffer, recordingClock.CurrentTimestamp);
    }
     
  34. unityCZH

    unityCZH

    Joined:
    Nov 12, 2018
    Posts:
    16
    Hello:
    I packaged your demo and it crashed on IOS 11.4.1.

    2019-01-17 17:34:21.707523+0800 ProductName[4344:297505] NatMic: Microphone started recording with format: 1@24000Hz
    2019-01-17 17:34:21.761576+0800 ProductName[4344:297134] NatCorder: Prepared MP4 video encoder at resolution 1536x2048@30Hz with average bitrate 5909759 and keyframe interval 3s
    2019-01-17 17:34:21.795463+0800 ProductName[4344:297134] NatCorder: Prepared MP4 audio encoder for 1 channels at 24000Hz
    2019-01-17 17:34:21.795693+0800 ProductName[4344:297134] NatCorder: MP4 recorder started recording
    -> applicationWillResignActive()
    -> applicationDidBecomeActive()
    Message from debugger: Terminated due to memory issue
     
  35. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,533
    You seem to have edited your code since you posted this. Can you confirm that what you have up is your exact code? If so, email me one of the recorded videos with the broken audio.
     
  36. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,533
    Your recording resolution is too high, so the app crashes. Your device doesn't have enough memory to support recording at that size.
     
  37. unityCZH

    unityCZH

    Joined:
    Nov 12, 2018
    Posts:
    16
    Where can I set the resolution?I can't find it.I need IPad support.Does this video recording support IPad resolution?

    Here is the code I combined:

    public void StartRecording () {
    // Start the microphone
    var microphoneFormat = Format.Default;
    NatMic.StartRecording(microphoneFormat, OnSampleBuffer);
    // Start recording
    recordingClock = new RealtimeClock();
    var audioFormat = new AudioFormat(microphoneFormat.sampleRate, microphoneFormat.channelCount);
    NatCorder.StartRecording(Container.MP4, VideoFormat.Screen, audioFormat, OnReplay);
    // Create a camera recorder for the main cam
    videoRecorder = CameraRecorder.Create(Camera.main, recordingClock);
    }
    private void OnSampleBuffer(AudioEvent audioEvent, float[] sampleBuffer, long timestamp, Format format)
    {
    // Send sample buffers directly to NatCorder for recording
    if (audioEvent == AudioEvent.OnSampleBuffer && NatCorder.IsRecording)
    NatCorder.CommitSamples(sampleBuffer, recordingClock.CurrentTimestamp);
    }
     
  38. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,533
    The VideoFormat that is passed into StartRecording. Please look through the NatCorder README.md for more info on how to use the API. The rest of your code looks good.
     
  39. unityCZH

    unityCZH

    Joined:
    Nov 12, 2018
    Posts:
    16
    I have run the demo you provided today.When I set the resolution to 1920*1080, I ran out of memory in a minute while recording video.

    Below is the error message and code
    2019-01-22 17:50:50.295185+0800 ProductName[24557:1527854] NatCorder: Prepared MP4 video encoder at resolution 1080x1920@30Hz with average bitrate 5909759 and keyframe interval 3s
    2019-01-22 17:50:50.301838+0800 ProductName[24557:1527854] NatCorder: Prepared MP4 audio encoder for 1 channels at 24000Hz
    2019-01-22 17:50:50.302085+0800 ProductName[24557:1527854] NatCorder: MP4 recorder started recording
    2019-01-22 17:50:50.313097+0800 ProductName[24557:1528199] NatMic: Microphone started recording with format: 1@24000Hz
    -> applicationWillResignActive()
    -> applicationDidBecomeActive()
    Message from debugger: Terminated due to memory issue

    public void StartRecording () {

    var microphoneFormat = Format.Default;

    NatMic.StartRecording(microphoneFormat, OnSampleBuffer);

    var width = 1080;

    var height = 1920;

    var framerate = container == Container.GIF ? 10 : 30;

    var videoFormat = new VideoFormat(width, (int)height, framerate);

    var audioFormat = new AudioFormat(microphoneFormat.sampleRate, microphoneFormat.channelCount);
 recordingClock = new RealtimeClock(); 

    NatCorder.StartRecording(container, videoFormat, audioFormat, OnReplay);

    videoRecorder = CameraRecorder.Create(Camera.main, recordingClock);


    }


    private void OnSampleBuffer(AudioEvent audioEvent, float[] sampleBuffer, long timestamp, Format format)

    {

    if (audioEvent == AudioEvent.OnSampleBuffer && NatCorder.IsRecording)
 NatCorder.CommitSamples(sampleBuffer, recordingClock.CurrentTimestamp);

    }
     
  40. unityCZH

    unityCZH

    Joined:
    Nov 12, 2018
    Posts:
    16
    When I create an empty scene in my project and record the video resolution as 1080*810, I record normally.I used the same script for other scenarios in my project, with resolutions 1080*810 and 720*410, respectively, and memory errors causing project flashbacks (I had more UI in my project).Does the plug-in consume a lot of memory to support recording video?

    I need your help.Fixed a memory error that caused flashbacks.thank you
     
  41. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,533
    Let's move this conversation to the NatCorder forum thread. Recording can consume a lot of memory. It depends on a lot of factors including how much memory the app uses when not recording and how much RAM is available on the device.
     
  42. shin_unity197

    shin_unity197

    Joined:
    Oct 30, 2017
    Posts:
    18
    Hi Lanre,

    I updated NatMic to 1.2 by deleting the previous version first. I updated my code to be up to date to the API. But NatMic doesn't work anymore anymore and I'm getting an ArgumentOutOfRangeException. Below is the snippet of code I'm using and also attached the XCode error as well below. It seems on Mac which is running the iOS platform in unity, it has no trouble recording the mic audio but on device there are errors. Is there anything specific I'm missing?
    Code (CSharp):
    1.                 var width = 480;
    2.                 var height = 640;
    3.                 var framerate = 30;
    4.                 var videoFormat = new VideoFormat(width, (int)height, framerate);
    5.            
    6.                 var device = Device.Default;
    7.                 var microphoneFormat = Format.Unity;
    8.                 AudioListener audioListener = Camera.main.GetComponent<AudioListener>();
    9.  
    10.                 m_recordingClock = new RealtimeClock();
    11.                 NatMic.StartRecording(device, audioListener, OnSampleBuffer);
    12.                 var audioFormat = m_recordMicrophoneAudio ? new AudioFormat(microphoneFormat.sampleRate, microphoneFormat.channelCount): AudioFormat.None;
    13.  
    14.  
    15.                 NatCorder.StartRecording(m_container, videoFormat, audioFormat, OnReplay);
    16.                 m_videoRecorder = CameraRecorder.Create(Camera.main, videoFormat, m_recordingClock);
     

    Attached Files:

    Last edited: Feb 5, 2019
  43. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,533
    I'm looking into this. I'll push an update with a fix tomorrow.
     
  44. Soqman

    Soqman

    Joined:
    Nov 20, 2016
    Posts:
    2
    Now there is no possibility to quote, but in 1.2 Xcode now swears that something is missing for the work of ARMV7
     
    Last edited: Feb 8, 2019
  45. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,533
    I'm looking into this. We did not drop support for armv7.
     
  46. Soqman

    Soqman

    Joined:
    Nov 20, 2016
    Posts:
    2
    ignoring file /Projects/xcode/2.2.0_a/Libraries/NatCorder/Plugins/iOS/libNatRender.a, file was built for archive which is not the architecture being linked (armv7): /Projects/xcode/2.2.0_a/Libraries/NatCorder/Plugins/iOS/libNatRender.a
    ld: warning: ignoring file /Projects/xcode/2.2.0_a/Libraries/NatMic/Plugins/iOS/libNatMic.a, file was built for archive which is not the architecture being linked (armv7): /Projects/xcode/2.2.0_a/Libraries/NatMic/Plugins/iOS/libNatMic.a
    ld: warning: object file (/Projects/xcode/2.2.0_a/Libraries/NatCorder/Plugins/iOS/libNatCorder.a(Bridge.o)) was built for newer iOS version (10.2) than being linked (10.0)
    ld: warning: object file (/Projects/xcode/2.2.0_a/Libraries/NatShare/Plugins/iOS/libNatShare.a(NatShare.o)) was built for newer iOS version (10.2) than being linked (10.0)
    ld: warning: object file (/Projects/xcode/2.2.0_a/Libraries/NatCorder/Plugins/iOS/libNatCorder.a(GIFRecorder.o)) was built for newer iOS version (10.2) than being linked (10.0)
    ld: warning: object file (/Projects/xcode/2.2.0_a/Libraries/NatCorder/Plugins/iOS/libNatCorder.a(MP4Recorder.o)) was built for newer iOS version (10.2) than being linked (10.0)
     
  47. fourthevision

    fourthevision

    Joined:
    Oct 16, 2017
    Posts:
    2
    Hello, I purchased NatMic on AssetStore and I'd like to know how to record mic and audio properly together!. I tried sereal times but recorded only slow in editor and pc....one more thing is is there way to hear my voice while on NatMic recording on pc?. Thanks.
     
    Last edited: Feb 12, 2019
  48. niv_onehamsa

    niv_onehamsa

    Joined:
    Oct 30, 2016
    Posts:
    11
    Any updates??

    I'm also pretty much stuck with this update. I tried to resolve it myself but it seems there's something seriously wrong in the process.

    The timestamp that arrive with the microphone samples is VASTLY different than the ones that come with unity listener samples (seems like a made up number altogether). Which is very weird because they both seem to come from NatCorder.Implementation.CurrentTimeStamp.

    On top of causing an an exception (because the read count is negative), when I try to sidestep this exception the recorded audio is still wrong because the microphone recording with Format.Unity (stereo) seems to get the sample count wrong somehow.

    So the entire codepath of recording microphone with the scene's listener is broken on iOS. So while I appreciate the hard work, responsiveness - waiting for so long for the NatMic update only to find it unusable and having to revert back is just a big waste of time - the opposite of what using a plugin should be like.

    Cheers
     
  49. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,533
    How do you mean that the recording is slow?
    You can use the RealtimeClip script for this because it can double as a regular AudioClip that can be played by an AudioSource. But you should note that NatMic doesn't support adaptive echo cancellation on Windows or macOS, so you might hear a feedback loop in the audio.
     
  50. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,533
    I've fixed it on my end. Just gotta do some tidying up before submitting an update.
    Yup, this is exactly the problem. The timestamp that comes from the mic itself is based on CoreAudio's sample buffers, whereas the timestamp from INatMic.CurrentTimestamp are based on mach_absolute_time. The fix I've implemented is to use mach_absolute_time instead of CoreAudio's timestamps.
    How do yo mean? The microphone isn't expected or required to provide sample buffers with the same size. The sample count varies as time goes on, so our mixing logic has to buffer the samples.