Search Unity

  1. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

How to Record In-game audio and microphone audio together?

Discussion in 'General Discussion' started by reysantana, Mar 4, 2021.

  1. reysantana

    reysantana

    Joined:
    Dec 13, 2019
    Posts:
    8
    Hi!

    I have a video playing in unity and I enabled the microphone in unity.
    I need to record both video and microphone audio together.
    I can't find a way to record both video's audio and microphone's audio together.

    I am using AVPro video for playing video and I am using Microphone. Start to use the microphone.
    I tried it with NatCorder and unmuted microphone audio. And recorded AudioListener of the scene.
    But when I open the recorded file, I can hear microphone audio only but microphone audio was not accurate(maybe lagging) or no audio at all
    It looks like Audio Listener can't listen AVPro Video's audio playback.
    If someone has a solution, please let me know
    Thanks.
     
    ROBYER1 likes this.
  2. ROBYER1

    ROBYER1

    Joined:
    Oct 9, 2015
    Posts:
    1,449
    Stuck with this same issue here, did you find a way to do this?
     
  3. neginfinity

    neginfinity

    Joined:
    Jan 27, 2013
    Posts:
    13,497
    One way to do it is to mix Microphone and Game sounds by hand while recording.

    To record in-game sounds, attach a monobehavior with "OnAudioFilterRead()" to the audio source you want to record. Supposedly it also should work on AudioListeners, though I haven't tried t hat.

    Then mix the sound you receive with the microphone sound and feed it into recorder.

    Be aware that audio programming in unity is not the most trivial thing to do, as it is done asynchronously in a different thread.

    One other problem is that unity records microphone audio to an audio clip, which is idiotic.
    But that may be solveable with a 3rd party asset.
     
  4. EternalAmbiguity

    EternalAmbiguity

    Joined:
    Dec 27, 2014
    Posts:
    3,144
    Can you pull the audio signal from the video? If so you can mix them. Beware, you'll need to make sure you have the same sample rates and you handle mono or stereo appropriately.

    Why is this a problem? You can get the audio with GetData.
     
    lmbarns and ROBYER1 like this.
  5. neginfinity

    neginfinity

    Joined:
    Jan 27, 2013
    Posts:
    13,497
    You will not be able to record continuously, and periodically you'll need to refresh the clip, which will interrupt the stream and likely create clicks. It will also create issues trying to sync things.
    GetData also does not give you precise control over which samples you want to pull.

    A better option would be to have an equivalent for OnAudioFilterRead for pulling mic data. At least as an alternative option. Or some stream-like interface with an internal buffer.
     
    EternalAmbiguity and ROBYER1 like this.
  6. ROBYER1

    ROBYER1

    Joined:
    Oct 9, 2015
    Posts:
    1,449
    It doesn't have to be a perfect match, this is in an app that is a video chat call where we simply want to record what is on screen + both the local user mic and the incoming video call audio (with Agora).
    It seems way harder than I imagined to record that all together within Unity - we are using Natrecorder to record the video and currently one input which in our case is the local microphone which sort of picks up a bit of the in local audio feed but not enough.

    It doesn't have to be perfect we just want to combine the local microphone audio stream with the local unity game sound or another audio stream which can be recorded with the local video feed
     
  7. EternalAmbiguity

    EternalAmbiguity

    Joined:
    Dec 27, 2014
    Posts:
    3,144
    You might consider NAudio. https://github.com/naudio/NAudio/blob/master/Docs/WasapiLoopbackCapture.md

    I've never tried it from within Unity but otherwise, with loopback you should be able to get almost everything, and you can also record audio from the mic and store/merge it.
     
  8. neginfinity

    neginfinity

    Joined:
    Jan 27, 2013
    Posts:
    13,497
    Normally, a video code would have some sort of input you can feed framebuffer and audio data in. So, instead of feeding scene sound, you could mix your own and feed it in.
    That would require hacking natrecorder, if possible, or writing your own solution. For example, likely it shouldn't be too difficult to code a theora screen/audio grabber, although the video format is very rare these days. Obviously those would require programming skills.

    Another option is to talk to NatRecorder's developer and see if it has the functionality you want.
     
  9. EternalAmbiguity

    EternalAmbiguity

    Joined:
    Dec 27, 2014
    Posts:
    3,144
    I just got this working to record both loopback and the mic:

    Code (csharp):
    1.  
    2. using NAudio.CoreAudioApi;
    3. using NAudio.Wave;
    4. using System;
    5. using System.Collections.Generic;
    6. using System.IO;
    7. using System.Linq;
    8. using System.Threading;
    9.  
    10. namespace LoopbackWithMic
    11. {
    12.     class Program
    13.     {
    14.         static void Main(string[] args)
    15.         {
    16.  
    17.             var outputFolder = Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.Desktop), "NAudio");
    18.             Directory.CreateDirectory(outputFolder);
    19.             var outputFilePath = Path.Combine(outputFolder, "recorded.wav");
    20.             var enumerator = new MMDeviceEnumerator();
    21.             var output = enumerator.GetDefaultAudioEndpoint(DataFlow.Render, Role.Console);
    22.             var output_false = enumerator.EnumerateAudioEndPoints(DataFlow.Render, DeviceState.Active).Where(x => x != output).FirstOrDefault();
    23.             var input = enumerator.GetDefaultAudioEndpoint(DataFlow.Capture, Role.Console);
    24.             var capture_a = new WasapiLoopbackCapture(output);
    25.             var capture_b = new WasapiCapture(input);
    26.             List<IWaveProvider> pp = new List<IWaveProvider>();
    27.             var provider_a = new BufferedWaveProvider(capture_a.WaveFormat);
    28.             if(capture_a.WaveFormat.Channels > 1)
    29.             {
    30.                 var mon = new NAudio.Wave.SampleProviders.StereoToMonoSampleProvider(provider_a.ToSampleProvider());
    31.                 pp.Add(mon.ToWaveProvider());
    32.             }
    33.             else
    34.             {
    35.                 pp.Add(provider_a);
    36.             }
    37.             var provider_b = new BufferedWaveProvider(capture_b.WaveFormat);
    38.             if (capture_b.WaveFormat.Channels > 1)
    39.             {
    40.                 var mon = new NAudio.Wave.SampleProviders.StereoToMonoSampleProvider(provider_b.ToSampleProvider());
    41.                 pp.Add(mon.ToWaveProvider());
    42.             }
    43.             else
    44.             {
    45.                 pp.Add(provider_b);
    46.             }
    47.             MixingWaveProvider32 mix = new MixingWaveProvider32(pp);
    48.             WaveRecorder record = new WaveRecorder(mix, outputFilePath);
    49.             var speaker = new WasapiOut(output_false, AudioClientShareMode.Shared, true, 15);
    50.             speaker.Init(record);
    51.             List<byte> bytes = new List<byte>();
    52.             capture_a.DataAvailable += (s, a) =>
    53.             {
    54.                 provider_a.AddSamples(a.Buffer, 0, a.BytesRecorded);
    55.             };
    56.             capture_b.DataAvailable += (s, a) =>
    57.             {
    58.                 provider_b.AddSamples(a.Buffer, 0, a.BytesRecorded);
    59.             };
    60.  
    61.             capture_a.StartRecording();
    62.             capture_b.StartRecording();
    63.             speaker.Play();
    64.             DateTime now = DateTime.Now;
    65.             while ((DateTime.Now - now).TotalSeconds < 15)
    66.             {
    67.                 Thread.Sleep(50);
    68.             }
    69.             speaker.Stop();
    70.             capture_a.StopRecording();
    71.             capture_b.StopRecording();
    72.             record.Dispose();
    73.             record = null;
    74.         }
    75.     }
    76. }
    77.  
    This would ONLY work in situations where you have an active output that doesn't go to your regular speakers, so maybe not what you want (I didn't see anything in NAudio to create a virtual output).

    Edit - found this (https://vb-audio.com/Cable/index.htm), which you might be able to use.
     
  10. ROBYER1

    ROBYER1

    Joined:
    Oct 9, 2015
    Posts:
    1,449
    This looks promising but is it for Windows or desktop only? I forgot to mention our application is on IOS/Android as well as Desktop (windows currently and soon Mac)

    Edit: looks like it definitely is Windows API limited
     
    Last edited: Jun 22, 2021
  11. ROBYER1

    ROBYER1

    Joined:
    Oct 9, 2015
    Posts:
    1,449
    I'm in talks with the Natcorder dev about this, they had a (since removed) piece called the MixerDevice which allowed capture of multiple inputs mixed together ie. game and mic sound. https://forum.unity.com/threads/natcorder-video-recording-api.505146/page-77#post-7257713

    I'm testing out the MixerDevice today and trying to identify an IOS crash we had with it when I last tried it.

    The creator of next Gen Recorder who also seems to work at Unity had his own extra tool (which all the links for are dead) which records directly the audio going through the audio mixer while combining it with the microphone audio (which is unheard by the local user). I have also reached out to that developer as they claimed the links were still live so I'll try a VPN or something silly to grab it but I think they removed it from the web for some reason and forgot.
    https://forum.unity.com/threads/next-gen-recorder-news-support.841891/#post-6126152
     
  12. Ryiah

    Ryiah

    Joined:
    Oct 11, 2012
    Posts:
    20,757
    A quick search for realtime playback of the microphone came up with the following link.

    https://support.unity.com/hc/en-us/...-to-playback-a-Microphone-input-in-real-time-

    Which has this code that surprisingly works and without that much latency either.

    Code (csharp):
    1. using UnityEngine;
    2.  
    3. public class MicrophoneHandler : MonoBehaviour
    4. {
    5.     void Start()
    6.     {
    7.         AudioSource source = GetComponent<AudioSource>();
    8.         source.clip = Microphone.Start(null, true, 1, 44100);
    9.         source.loop = true;
    10.         while (!(Microphone.GetPosition(null) > 0)) {}
    11.         source.Play();
    12.     }
    13. }
     
    Last edited: Jun 22, 2021
  13. ROBYER1

    ROBYER1

    Joined:
    Oct 9, 2015
    Posts:
    1,449
    Using that here too it was quite simple code to figure out, just about to test if natcorder's mixingdevice will work okay for recording on IOS as when I last tried to use it, it had a crash bug
     
  14. ROBYER1

    ROBYER1

    Joined:
    Oct 9, 2015
    Posts:
    1,449
    I am pleased to announce I cracked how to do it, it was actually very simple in the end. Make sure you are using Natcorder, use an older version that still has the MixerDevice.cs, make sure to incorporate the fix from here if you are on IOS: https://forum.unity.com/threads/natcorder-video-recording-api.505146/page-69#post-6493429

    [Additional fix]
    Further searching fixed an issue with iOS. https://forum.unity.com/threads/natdevice-media-device-api.374690/page-50
    Destination array was not long enough. Check destIndex and length, and the array's lower bounds
    Edit MixerDevice.cs to increase buffer size .
    var copyBuffer = new float[16384];

    Then use the same code as them (see link above or use below)

    Code (CSharp):
    1. var query = new MediaDeviceQuery(MediaDeviceQuery.Criteria.AudioDevice);
    2.         var microphone = query.currentDevice as IAudioDevice;
    3.         audioDevice = new MixerDevice(microphone, Camera.main.gameObject.GetComponent<AudioListener>());
    4.         Debug.Log("SampleRate: Device: "+audioDevice.sampleRate);
    5.         Debug.Log("SampleRate: AudioSettings: "+AudioSettings.outputSampleRate);
    6.         // Recorder setyop
    7.         var clock = new RealtimeClock();
    8.         recorder = new MP4Recorder((int) prod.x, (int) prod.y, 30, AudioSettings.outputSampleRate, 2);
    9.         cameraInput = new CameraInput(recorder, clock, Camera.main);
    10.         /// Game + Mic Audio
    11.         //_audioInput = new AudioInput(recorder, clock, Camera.main.gameObject.GetComponent<AudioListener>());
    12.         audioDevice.StartRunning((sampleBuffer, timestamp) => recorder.CommitSamples(sampleBuffer, clock.timestamp));
    Now you are able to record the screen, the device microphone and any game audio received by the audiolistener in your scene!
     
    Rose932 likes this.
  15. neginfinity

    neginfinity

    Joined:
    Jan 27, 2013
    Posts:
    13,497
    One thing:
    This is actually going to block the main thread until some data has been written by the microphone.

    For example, OnAudioFilterRead is called at roughly 50fps every time, here something similar could happen.
     
    Rose932 likes this.
  16. ROBYER1

    ROBYER1

    Joined:
    Oct 9, 2015
    Posts:
    1,449
    Just wanted to add this only works in some cases, in some cases we would still get an error with recording where the copybuffer in MixerDevice is too slow to fill or overflowing which I am looking into. Absolute headache.
     
    Rose932 likes this.
  17. ROBYER1

    ROBYER1

    Joined:
    Oct 9, 2015
    Posts:
    1,449
    I am working with the Natcorder dev on an audio mixed recording solution which I have nearly cracked now - I have both the game mic and game sound mixing together successfully just need to sort some stuff around saving it properly with Natcam which hopefully the dev will look into soon.

    I will post the fixed solution back here and on the Natcorder main thread when it's done
     
    Rose932 likes this.
  18. Rose932

    Rose932

    Joined:
    May 22, 2021
    Posts:
    5

    Hi, i hate to intrude, but would this work on Android & MacOS devices or is it windows specific?
     
  19. qazmlp12212

    qazmlp12212

    Joined:
    Nov 1, 2021
    Posts:
    1
    I don't know. But if you want to record screen on PC, you can use Joyoshare VidiKit or OBS. They are very useful fo you to record all activities.
     
  20. EricBeets

    EricBeets

    Joined:
    Aug 30, 2017
    Posts:
    9
    Hi Robyer1,

    Do you know where I can find the older NatCorder/NatDevice version ? I saw NatDevice removed `MixerDevice` class in the 1.1.0 version (I guess 1.0.2 should be ok, but I don't know where to get it ..).

    Thanks in advance ..
     
  21. ROBYER1

    ROBYER1

    Joined:
    Oct 9, 2015
    Posts:
    1,449
    Should be in your asset store where you purchased it you could choose older versions, if that is removed or unavailable I am not sure how to help. I don't have the files any more but you can contact the developer here or on Discord to ask him.
     
    EricBeets likes this.
  22. WeiWuDe

    WeiWuDe

    Joined:
    Nov 3, 2020
    Posts:
    26
    Hi,
    stupid question, how can I use NAudio in Unity, I did a lot research but still don't get it.
    Thanks in advance and best regards
     
  23. EternalAmbiguity

    EternalAmbiguity

    Joined:
    Dec 27, 2014
    Posts:
    3,144
    Unfortunately I wasn't able to get NAudio loopback capture (recording audio output) working. I took a brief look around and found stuff for recording Unity in-game audio. That could be paired with Unity's existing microphone recording tools to do the two things mentioned in this thread's title.
     
  24. FreedLOW

    FreedLOW

    Joined:
    Dec 7, 2019
    Posts:
    15
    Hello, do you have any success with that?
     
  25. Senyamnesi

    Senyamnesi

    Joined:
    Oct 9, 2023
    Posts:
    1
    I've encountered a similar challenge when recording in Unity with a microphone and in-game audio. One workaround I've used is to export the microphone audio separately and then combine it with the in-game audio in post-production using video editing software like Adobe Premiere or Audacity for audio. It gives you more control over the final result. Also, it's better to use high-quality micro from this guide https://asmrmicrophones.com/. I hope this helps!
     
    Last edited: Oct 11, 2023
  26. Antypodish

    Antypodish

    Joined:
    Apr 29, 2014
    Posts:
    10,732
    OBS is capable to record various of channels independny and combined, as well as video.