MicControl 3

Hey Stebello,

By default MicControl 3 provides a "loudness" value, this is presented in the way of a single float value.
A default pitch detection is not present as it is to "abstract" to include as a "raw tool" to use. MicControl 3 is an acces tool in which users can acces microphone data in a usable way (float values), to create interaction in their projects.

However!
With MicControl's spectrumData array value, you have acces to the full frequency array (depended on your sample rate), so you could write a pitch detection based on that (a comparing if statement gets you a long way). The spectrumData value provides every single frequency detected in realtime and you can acces them individually (like the loudness value) or loop through the array and analyse it in one frame.

All this wall of text aside, there have been clients who where successful in creating "Singstar (have to be careful)" clones. Some made a very basic one simply by only using the loudness value (the single float), thus creating a singing app in which the loudness was more important than the actually pitch. Other clients have used the frequencyData to analyse per frame which frequencies where present and then did a similar comparison (frequency x and x had to be a certain strength) to detect different pitches and tones of the players.


I hope this information was useful to you.

 

Hi Mark,

Thanks for the awesome plug-in. I purchased the MC 2 right before you released the MC 3.
I am very new to Unity however I want to create a demo with multiple mic inputs in VR where a 3 persons band can play the song together. This might be something like a live performance.

I have a few questions here;

- What is the best way to create a simple dropdown menu UI interface to assign a mic device to each player? I know you can select the microphone device from the inspector. But what if I need to select the device in the build. Do you have any examples similar to this?

- Is there anyway to upgrade MC2 to MC3 ?Or I just need to buy it again

Thanks!

 

No need to buy MC3 again as the package is uploaded as an overwrite on MC2, meaning you already have acces to it in your download section.

Make sure you completely delete MC2 in your project before installing MicControl3. They will otherwise conflict.

A drop down can be created in the following manner:

You will need to loop through all your devices (They are provided by your system in an array and position number. However Unity's api uses the full device name, which can also be received).


I have written an example script. Because I suspect more people want to know how to set up a basic dropdown menu during run-time, I have added the script in the reference folder for in a future update. This was a good and usefull question!

I hope this answers your question.

cheers,



In the meantime you can find the code below on how to approach a basic setup. Just paste it in a C# script and place that script on a dropdown menu item in your canvas:

Code (CSharp):

1. using System.Collections;
2. using System.Collections.Generic;
3. using UnityEngine;
4.  
5. using UnityEngine.UI;
6.  
7. public class UI_devices_dropdown_list : MonoBehaviour {
8.  
9.     //Place this script on a dropdown menu in the Canvas
10.     //The script will place every detected device as an item in the list.
11.     //Then we will set the selected device as an input in MicControl 3.
12.     //These variables are used by MicControl 3 " public int InputDevice, String selectedDevice;"
13.     //InputDevice is the position the devices has on your system's device array and MicControl3 will use that position to read out the name in selectedDevice.
14.     //selectedDevice is then used by MicControl to setup unity's microphone class to call the proper device.
15.  
16.     //In this script we will only be setting the InputDevice variable.
17.  
18.     //One more thing to know: the device on Position 0 is always the system's default device. So the first dropdown item we will always see, will be the default device.
19.  
20.     //in here we place the controller that we want to call and control.
21.     public MicControlC getController;
22.  
23.     //here we store the detected devices
24.     List <string> getDeviceList= new List <string>();
25.  
26.  
27.  
28.     void Start () {
29.  
30.         //clear the previous list
31.         transform.GetComponent<Dropdown> ().ClearOptions();
32.  
33.  
34.         //loop through all detected devices on the system and add them in the list
35.         foreach(string device in Microphone.devices){
36.                 getDeviceList.Add (device);
37.  
38.         }
39.  
40.  
41.         //place the filled list as clickable options.
42.         transform.GetComponent<Dropdown> ().AddOptions ( getDeviceList);
43.          
44.     }
45.  
46.  
47.  
48.     //now that we have a drop down list, we can use Dropdown.value to set the MicController's InputDeivce variable.
49.  
50.     void Update(){
51.  
52.         //to prevent setting it every frame, we only update the value when the selection has changed in the dropdown menu.
53.  
54.         if (transform.GetComponent<Dropdown> ().value != getController.InputDevice) {
55.  
56.             //before we do anything to the device, we must first stop the current connection
57.             getController.StopMicrophone();
58.  
59.             //change the values you want to change here. In this case we update the input device.
60.             getController.InputDevice = transform.GetComponent<Dropdown> ().value;
61.  
62.             //next we re-initialise and start the new connected device.
63.             getController.InitMic();
64.  
65.             //dubbel check in your controller's inspecter during runtime wether or not the new connection was a succes.
66.         }
67.  
68.     }
69.  
70.  
71. }
72.  

 

Hey Mark,

Appreciate your help, awesome support for the assets and very well organised documentation. The dropdown code is working great. I will be spending a next few days building the live performance demo.

I would just ike to say thanks and I am sure there will be more questions for ya on the way.
THANKS!

Tu

 

How easy is it to record the mic input?

For the idea I'm thinking of using this for, it wouldn't necessarily have to write to file, but keep it memory for a short while. I'm thinking of adding a small delay and passing it through a real-time audio filter to modulate it to playback in game.

 

Hi, although it is possible for MicControl to playback audio, MicControl is designed to provide a realtime data stream (audio is converted to floats, arrays,etc..) for the user to utilize in code. It is a big wrapper around Unity's Microphone class, making things much simpler to setup for users.

The ability to playback audio in realtime is more of a side effect because an AaudioSource is used to sample the audio coming from the microphone device. I would not recommend using MicControl for any "in game-communication" system.
But if the goals is to playback the player's interaction (hearing his own voice in-game), then this will be perfectly possible.

Another thing which is perfectly possible is to add Unity's build in Audio filters on the controller or even have it manipulated by reverb zones in your scene. Since everything is passed through the audioSource you can deform your voice on the fly. You can even set the controller's AudioSource to 3D space and move the controller around to have the player's voice displace in 3D space (note that the converted audio data in float form is always treated without 3D manipulation).

That said, since the audio clip will be present in the AudioSource (the clip is generated on the fly and written to continuously). One could catch that clip in another AudioClip variable and write it down to a file. But the user will have to write that himself and is not supported out of the box by MicControl.


I know it is allot of text, but hopefully you find some useful information in it.

greetings,

 

do you have any recommendations for background noise filtering? I'd like to dampen the crowd noise a bit if the app was running in a public place

 

Not really, background noise is a whole beast of its own. The only thing you could do is comparing your spectrumData against another array of values (this value would be containing spectrumdata of the background noise) and only allow values that are not or close to the comparison array.

It is a hefty job. You could catch a "snapshot" of the bgn by saving the spectrumData with your own script (while the player is not using the device).

But unfortunately there is not a clean cut way of doing it directly. You could use a less sensitive device, but that might interrupt your setup.

 

ok great, yeah that's what I was thinking.. having a spectrumData array that slowly moves to the average of a chunk of frames or something. it's a tough one

thanks!

 

Hi,

What is the highest sampling rate you support? I'm trying to use the plugin to sample frequencies between 17Khz and 20Khz. Will 44100 work well for this?

Thanks!

 

44100hz should provide enough data to read out samples of 17 and 20khz, however precision will vary depending on the device used.

The way MicControl provides its data is by filling up data with the "Sample amount". For instance, If you have set the controller to 44100hz and the Sample amount to 1024. Then MicControll will take 1024 samples across the available 44100hz of data. Thus the higher the Sample amount the more precise the .loudness value will be.

When you enable Spectrum data analysis, every single one of these samples will be available as separate float values within an array.

Thus should your device use 48000hz even more data will be available. (Remember that you have to set your device on your OS to this value otherwise you will not profit from the extra data).

 

Hi,

I am interested to buy this plugin.
However, Can this plugin support below scenario :
"record user's speech then change it's pitch, then playback, similiar to 'talking tomcat' "

please let me know...thanks...

 

Hi,

In short, no MicControl does not support recording of audio out of the box. Its main purpose is providing a real-time data stream of the current audio input form a microphone device.

However, since the audio is saved through an audiosource. One could catch this clip during gameplay, duplicate it and add filtering to it themselves. But this has to be "modded in" by the user himself, and I suspect you want a more "out of the box solution". Which MicControl cannot provide.

Hopefully this answers your question.

greetings

 

What is the best way to use this in an on/off fashion with a Unity object? For example, when something collides with a sphere (on/off switch), the mic script is enabled and when another collision occurs, the script is disabled? I don't want the overhead of the mic when it is not needed, and I also want its functionality to be controllable.

Should I simply enable/disable miccontroller object from within a script attached to the on/off button? That seems a little too brute force.

 

Enabling/Disabling the controller will most likely crash the connection and lock up your device. MicControl has build in functionality to enable and disable the device connection.

These are the most important variables when you are manually stopping and starting connections.
copy pasted from the Documentation file in the MicControl folder:

- public bool Initialized; The initialized bool is handled automatically by the system and is used by users to check if the controller is ready doing its basic setup and has started reading data. This value is very important when using spectrumData as you only want your code to read the array when it is actually filled with data. Normally one should treat this value as READ ONLY, however, if you have stopped the microphone manually with StopMicrophone (), this value will be automatically set to false. (see chapter 3).
​

- public void InitMic(): Used by the system to initialize the selected device with the set controller settings. Calling this function from an external script will re-initialize the device with the new set settings. Do not forget to first stop the device before calling this method.

- public IEnumerator StartMicrophone(): Called at the end of InitMic() to fully start the device and receive data. This can be called manually, however this is not a good idea as your device still has to be initialized. When changing settings through script, it is best to always call InitMic() instead, as this will automatically end up calling StartMicrophone() anyway.

- public void StopMicrophone (): Stops the data stream from your device and stops reading from the AudioSource. It also resets Initialized and recording to false. Meaning you will have to call InitMic() to restart your device.

I have underlined the two functions you are interested in. Calling YourControllerVariable.StopMicrophone() will stop the device completely and all connections and data streams are stopped.

To properly restart your device you have to call YourControllerVariable.InitMic(). This will initialize the device again and automatically call the StartMicrophone() function to start the stream again.


So if you want to completely disable the GameObject or script, first call StopMicrophone and then disable the script.
Disabling the script can be smart if you want to prevent MicControl from restarting the connection if you went outside of a window (re-focusing on your game window can trigger MicControl's anti lag function and start the device).

Then if you re-enable your GameObject or script, simply call InitMic() and the stream is back live.

Hopefully this answers your question.

greetings,
Mark

 

Hi, I downloaded the tech demo, but without success. The macOS app isn't working (on macOS Sierra at least), and the iOS and Android versions are missing. There is an empty Android folder, and no iOS folder at all.

I would also like to know if this Unity package supports live streaming from the microphone, i.e. without the 1s delay you get if you just use Microphone.Start().

 

Hi waldgeist,

First the dropbox issue: All files should be present and working. I am looking into the dropbox issue to see if anything went wrong with the files.

On your second quetion: MicControl is a wrapper around Unity's microphone class. This includes Microphone.Start(). However, MicControl does not suffer from the one second delay and all streams are (near) realtime. Live streaming from the microphone (data that is) is what MicControl was created for in the first place. MicControl is NOT however a voip communication system. It is possible to project ones voice in the scene, this however more of a side effect rather than a feature, because an audioSource is used to sample the stream. By default the users voice is muted and only the audio in data form is available (floats and arrays).

Hopefully this makes some sense.

If you want more information or require a specific build for X platform, please contact me http://markduisters.com/contact/
This way I can properly assist you with further questions and or demo assistance.

greetings,
Mark

edit: ps. be sure to unpak the files with 7zip. 7zip is used due to higher compression rates on file size.

 

Hi,
I bought this asset recently and cannot select the correct microphone. I wanted to use a bluetooth headphone (Bose QC 35) and I have other microphones connected to the computer. However, whenever I select the bose microphone or any other it always defaults to the same microphone, even in the provided samples. The device name changes but the microphone being used does not.
Is it related to this Unity Issue?
https://issuetracker.unity3d.com/issues/microphone-dot-start-is-not-recording-the-audio-from-selected-recording-device
Is there any workaround?

 

Hi,

In most user cases there are one of two reasons this happens:

1. Have you used GameObject-Audio-MicControl to place a controller? Dragging the scripts onto objects yourself might miss some part of the setup and thus failing to connect to the devices, defaulting to your systems default.

2. Double check your device's real name in your system. If there are non roman symbols in it (for instance mandarin signs), Unity will not be able to connect to the devices as it can not process those symbols. (I do not know if this is caused by the system or Unity itself, but experiments have shown me that non roman signs in the device name give these issues).

I had a user yesterday with similar problems, he fixed it by following option 1. Let me know how it goes!

 

Hi again Mark, thanks for your prompt reply.

Number 1 did not work for me. It seems that when I try to switch device now I start getting a null pointer exception in line 364 in miccontrolc.cs. But if I choose the default microphone, it works and the exception does not happen anymore.

Oh and BTW, It seems that in the latest beta, number 2 is solved:
https://issuetracker.unity3d.com/is...icrophone-devices-on-specific-computer-builds

Let me know if you have any other ideas of what is my problem. Could it be the beta version of Unity? Have you tried the latest one?

Thanks,
André

 

Nullpointer from devices only appear if the device is not "there". For instance you have selected the device because MicControl can see it in the system list, but if that device has an issue and does not send data or the device is already occupied (another software is reading/recording from it), then MicControl will receive no device to read from once it tries to do so, resulting in the nullpointer.

I am running Unity 2017.2 and have had no issues. Other clients who did have connection issues had either the non roman symbol issue or the occupied device issue.

I have not tried the beta, so it "could be" that the beta is the issue. It is a beta after all. However, if I have some time this evening, I will experiment on the beta (2017.3).

In the mean time, if you mail me some more detailed information about your system (OS, input device list/names, etc), I might be able to assist you a bit more in what might be wrong.

http://markduisters.com/contact/

I check my email a bit more frequently than the forums.

Greetings,
Mark

 

Hi,

Is it possible to get a length of time above a certain volume? I am trying to determine the time of a sustained volume across Update() but the values I'm getting back keeping getting zero values during the voice input.

It seems to consistently return a 0 value every 0.22 seconds.

 

Hi rd,

Could you give me some more information? It is hard to pinpoint the exact reason a 0 value is given. This could be caused by several things: Your device has some wrong inputs, software might be interrupting your device and lastly, there might be a bug that I am not yet aware off.

If you contact me through mail I will most likely respond faster, : http://markduisters.com/contact/

greetings,
Mark

 

Hi,

MicControl3 provides no actual audio, but automatically converts the input to a useble float value. This is called "loudness" within MicControl3 and is an average representation of how hard some one is talking/yelling/breathing into the device.

MicControl3 also provides the user with a float array, in which the spectrum data of that device is stored. With this you could make array comparisons against your own array to determine more precisely what the input is.

Users have made things similar to singstar before. If simply pitch dection is what you want, then you should have enough with the .loudness variable. It is up to you however to write that detection code. As said MicControl3 only provides a value, it is a tool.

For instance you could detect 3 different ranges of input. Low, medium and high with 3 if statements or with a switch.

here is a very rough example.

if(MicControl.loudness >lowCompareFloat){
do your stuff;
}


greetings,
Mark

 

Hi,

how can you detect if someone is exhaling (blowing in the microphone)?
At the moment I use a calibration step, where the user needs to exhale 3 times. From this I extract the
average minimum and maximum loudness value.
I then set these min and max values as triggers for a certain action.
But turns out blowing falls in the same range as normal talking.
Is there any other way to only trigger an action when someone is blowing?

 

Doing a callibration is indeed the best way, since everything is converted to a float value, there is no difference between blowing and yelling into the microphone.

You could use the spectrum data and analyse the array to specifically detect the difference between "speech" and blowing, but that would result in a more precise approach as using the loudness value, in which case if you only need blowing input, unnecessary work as loudness will do just fine..

 

Hi Mark, I bought MicControl3 recently and I'm trying to setup a stereo input environment with 2 microphones, 1 for the left and another 1 for the right channel using Scarlett 18i8, however, it appears that MicControl3 reacts only to the left channel and all the sounds captured from the right microphone are ignored, is there any way to fix this issue? Thanks.

 

Hi,

Are the microphones connected as one device through an interface? Or are they two individual inputs?

if they are connected through an interface, then Unity itself will only detect it as one device.

If they are two unique inputs (two usb, two analog, etc...) then you need one MicController in your scene for each device.


Sincerely,
Mark

 

They are connected as 1 device through USB connection, but the sound from right channel isn't working.

 

As far as I am aware about Unity its microphone handling (which MicControl also uses), all devices are treated as a mono device. So if you have a stereo mic, only one channel will be used.

The only way for a split input, is to have two separate connections.

 

Hi,
I wanted to know if I can use this plugin for breathing tracking. I mean can i detect breath in breath out accurately.

 

Hi,

Although you can detect "Breath", if the breath in and out have the same strength/loudness,then there will be now way to differentiate which one of the two is happening, since all input audio gets represented in float form.

 

Yeah really difficult to differentiate breath in and out. Also cheating can be an issue
Anyway thank you for your help.

 

Hi MarkD,

I'm interested in building a Unity app which requires real-time audio input. Do you have a rough estimation of the roundtrip latency when using MicControl 3?

Thanks!

 

Hi,

The latency is "near real-time". But it depends mostly on the processor and how heavy your app is on the CPU.

On mobile for instance. If your app's graphics and logic eats up 99% of the CPU, then it might happen that MicControl gets a slight latency hit.

Normally as long as the cpu has no heavy overheads and is on (pc) standalone platforms, there will be none to very rarely any latency issues.

The Mobile platforms are mostly the only platforms that might come up with latency issues, due to the reasons mentioned above.


Greetings,
Mark

 

Hey I need some information

How can I differentiate between normal loud speech and Blow into the microphone. I only want to detect the Blow, not when user is speaking or some noise is near phone.

 

Hi,

differentiating is quite hard with the default .loudness value, you could write an "if statement" and set some kind of threshold (blows are a lot louder than speech or noise).

If you really want to be able to differentiate you have to use the .spectrumdata array and write some sort of comparison that can recognize the current values in the array as "not speech and not noise". This however is something the user has to do by himself.

Hope this was of some help

sincerely,
Mark

 

Hello! I try to use your plugin to play sounds from the real world through the microphone in the game scene, BUT for some reason at the very start of the game scene, the sound from the microphone comes in the form of an echo, one word is repeated about 10 times with loudness damping. If you turn on the volume of multimedia to the maximum, then there is just a loud noise. I use in Units 2018.3.5 for the Androyd platform
Help me please!

 

Hi,

Could it be that your Android device is giving audio feedback? Try using earplugs/headphones and try to listen if the echo still occurs. If no echo is present, then the Audio is getting feedback from the device's hardware itself. Similar to holding a singers microphone to the speakers.

Or is the Audio in the engine itself looping? I made a quick build for Android and ran it on my OnePlus one with earplugs, I did not encounter any echo.

If hardware feedback is not the issue, could you perhaps make a recording of the echo happening? That would give me more insight in what the issue might be.

Sincerely,
Mark

 

True, with the use of headphones such a problem is not observed. My script records video from a game camera and also needs to record sound from a microphone in real time. I use Microphone.Start. I output the sound on the Audio Source, but because the sound is immediately output to the speakers, then this option does not suit me. Perhaps there is an opportunity to force the application to perceive the sound from the audio resource (microphone), but at the same time this sound is not output to the speakers of the smartphone?

 

There is, with MicControl you can select mute device, which will mute the audio but keep the data stream open (by using its own mixer), so if you save said audio clip to a file, you should have an audible audio file.

If you are using your own setup, simply apply a mixergroup to the AudioSource and lower its volume, the audiosource will output audio, but it will not be audible through the speakers. This works because the mixer is applied after the audio source itself before going to your speakers.

 

Understood! My problem was that I recorded the data from Audio Listener, but as soon as I activated the recording from the audio source, immediately everything started to work as required)

 

Well, if you "record" from the listener, then it basically catches the audio when it comes out of the mixer, in between the mixer and the speakers. So if the mixer's volume is down, no audio will be present. It would be better to catch the data from the audioSource's audio clip. As this volume is determined before the mixer "mixes it down". And thus the audio will be un-muted.

 

Ah darn, it freezes when I use my AKAI EIE audio interface, but not if I use my Logitech C920's microphone.

Might have something to do with professional audio interfaces not being made to the highest compatibility standards

 

Hi
MicControl can only detect the devices Unity lists. And in turn Unity gets these devices from the OS. If the device is listed as a recording device in your OS and Unity can detect it, then there should be no trouble reading the data from it.

However there are known issues that Unity can't detect or read from devices which have non roman characters in their device name(such as mandarin) and indeed there are cases where some advanced professional audio interfaces also do not show up in Unity.

I wish we had more control over this, but all assets and applications can only receive data of devices that Unity can detect.

 

Hi Mark, are you still supporting this asset? I downloaded the non-commercial version to check it out, but every time I try to start either of the example scenes in either unity 2020.1 or 2019.4, the editor freezes

 

Hi

Yes the asset is still supported. Probably the scenes are not compatible with the unity build. The asset itself should still work.

Thanks for the reporting the issue, I will have a look at it.

 

Hi MarkD,

I have a quite limited programming knowledge but can read some code still, will I be able to use your asset fine using examples and tweaking them? I will use it to control transforms, materials properties and scale of gameobjects mainly.

Thanks!

 

MicControl provides input translated to usable data (mostly floats). Some minor programming knowledge is needed as the Controller only provides data. However, things like controlling transforms, materials and similar are fairly easy. There are some example scripts provided that show some approaches on how to manipulate things. You can however go to the my website :https://markduisters.com/project/miccontrol/ and download the non commercial version. It is identical to the commercial one, so it should give you enough playroom to test whether or not this asset fits your needs.

 

I was just wondering what the loudness values can be (minimum and maximum) as a float. I'm trying to do a simple lipsync based on loudness (any other pointers/links on using the spectrum analysis would be useful if you have any ideas!) and so far have got values such as 7.507406E-06?!...but mostly from 0 to about 3.9f (for a yell) maybe? So looking to have Idle, 'normal' talking loudness, and 'loud' talking values....at the moment I'm running with > 2f is loud, and > 1f is normal and anything else is 'Idle'.

Is this unit across the board, or does the sensitivity/sensitivity range change the values available for example? ANy help would be appreciated, or links, if this has been written elsewhere, would also work! Liking the asset so far, lots of things to play with.