Audio How do I detect which musical note was played (by an instrument)?

Hi there!
First of all, sorry if this is a simple/stupid question, I have no musical background and I don't have any clue where to even begin with making this work =)


I have recently bought a Kalimba (also known as a thumb piano) that has 17 keys, from C4 to E6 and I was thinking of using it in a game, where you'd play a sequence of notes on it and an action would happen.

For that I would need to detect, in real time, which note was played and when.
The Kalimba I've got has a pickup microphone that I have been able to connect to a Pc and record audio from,
and I figure it should be possible to detect from that audio which note was played by checking it's frequency
(because there are apps on phones that can be used to do this, in order to tune it), but I have no idea how to do this inside Unity.

I thought about using a 3rd party program to detect the notes played and then somehow turn that output into a fake keypress that could be detected from Unity, but I don't know which programs I could use for that, and it would be a pretty janky solution for an end user to deal with..


So this is where I'm at. If any of you have had to do similar things, or if you've got any ideas as to how to make this work, then please let me know. Thank you! =D

 

Update:

So I've found a function:

Code (CSharp):

1. audioSource.GetSpectrumData(spectrum, 0, FFTWindow.BlackmanHarris)

which should allow me to work out how loud(?) a sound is at a specific frequency range,
and also how to get the input from the microphone to play inside Unity:

Code (CSharp):

1.     AudioSource audioSource;
2.  
3.     void Start() {
4.         audioSource = GetComponent<AudioSource>();
5.         audioSource.clip = null;
6.  
7.         if (!Microphone.IsRecording(null)) {
8.             audioSource.clip = Microphone.Start(null, true, 10, 44100);
9.             while (!(Microphone.GetPosition(null) > 0)) {}
10.             audioSource.Play();
11.         }
12.     }

...however, GetSpectrumData() seems to work only for an audioclip that has a finite length, not one that is constantly being updated (like the output from a microphone), so I'm at a bit of a loss here..

 

this wrong snippet find its way everywhere, including some assets on the store
- correct version
(which properly yields the startup loop (which is a (rather rare) problem with certain hardware)):

Code (CSharp):

1.     IEnumerator Start()
2.     {
3.         audioSource = GetComponent<AudioSource>();
4.         audioSource.clip = null;
5.         audioSource.loop = true;
6.  
7.         if (!Microphone.IsRecording(null))
8.         {
9.             audioSource.clip = Microphone.Start(null, true, 10, 44100);
10.             while (!(Microphone.GetPosition(null) > 0))
11.             {
12.                 yield return null;
13.             }
14.             audioSource.Play();
15.         }
16.     }
17.  

set clip to loop to get continuous input
note that by default you'll get audio feedback loop == if the output will be picked by the microphone == since the source is being played on the audio source - so you need to silence the output after it was played (otherwise you'd get no signal in e.g. Update for processing)
- either feed it with zeroes - or process accordingly - in OnAudioFilterRead, or plug audio source output into a mixer group which is silenced

 

Hi! Thanks for the help!

After finally realizing that I need to put GetSpectrumData() inside Update (which took me waaay too long), I can now get all of the loudness values from the mic with "spectrum[SampleIndex]".

'audioSource.loop = true;' worked great! I figured since the Microphone.Start() function had a bool for "loop" that it would've done the trick, but nevertheless, now it's recording indefinitely.


I do have a problem with changing the Start to an IEnumerator; it seems to be adding about 1 - 2 seconds of delay...
Without it, the delay isn't 0, but it isn't annoying either.

I figure it has to do with the waiting till microphone starts recording, so I've tried adding this:

audioSource.time = audioSource.clip.length;

above audioSource.Play(), but Unity is telling me that it's an "invalid seek position"

Is there a way to remove/shorten this delay?
If there isn't, should I just leave it as a simple void, or would that be less stable (it is working with my Pc & microphone..)

 

the yield loop waits until it's ready to record
I suppose you can shorten initial clip creation time by making it as short as possible ((very) low values might not be safe on all systems, if at all)

 

Did you ever figure this out? Trying it now!

 

Yup, I did manage to get it to work, at least in Unity 2019, not sure if the code still works for the newer versions tho

This is the code I used to get the "loudness" data from the microphone (stored in the 'spectrum' array as floats):

Code (CSharp):

1. using UnityEngine;
2. [RequireComponent(typeof(AudioSource))]
3. public class MicCode : MonoBehaviour
4. {
5.     AudioSource audioSource;
6.     public float[] spectrum = new float[4096]; //Multiple of 2; Each float represents a range of frequences, from lowest to highest; More floats means fewer frequences sharing the same float variable.
7.    
8.     void Start() {
9.         audioSource = GetComponent<AudioSource>();
10.         audioSource.clip = null;
11.         audioSource.loop = true;
12.  
13.         if (!Microphone.IsRecording(null)) {
14.             audioSource.clip = Microphone.Start(null, true, 10, 44100); //44100Hz is the microphone's frequency. You may want to change this based on what frequency your mic is running at, or have it auto-detect)
15.             while (!(Microphone.GetPosition(null) > 0)) {}
16.             audioSource.Play();
17.         }
18.     }
19.  
20. /* Never managed to get this working though, seems to just not get any data from the mic
21.     IEnumerator Start() {
22.         audioSource = GetComponent<AudioSource>();
23.         audioSource.clip = null;
24.         audioSource.loop = true;
25.  
26.         if (!Microphone.IsRecording(null)) {
27.             audioSource.clip = Microphone.Start(null, true, 10, 44100);
28.             while (!(Microphone.GetPosition(null) > 0)) { yield return null; }
29.             audioSource.Play();
30.         }
31.     }
32.     */
33.  
34.     void Update() {
35.         if (Microphone.IsRecording(null)) {
36.             //Updating data in spectrum array to the microphone's data this frame
37.             audioSource.GetSpectrumData(spectrum, 0, FFTWindow.BlackmanHarris);
38.         }
39.     }
40. }

And then by checking if spectrum[index] > threshold I would know that the mic picked up something on that frequency.

Didn't find a way to get the dealy to be shorter though, but it should be fine enough if you only want to detect the notes and don't plan to use the mic's audio in-game. Hope this helps, good luck!