Making sense of Spectrum Data

I'm working on a "simple" audio visualiser. My program successfully gets the spectrum data from my music but I'm at a loss as to how I would seperate the different "parts" (bass, etc)

So for example here is the code I attach to a cube gameobject which acts like a visualiser bar.

Code (csharp):

1. ...

This code works and the bars scale and change color in reaction to the music... sort of.

Now I have no idea what frequencies would represent bass, or other parts. The way I envisioned it working would be to specify a range of indexes in my sampleData array for each bar and that would (magically) make each bar represent each part of the music. So for example sampleData[10] to sampleData[20] might represent the bassier beats, and sampleData[50] to sampleData[60] might represent the higher pitched stuff.

So my questions here:

• Is the above correct? Is this how it works?
• How do you know which sampleData entries represent what parts?

 

I take it no one has any ideas :-/
I've looked around for examples but they all stop short of seperating out the different "parts" of the music...

 

What does this mean?

 

For example if I was thinking sampleData[0] to sampleData[10] might represent bassier parts. So I could use that data to have one object scale scale in time with the bass beat. And sampleData[100] to sampleData[120] might represent higher notes so I could use that data to make another object react to the higher frequencies.

This is what I meant by "parts." How do I know what part/frequency of the music each sampleData represents?

I have a crude prototype but ideally I would want to have a bar bouncing to the bass and other bars bouncing to other parts of the music...

http://www.youtube.com/watch?v=AEQIhMdsEM4

 

Thanks you asking this question, I do have knowledge of audio engineering, but never have time to program on this subject on unity. So that, it is a good time for me to refresh my memory too

The 'part' that you have mentioned is called harmonic
http://en.wikipedia.org/wiki/Harmonic

For example, when the GetSpectrumData() return you an array, freq_data[], the first element of the array freq_data[0] is the 1st harmonic or fundamental tone, freq_data[1] is 2nd harmonic... etc, etc

I tested the unity by import different frequency of test tone (sine wave), and found that the period of the 1st harmonic frequency is 0.34133333333 second when set up the array to 8192, 'new freq_data[8192]' .
(I got the results from the above test that is same as the calculation based on this. 1/48000 * 8192 *2)

So that, if you looking for 440Hz, the element of the array should be freq_data[150], because 440 * 0.34133 = 150.1852.

Have fun.

Edit, the unit for the magic number 0.34133333333 should be second not Hz Sorry for the confusion. so that, the fundamental tone of freq_data[0] is 1.0/.34133333333 = 2.9296875 Hz

 

PS. based on the above result, I made this too....
[youtube]
http://www.youtube.com/watch?v=dQKXLYrrxKg

[web-player]
Audio_Spectrum_v002_web.html

....and here is the source code for the sample project.

Code (csharp):

1. var freqData:float[] = new float[8192];
2. var listener : AudioListener;
3.  
4. var band:float[];
5. var g:GameObject[];
6.  
7. function Start()
8. {
9.     var n:int = freqData.length;
10.     var k:int = 0;
11.     for(var j=0;j<freqData.length;j++)
12.     {
13.         n /= 2;
14.         if(!n) break;
15.         k++;
16.     }
17.     band  = new float[k+1];
18.     g     = new GameObject[k+1];
19.     for (var i=0;i<band.length;i++)
20.     {
21.         band[i] = 0;
22.         g[i] = new  GameObject.CreatePrimitive(PrimitiveType.Sphere);
23.         g[i].transform.position = Vector3(i,0,0);
24.     }
25.     InvokeRepeating("check", 0, 1.0/15.0); // update at 15 fps
26. }
27.  
28. function check()
29. {
30.     listener.GetSpectrumData(freqData, 0, FFTWindow.Rectangular);
31.        
32.     var k:int = 0;
33.     var crossover:int = 2; 
34.     for(var i:int;i< freqData.length;i++)
35.     {  
36.         var d = freqData[i];
37.         var b = band[k];       
38.         band[k] = (d>b)? d:b;   // find the max as the peak value in that frequency band.
39.         if (i>crossover-3)
40.         {
41.             k++;
42.             crossover *= 2;   // frequency crossover point for each band.
43.             g[k].transform.position.y = band[k]*32;
44.             band[k] = 0;
45.         }  
46.     }
47. }
48. private var sqrt = Mathf().Sqrt;

In fact, you could change the update speed from the InvokeRepeating(). I made it slower as 15 fps becuase its seem to be look better with some 'lag' effect

 

apple_motion, you ROCK man! You ROOOOOCK!
Thank you sooo much for clearing the mystery behind GetSpectrumData( ) !!! And thank you sooo much for that great code example!

I don't know what this forum would be without people like you mate!

 

apple_motion, do you mind if I ask you a few questions regarding the above explanations. It seems that I'm a bit hard-headed at these things


1) When you say:

What does the float array size 8192 actually represent? Is it the number of harmonics you tell GetSpectrumData( ) to read every time from the sound data?

2) You said that the magic number "0.34133333333" is calculated with this formula: 1/48000 * 8192 *2.
If I would break down the formula into pieces, do I understand correctly the meaning of these numbers:
- 48000 the sample rate of the song?
- 8192 the number of harmonics per sample?
- 2 the number of channels in your test song (in this case being stereo?)
Am I close to understanding this?

3) Looking at the above code example you posted, if I would want to calculate the frequency corresponding to each of those spheres would I have to calculate it based on that "magical" number like below?
In the "for" statement of the check() function:

Code (csharp):

1.  
2. var sphereFreq:float = i / 0.34133333333;
3.  

I would greatly appreciate if you could help me with these questions my friend.

Thank you so much!

 

Hi!

I try to remake apple_motion's example code in c#. I'm a little bit confused at this part:

Code (csharp):

1.  
2. ...
3.     var n:int = freqData.length;
4.     var k:int = 0;
5.     for(var j=0;j<freqData.length;j++)
6.     {
7.         n /= 2;
8.         if(!n) break;
9.         k++;
10.     }
11. ...
12.  

in c# there aren't "!" operator for integers.

And when i'm tied out in JavaScript the value of the k is 13. but how?

Why are you doing this? What i'm doing wrong?

Also it's seems like if i set the k in c# to 13 only the first 3-4 Sphere are moving (except the first one) but it generate 14 of them.

Sorry for my bad English, and i hope i'm going to get answer soon.

Update:

Ohh... I'm solved it! Here is apple_motion's example in C#

Code (csharp):

1.  
2. using UnityEngine;
3. using System.Collections;
4.  
5. public class Spectrum : MonoBehaviour {
6.    
7.     public int numOfSamples = 8192; //Min: 64, Max: 8192
8.    
9.     public AudioSource   aSource;
10.    
11.     public float[] freqData;
12.     public float[] band;
13.    
14.     public GameObject[] g;
15.    
16.     void Start ()
17.     {
18.         freqData = new float[numOfSamples];
19.        
20.         int n  = freqData.Length;
21.        
22.         // checks n is a power of 2 in 2's complement format
23.         if ((n  (n - 1)) != 0) {
24.             Debug.LogError ("freqData length " + n + " is not a power of 2!!! Min: 64, Max: 8192.");        
25.             return;
26.         }
27.        
28.         int k = 0;
29.         for(int j = 0; j < freqData.Length; j++)
30.         {
31.             n = n / 2;
32.             if(n <= 0) break;
33.             k++;
34.         }
35.        
36.         band  = new float[k+1];
37.         g     = new GameObject[k+1];
38.        
39.        
40.         for (int i = 0; i < band.Length; i++)
41.         {
42.             band[i] = 0;
43.             g[i] = GameObject.CreatePrimitive(PrimitiveType.Sphere);
44.             g[i].renderer.material.SetColor("_Color", Color.cyan);
45.             g[i].transform.position = new Vector3(i, 0, 0);
46.  
47.         }
48.  
49.         InvokeRepeating("check", 0.0f, 1.0f/15.0f); // update at 15 fps
50.  
51.        
52.     }
53.    
54.     private void check()
55.     {
56.         aSource.GetSpectrumData(freqData, 0, FFTWindow.Rectangular);
57.        
58.         int k = 0;
59.         int crossover = 2;
60.        
61.         for (int i = 0; i < freqData.Length; i++)
62.         {
63.             float d = freqData[i];
64.             float b = band[k];
65.            
66.             // find the max as the peak value in that frequency band.
67.             band[k] = ( d > b )? d : b;
68.            
69.             if (i > (crossover-3) )
70.             {
71.                 k++;
72.                 crossover *= 2;   // frequency crossover point for each band.
73.                 Vector3 tmp = new Vector3 (g[k].transform.position.x, band[k]*32, g[k].transform.position.z);
74.                 g[k].transform.position = tmp;
75.                 band[k] = 0;
76.             }
77.         }
78.     }
79.    
80.     void Update () {}
81. }
82.  
83.  

 

I get this error

Assets/Scripts/Bars.cs(23,22): error CS0119: Expression denotes a `variable', where a `method group' was expected

can u help me please!

 

@apple_motion
Hi, while we're on the subject of spectrum data inside Unity, I'm trying to reproduce the Blackman Harris algorithm that unity has, because Unitys version has an overhead, and I wish to use custom sin and cos functions that I created that are more than twice as fast as even the ones in System.Math. However, I'm only getting close, not quite getting it right, I was wondering in the documented math what the n/N represents? Any help would be appreciated, thanks!

 

Amusingly, Iive just implemented something like this myself, Heres what I have for the BlackmanHarris Window

Code (CSharp):

1.         for ( int i = 0; i < spectrumBandCount; i++ ) {
2.                
3.           windowingValues[i] =
4.              0.35875f -
5.              0.48829f * Mathf.Cos( 2f * Mathf.PI * i / spectrumBandCount ) +
6.              0.14128f * Mathf.Cos( 4f * Mathf.PI * i / spectrumBandCount ) -
7.              0.01168f * Mathf.Cos( 6f * Mathf.PI * i / spectrumBandCount );
8.  
9. }

This is inline with the wikipedia article on fft winows.

<protip> precompute the windowing values like this, since each time you try to analyse the spectrum, you use the same window multipliers

 

yay thank you and I know right @ precomputing,

EDIT:
hmm I put two versions of your spectrum script side by side, one using my implementation of BlackmanHarris revised with your help, and the built in BlackmanHarris window, and the results r different? :S

~
Another walk in the park.

 

I think is missing an "&": if ((n & (n -1)) !=0) {
Or just use: if (! Mathf.IsPowerOfTwo(n))

 

still no c# for this?

 

Did you read the thread?

 

Updated for Unity 5 and removed the power of 2 error check that throws compiler errors

Code (CSharp):

1. using UnityEngine;
2. using System.Collections;
3.  
4. public class APSpectrum : MonoBehaviour
5. {
6.  
7.     //reduced to 512 sampling the max is a bit much ;)
8.     public int numOfSamples = 512; //Min: 64, Max: 8192
9.  
10.     //Removed Updated to AudioListener
11.     //public AudioSource aSource;
12.  
13.     public float[] freqData;
14.     public float[] band;
15.  
16.     public GameObject[] g;
17.  
18.     void Start()
19.     {
20.         freqData = new float[numOfSamples];
21.  
22.         int n = freqData.Length;
23.  
24.         // checks n is a power of 2 in 2's complement format
25.         // check removed, bad code left in for referencing
26.         //if ((n(n - 1)) != 0)
27.         //{
28.         //    Debug.LogError("freqData length " + n + " is not a power of 2!!! Min: 64, Max: 8192.");
29.         //    return;
30.         //}
31.  
32.         int k = 0;
33.         for (int j = 0; j < freqData.Length; j++)
34.         {
35.             n = n / 2;
36.             if (n <= 0) break;
37.             k++;
38.         }
39.  
40.         band = new float[k + 1];
41.         g = new GameObject[k + 1];
42.  
43.  
44.         for (int i = 0; i < band.Length; i++)
45.         {
46.             band[i] = 0;
47.             g[i] = GameObject.CreatePrimitive(PrimitiveType.Sphere);
48.             g[i].GetComponent<Renderer>().material.SetColor("_Color", Color.cyan);
49.             g[i].transform.position = new Vector3(i, 0, 0);
50.  
51.         }
52.  
53.         InvokeRepeating("check", 0.0f, 1.0f / 15.0f); // update at 15 fps
54.  
55.     }
56.  
57.     private void check()
58.     {
59.         //Updated to Audio Listener this removes the tie to your audio source
60.         //Have one Audio Source in the scene playing, this allows multiple Listeners
61.         //to process the same Audio Source in a scene without tying up resources for
62.         //Source playing multiple sounds
63.         AudioListener.GetSpectrumData(freqData, 0, FFTWindow.Rectangular);
64.  
65.         int k = 0;
66.         int crossover = 2;
67.  
68.         for (int i = 0; i < freqData.Length; i++)
69.         {
70.             float d = freqData[i];
71.             float b = band[k];
72.  
73.             // find the max as the peak value in that frequency band.
74.             band[k] = (d > b) ? d : b;
75.  
76.             if (i > (crossover - 3))
77.             {
78.                 k++;
79.                 crossover *= 2;   // frequency crossover point for each band.
80.                 Vector3 tmp = new Vector3(g[k].transform.position.x, band[k] * 32, g[k].transform.position.z);
81.                 g[k].transform.position = tmp;
82.                 band[k] = 0;
83.             }
84.         }
85.     }
86. }
87.  
88.  
89.  

 

Hello!

The "power of two check" works if you write it correctly. What you forgot is the bitwise operator right after the first N.
It should be written this way:

Code (CSharp):

1.  
2. if ((numOfSamples & (numOfSamples - 1)) != 0)
3. {
4.     Debug.LogError("FreqData length " + numOfSamples + " is not a power of 2!!! Min: 64, Max: 8192.");
5.     return;
6. }
7.  

 

This script work great, except it causes my audio listener to become inactive after about ten seconds. Do you know why that might happen?

Thanks