Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Beat detection/peaks at certain frequencies

Discussion in 'Scripting' started by Bogaland, Oct 12, 2019.

  1. Bogaland

    Bogaland

    Joined:
    Feb 19, 2017
    Posts:
    32
    Hello!

    I've found the following article on beat mapping in Unity, and managed to use it and adapt it into my game: https://medium.com/giant-scam/algor...nity-preprocessed-audio-analysis-d41c339c135a

    I've mostly edited how the peaks are presented and used in the game, and not much of the actual audio analysis (FFT etc). Now to my question. From the code in the article I get peaks from the entire audio spectra. The audio is sampled at 44100 Hz, and I get 1024 samples after the FFT. This would mean each sample would correspond to about 43 Hz. What I would like to do is to filter these samples somehow, so I only detect peaks within a certain frequency range (from 200-500 Hz for example, or any other interval).

    I've provided the code of the two relevant files, SongController and SpectralFluxAnalyzer. SongController reads the audio file, iterates through it, performs FFT and then sends the data to SpectralFluxAnalyzer which does the beat detection.

    SongController:

    Code (CSharp):
    1. using System;
    2. using System.Collections;
    3. using System.Collections.Generic;
    4. using System.Threading;
    5. using UnityEngine;
    6.  
    7. using System.Numerics;
    8. using DSPLib;
    9.  
    10.  
    11. public class SongController : MonoBehaviour {
    12.  
    13.     float[] realTimeSpectrum;
    14.     SpectralFluxAnalyzer realTimeSpectralFluxAnalyzer;
    15.     PlotController realTimePlotController;
    16.     public List<SpectralFluxInfo> spectralFluxSamples;
    17.  
    18.     public bool completed = false;
    19.     public double lowerFrequency = 0;
    20.     public double higherFrequency = 44100;
    21.     int numChannels;
    22.     int numTotalSamples;
    23.     int sampleRate;
    24.     float clipLength;
    25.     float[] multiChannelSamples;
    26.     SpectralFluxAnalyzer preProcessedSpectralFluxAnalyzer;
    27.     PlotController preProcessedPlotController;
    28.  
    29.     AudioSource audioSource;
    30.  
    31.     public bool realTimeSamples = false;
    32.     public bool preProcessSamples = true;
    33.  
    34.     void Start() {
    35.         audioSource = GetComponent<AudioSource> ();
    36.  
    37.         // Preprocess entire audio file upfront
    38.         if (preProcessSamples) {
    39.             preProcessedSpectralFluxAnalyzer = new SpectralFluxAnalyzer ();
    40.             //preProcessedPlotController = GameObject.Find ("PreprocessedPlot").GetComponent<PlotController> ();
    41.  
    42.             // Need all audio samples.  If in stereo, samples will return with left and right channels interweaved
    43.             // [L,R,L,R,L,R]
    44.             multiChannelSamples = new float[audioSource.clip.samples * audioSource.clip.channels];
    45.             numChannels = audioSource.clip.channels;
    46.             numTotalSamples = audioSource.clip.samples;
    47.             clipLength = audioSource.clip.length;
    48.  
    49.             // We are not evaluating the audio as it is being played by Unity, so we need the clip's sampling rate
    50.             this.sampleRate = audioSource.clip.frequency;
    51.  
    52.             audioSource.clip.GetData(multiChannelSamples, 0);
    53.             Debug.Log ("GetData done");
    54.  
    55.             Thread bgThread = new Thread (this.getFullSpectrumThreaded);
    56.  
    57.             Debug.Log ("Starting Background Thread");
    58.             bgThread.Start ();
    59.         }
    60.     }
    61.  
    62.     void Update() {
    63.         // Preprocessed
    64.         if (preProcessSamples) {
    65.             int indexToPlot = getIndexFromTime (audioSource.time) / 1024;
    66.             spectralFluxSamples = preProcessedSpectralFluxAnalyzer.spectralFluxSamples;
    67.             //Debug.Log(audioSource.time);
    68.             //preProcessedPlotController.updatePlot (preProcessedSpectralFluxAnalyzer.spectralFluxSamples, indexToPlot);
    69.         }
    70.     }
    71.  
    72.     public int getIndexFromTime(float curTime) {
    73.         float lengthPerSample = this.clipLength / (float)this.numTotalSamples;
    74.  
    75.         return Mathf.FloorToInt (curTime / lengthPerSample);
    76.     }
    77.  
    78.     public float getTimeFromIndex(int index) {
    79.         return ((1f / (float)this.sampleRate) * index);
    80.     }
    81.  
    82.     public void getFullSpectrumThreaded() {
    83.         try {
    84.             // We only need to retain the samples for combined channels over the time domain
    85.             float[] preProcessedSamples = new float[this.numTotalSamples];
    86.  
    87.             int numProcessed = 0;
    88.             float combinedChannelAverage = 0f;
    89.             for (int i = 0; i < multiChannelSamples.Length; i++) {
    90.                 combinedChannelAverage += multiChannelSamples [i];
    91.  
    92.                 // Each time we have processed all channels samples for a point in time, we will store the average of the channels combined
    93.                 if ((i + 1) % this.numChannels == 0) {
    94.                     preProcessedSamples[numProcessed] = combinedChannelAverage / this.numChannels;
    95.                     numProcessed++;
    96.                     combinedChannelAverage = 0f;
    97.                 }
    98.             }
    99.  
    100.             Debug.Log ("Combine Channels done");
    101.  
    102.             // Once we have our audio sample data prepared, we can execute an FFT to return the spectrum data over the time domain
    103.             int spectrumSampleSize = 1024;
    104.             int iterations = preProcessedSamples.Length / spectrumSampleSize;
    105.  
    106.             FFT fft = new FFT ();
    107.             fft.Initialize ((UInt32)spectrumSampleSize);
    108.  
    109.             Debug.Log (string.Format("Processing {0} time domain samples for FFT", iterations));
    110.             double[] sampleChunk = new double[spectrumSampleSize];
    111.             for (int i = 0; i < iterations; i++) {
    112.                 // Grab the current 1024 chunk of audio sample data
    113.                 Array.Copy (preProcessedSamples, i * spectrumSampleSize, sampleChunk, 0, spectrumSampleSize);
    114.  
    115.                 // Apply our chosen FFT Window
    116.                 double[] windowCoefs = DSP.Window.Coefficients (DSP.Window.Type.Hanning, (uint)spectrumSampleSize);
    117.                 double[] scaledSpectrumChunk = DSP.Math.Multiply (sampleChunk, windowCoefs);
    118.                 double scaleFactor = DSP.Window.ScaleFactor.Signal (windowCoefs);
    119.  
    120.                 // Perform the FFT and convert output (complex numbers) to Magnitude
    121.                 Complex[] fftSpectrum = fft.Execute (scaledSpectrumChunk);
    122.  
    123.                 double[] scaledFFTSpectrum = DSPLib.DSP.ConvertComplex.ToMagnitude (fftSpectrum);
    124.                 scaledFFTSpectrum = DSP.Math.Multiply (scaledFFTSpectrum, scaleFactor);
    125.  
    126.  
    127.                 // These 1024 magnitude values correspond (roughly) to a single point in the audio timeline
    128.                 float curSongTime = getTimeFromIndex(i) * spectrumSampleSize;
    129.  
    130.                 // Send our magnitude data off to our Spectral Flux Analyzer to be analyzed for peaks
    131.                 //preProcessedSpectralFluxAnalyzer.analyzeSpectrum (Array.ConvertAll (scaledFFTSpectrum, x => (float)x), curSongTime);
    132.                 preProcessedSpectralFluxAnalyzer.analyzeSpectrum(Array.ConvertAll(scaledFFTSpectrum, x => (float)x), curSongTime);
    133.             }
    134.  
    135.             Debug.Log ("Spectrum Analysis done");
    136.             Debug.Log ("Background Thread Completed");
    137.             completed = true;
    138.            
    139.         } catch (Exception e) {
    140.             // Catch exceptions here since the background thread won't always surface the exception to the main thread
    141.             Debug.Log (e.ToString ());
    142.         }
    143.     }
    144. }
    SpectralFluxAnalyzer:

    Code (CSharp):
    1. using System;
    2. using System.Collections;
    3. using System.Collections.Generic;
    4. using UnityEngine;
    5.  
    6. public class SpectralFluxInfo {
    7.     public float time;
    8.     public float spectralFlux;
    9.     public float threshold;
    10.     public float prunedSpectralFlux;
    11.     public bool isPeak;
    12.     public float peakFrequency;
    13. }
    14.  
    15. public class SpectralFluxAnalyzer {
    16.     int numSamples = 1024;
    17.  
    18.     // Sensitivity multiplier to scale the average threshold.
    19.     // In this case, if a rectified spectral flux sample is > 1.5 times the average, it is a peak
    20.     float thresholdMultiplier = 1.5f;
    21.  
    22.     // Number of samples to average in our window
    23.     int thresholdWindowSize = 50;
    24.  
    25.     public List<SpectralFluxInfo> spectralFluxSamples;
    26.  
    27.     float[] curSpectrum;
    28.     float[] prevSpectrum;
    29.  
    30.     int indexToProcess;
    31.  
    32.     public SpectralFluxAnalyzer () {
    33.         spectralFluxSamples = new List<SpectralFluxInfo> ();
    34.  
    35.         // Start processing from middle of first window and increment by 1 from there
    36.         indexToProcess = thresholdWindowSize / 2;
    37.  
    38.         curSpectrum = new float[numSamples];
    39.         prevSpectrum = new float[numSamples];
    40.     }
    41.  
    42.     public void setCurSpectrum(float[] spectrum) {
    43.         curSpectrum.CopyTo (prevSpectrum, 0);
    44.         spectrum.CopyTo (curSpectrum, 0);
    45.     }
    46.        
    47.     public void analyzeSpectrum(float[] spectrum, float time) {
    48.         // Set spectrum
    49.  
    50.         setCurSpectrum(spectrum);
    51.  
    52.         // Get current spectral flux from spectrum
    53.         SpectralFluxInfo curInfo = new SpectralFluxInfo();
    54.         curInfo.time = time;
    55.         curInfo.spectralFlux = calculateRectifiedSpectralFlux ().X;
    56.         curInfo.peakFrequency = calculateRectifiedSpectralFlux().Y;
    57.         spectralFluxSamples.Add (curInfo);
    58.  
    59.         // We have enough samples to detect a peak
    60.         if (spectralFluxSamples.Count >= thresholdWindowSize) {
    61.             // Get Flux threshold of time window surrounding index to process
    62.             spectralFluxSamples[indexToProcess].threshold = getFluxThreshold (indexToProcess);
    63.  
    64.             // Only keep amp amount above threshold to allow peak filtering
    65.             spectralFluxSamples[indexToProcess].prunedSpectralFlux = getPrunedSpectralFlux(indexToProcess);
    66.  
    67.             // Now that we are processed at n, n-1 has neighbors (n-2, n) to determine peak
    68.             int indexToDetectPeak = indexToProcess - 1;
    69.  
    70.             bool curPeak = isPeak (indexToDetectPeak);
    71.  
    72.             if (curPeak) {
    73.                 spectralFluxSamples [indexToDetectPeak].isPeak = true;
    74.             }
    75.             indexToProcess++;
    76.         }
    77.         else {
    78.             Debug.Log(string.Format("Not ready yet.  At spectral flux sample size of {0} growing to {1}", spectralFluxSamples.Count, thresholdWindowSize));
    79.         }
    80.     }
    81.  
    82.     System.Numerics.Vector2 calculateRectifiedSpectralFlux() {
    83.         float sum = 0f;
    84.         int maxIndex = 0;
    85.         float maxVal = 0f;
    86.         float tempMaxVal = 0f;
    87.  
    88.         // Aggregate positive changes in spectrum data
    89.         for (int i = 0; i < numSamples; i++) {
    90.             tempMaxVal = Mathf.Max(0f, curSpectrum[i] - prevSpectrum[i]);
    91.             if (tempMaxVal>maxVal)
    92.             {
    93.                 maxIndex = i;            
    94.             }
    95.             sum += tempMaxVal;
    96.         }
    97.  
    98.         System.Numerics.Vector2 temp = new System.Numerics.Vector2(sum, maxIndex);
    99.  
    100.         return temp;
    101.     }
    102.  
    103.     float getFluxThreshold(int spectralFluxIndex) {
    104.         // How many samples in the past and future we include in our average
    105.         int windowStartIndex = Mathf.Max (0, spectralFluxIndex - thresholdWindowSize / 2);
    106.         int windowEndIndex = Mathf.Min (spectralFluxSamples.Count - 1, spectralFluxIndex + thresholdWindowSize / 2);
    107.        
    108.         // Add up our spectral flux over the window
    109.         float sum = 0f;
    110.         for (int i = windowStartIndex; i < windowEndIndex; i++) {
    111.             sum += spectralFluxSamples [i].spectralFlux;
    112.         }
    113.  
    114.         // Return the average multiplied by our sensitivity multiplier
    115.         float avg = sum / (windowEndIndex - windowStartIndex);
    116.         return avg * thresholdMultiplier;
    117.     }
    118.  
    119.     float getPrunedSpectralFlux(int spectralFluxIndex) {
    120.         return Mathf.Max (0f, spectralFluxSamples [spectralFluxIndex].spectralFlux - spectralFluxSamples [spectralFluxIndex].threshold);
    121.     }
    122.  
    123.     bool isPeak(int spectralFluxIndex) {
    124.         if (spectralFluxSamples [spectralFluxIndex].prunedSpectralFlux > spectralFluxSamples [spectralFluxIndex + 1].prunedSpectralFlux &&
    125.             spectralFluxSamples [spectralFluxIndex].prunedSpectralFlux > spectralFluxSamples [spectralFluxIndex - 1].prunedSpectralFlux) {
    126.             return true;
    127.         } else {
    128.             return false;
    129.         }
    130.     }
    131.  
    132.     void logSample(int indexToLog) {
    133.         int windowStart = Mathf.Max (0, indexToLog - thresholdWindowSize / 2);
    134.         int windowEnd = Mathf.Min (spectralFluxSamples.Count - 1, indexToLog + thresholdWindowSize / 2);
    135.         Debug.Log (string.Format (
    136.             "Peak detected at song time {0} with pruned flux of {1} ({2} over thresh of {3}).\n" +
    137.             "Thresh calculated on time window of {4}-{5} ({6} seconds) containing {7} samples.",
    138.             spectralFluxSamples [indexToLog].time,
    139.             spectralFluxSamples [indexToLog].prunedSpectralFlux,
    140.             spectralFluxSamples [indexToLog].spectralFlux,
    141.             spectralFluxSamples [indexToLog].threshold,
    142.             spectralFluxSamples [windowStart].time,
    143.             spectralFluxSamples [windowEnd].time,
    144.             spectralFluxSamples [windowEnd].time - spectralFluxSamples [windowStart].time,
    145.             windowEnd - windowStart
    146.         ));
    147.     }
    148. }
    I'm trying to figure out some way to get at what frequencies the different peaks are detected. In that way, I can then only use peaks within a certain interval. Another approach would be to only perform the analysis within the preferred interval from the beginning.

    In SpectralFluxAnalyzer you can see I've tried to save the index of the rectified spectral flux, since I thought the most energy would correspond to the peak. But I don't really think this is the way to do it since the highest energy doesn't necessarily correspond to a peak.

    Any help is appreciated! I hope it is somewhat straight-forward what I am trying to do.
     
  2. Whacky0

    Whacky0

    Joined:
    May 16, 2020
    Posts:
    1
    How did u put 2 clases in one script?
     
  3. Vryken

    Vryken

    Joined:
    Jan 23, 2018
    Posts:
    2,106
    You can have multiple classes, structs, interfaces, and enums in the same file. It is valid C# code.

    What you can't do, specifically with Unity, is have multiple classes that inherit from
    MonoBehaviour
    in the same file. Unity will only recognize the first
    MonoBehaviour
    class and ignore the others.
     
    Bunny83 likes this.
  4. Bunny83

    Bunny83

    Joined:
    Oct 18, 2010
    Posts:
    3,919
    Even that is only half true :)

    Whenever you want to be able to attach a MonoBehaviour to a gameobject, it has to be inside its own file that matches the class name. However baseclasses that are never meant to be attached to a gameobject can be placed in any file.

    Though in general its good practise to place each class in its own file as it makes navigating inside your project easier. Yes, IDEs can help you finding classes, however from an organisatorial point of view (especially when larger teams work on a project) it's better to keep individual files simple. Of course there are always exceptions and trade-offs. For example a framework that is not meant to be extended / edited may be shipped as an assembly or just a single script file. I did this with my SimpleJSON framework. It's just a single file you have to copy, though it includes a total of 16 types (10x classes 3x structs and 3x enums). Same with my LogicExpressionParser which includes 32 classes in one file. This is not great when you're working on such a project, but it makes the usage simpler.

    When you define types that are tightly coupled to other types and they don't have much use outside this scope, it usually makes sense to include those types within the same file. Common examples are enums or structs that essentially belong to the actual class defined in the file. Though as mentioned above, in general its better to seperate types into their own classes. It helps with version control, debugging and navigating in your project.
     
    Vryken likes this.