RhythmTool - Music Analysis for Unity

Hi,

A quiet part of a song often doesn't have a different BPM. The tempo of a song doesn't depend on how loud it is.

You could try something like this. Here bpm slowly decreases each beat when there hasn't been an Onset for 1 second. Quiet parts of the song don't have a lot of onsets.

Code (CSharp):

1.  
2.     public float bpm;
3.     public float lastOnset;
4.  
5.     public RhythmEventProvider eventProvider;
6.  
7.     void Start ()
8.     {
9.         eventProvider.Register<Onset>(OnOnset);
10.         eventProvider.Register<Beat>(OnBeat);
11.     }
12.  
13.     public void OnOnset(Onset onset)
14.     {
15.         //when there is an onset, store the timestamp
16.         lastOnset = onset.timestamp;
17.  
18.         //you could also look for onsets above a certain strength like this
19.         //if(onset.strength > 1)
20.             //lastOnset = onset.timestamp;
21.     }
22.  
23.     public void OnBeat(Beat beat)
24.     {
25.         //if there hasn't been an onset for over 1 second, decrease bpm
26.         if (beat.timestamp - lastOnset > 1)
27.             bpm -= 5;
28.         else
29.             bpm = beat.bpm;
30.     }

 

Hi, many thanks for your help, I tried your code suggestion but it did not lower the BPM value (with silent audio track) for me, I suspect this is due to the code that actually lowers the bpm is in the OnBeat method, which probably is not called when there is no beat.
But, I think I have n alternative solution, I check which track is next in the audioclipselector script (where I have music>silent track > music track > silent track etc etc), if it is a silent piece of audio I set to play I set the BPM in the visualiser script to zero, that seems to work for me.

 

Hi, i sent you an email. Newbie question. Thank you

 

I found your earlier serializer example and that works, thanks

 

I analyze an the same audio clip multiple times and get slightly different results each time. Is this normal?

 

No, that shouldn't happen. Do the differences show up in one specific track or for a specific song?

 

I retested and you are correct. I'm not sure exactly what I was testing before, but it must have been resulting in slightly different audio clips.

 

Hi there,,

It's possible for the app detect live music note from microphone?

 

Unfortunately that's not possible. RhythmTool is designed to analyze AudioClips with a known length.

 

Hi HelloMeow, I would like to ask if the following would be possible, when analysing audio can it detect if there is singing/vocals present in the audio?
e.g. using Chroma (Chroma features are closely related to pitch and represent the most prominent notes at a certain time in the song).

 

Chroma features look at all frequencies, so it's not possible to tell the difference between vocals or another instrument. There do appear to be some methods that could work, but that is something I would have to look into.

 

Thanks for your reply, if you do find any option or method I would be interested to hear more. The game I am working on allows players to load audio to dance too that works very well so far, there is a band consisting of i.e. drummer, guitarist, singer, if the loaded audio does not contain any vocals I would like to change the animations for the singer NPC.

 

Hi there,
So, i'm basically mapping the notes to a specific location to occur on time, so I'm doing the following but I don't know if there's a better way to do it smoothly, I also tried with lerp but I want the note to not stop when it reaches the endpoint when using lerp, so I used the following code:

but I dunno, sometimes I feel like the note has a frame drop or something.
Thanks in advance.

Code (CSharp):

1.  
2. x.transform.position = new Vector3(x.transform.position.x, x.transform.position.y, startZ + (endZ - startZ) * (1f - (timestamp - player.time) / offset));
3.            
4.  

 

RhythmPlayer.time should be smooth. If RhythmPlayer.time has the same value multiple frames in a row, you can try setting Unity's DSP buffer size to Best Latency (Project Settings > Audio > DSP Buffer Size). This will make the timing more accurate in general.

The editor could also introduce stutter. It might be good to test the game in a standalone build to see if that reduces any stutter.

 

Any new on custom beatmaps? Super excited for this!

 

Yes! The RhythmData editor for the Unity editor is almost ready. I'm polishing it to get it ready for release right now. After that's done I'm going to start working on an in-game editor.

 

Awesome!
1. Will we be able to set the BPM manually and not rely on analysis?
2. How will mapping work exactly, events sent on each beat?

 

The editor lets you edit existing RhythmData objects that are created by analyzing a song, or you can create one from scratch. You can create and remove tracks and edit features on those tracks. You could create your own beat track and add your own beats with any BPM that you want.

 

When I change the AudioSource pitch at runtime, Beat.bpm (the current song's BPM) doesn't change. Is it normal?

 

Yes, that's normal. You can get the corrected bpm by multiplying the bpm with the pitch.

 

Something to add about this: You usually want to use AudioSource.time (or whatever it is called) as a reference for anything music / rhythm related. From that perspective, the tempo of the song does not change when you change the pitch. What does change is how time passes, so if you pitch down to 0.5, one minute in "real time" equals two minutes in "music time".

 

Hi, is it possible to skip to a certain position in a song the song when I call NextSong(). If I wanted to bypass a part of a song for a certain part of the game, but use the full song for others.
Thank you.

 

You should be able to do this by setting the RhythmPlayer's time.

 

Version 3.2 is now available! It includes the new RhythmTool editor that lets you view and edit RhythmData.

 

Hi there! Loving Rhythm Tools, the game I'm working on has many background features which animate in-sync to the music and RT has made that so much easier to handle.

This may be a newbie question, but I couldn't find any obvious answers in documentation or on here, is there a way to simply loop the Rhythm Player? So that the events keep sending after the Audio Source loops?

Thanks!

 

That's a good question. It should keep sending events after it loops. However, when you're using a RhythmEventProvider with an offset it will not loop around the events when you're nearing the end of the song. Instead it will catch up at the moment the AudioSource loops around.

If this is not working as expected or if you need to be able to use an offset, please let me know.

 

Oh, OK I see - so the AudioSource needs to be playing & looping on the same object as the RhythmPlayer for the RhythmPlayer to detect it and loop. That makes sense.
I have a custom audio manager that dynamically instantiates separate child AudioSource objects and controls their volume, settings, etc. I put the RhythmPlayer on the parent audio manager object, but the attached AudioSource wasn't doing anything, hence why RhythmPlayer wasn't looping. Set a silent reference audio clip to that audioSource and now it's working nicely.

 

Hi! Can i detect first beat/onset of the track with your asset? I need construct some type of help marks for metronome.

 

Doing this reliably is currently not possible. Many songs have an intro with different sound effects, or a long intro without any meaningful beat. The first onset or beat detected by Rhythmtool often does not correspond with the actual first beat.

 

Thanks for explanation! Did you use the same method like in this post. Or it's another algorithm?

 

For onset detection I do something similar, but I use different peak picking and spectral difference methods.

For beat detection you need to do some more work, because onsets aren't beats. An onset can be any sound, like a kick, a snare drum, a random instrument or even vocals. Some of these happen on a beat, but most do not. If you try to find the bpm by just looking at some onsets, you probably won't get anything useful.

 

I used the earlier serializer example in my project. The file generated is not compatible between computers. I am not even able to get compatibility between a build and a editor version on the same computer. Are you seeing the same results? I think the header or footer information is different between sources.

 

There appears to be a difference between how Unity serializes objects in the editor and standalone versions. In the editor it serializes the instance id as "instanceID" and in a standalone build it serializes the instance id as "m_FileID". This causes serialized files to be incompatible between the editor and standalone builds.

Are you seeing the same issue between standalone builds between different devices? I think the issue should only occur when you try to serialize and deserialize files between the editor and standalone builds.

A quick and dirty solution would be to check if you're in the editor and replace "instanceID" with "m_FileID" or the other way around in the json string if necessary.

 

Hi @HelloMeow, quick question; is there a way to normalize onset.strength what is the max value that strength can have?

 

Onsets are normalized by using the running standard deviation. The max value is around 10. You can re-normalize the onsets after the full song is analyzed by doing this:

Code (CSharp):

1. Track<Onset> onsets = rhythmData.GetTrack<Onset>();
2.  
3. float max = 0;
4.  
5. for(int i = 0; i < onsets.count; i++)
6. {
7.     if (onsets[i].strength > max)
8.         max = onsets[i].strength;
9. }
10.  
11. for (int i = 0; i < onsets.count; i++)
12.     onsets[i].strength /= max;

 

Hey @HelloMeow, thanks for this easy-to-use asset. Good work!

BUT Does play button on the RhythmTool Editor Window actually plays the song? Because I cannot hear anything. Timeline moves and I can edit tracks but not hearing the song doesn't help to do that :/

 

You should definitely be able to hear the song, as long as the RhythmData has an AudioClip assigned to it. Which version of Unity are you using?

 

Yes sure, we have a fully functional game right now. Everything is set correctly. I can see waveform in the editor window, I can see beats and everything but there is no sound. I also checked "Mute Audio" is unchecked in game window

MacOS 10.15.6 Catalina
Unity 2019.4.8

 

I was able to retest. You are right, I think editor and standalone are the only issue.

 

That's weird. If you can see the timeline moving that means the hidden AudioSource is actually playing. I'm not sure why you can't hear it.

You can change line 62 of AudioUtil.cs to this to unhide the editor's AudioSource.

Code (CSharp):

1. gameObject = EditorUtility.CreateGameObjectWithHideFlags("RhythmTool_AudioPlayer", HideFlags.DontSave);

After you do this you have to restart Unity. Maybe this can shed some light on what's happening. Is the AudioSource muted or is the volume set to 0?

 

I tried a lot of things with hidden AudioSource object's HideFlags, I put some logs in there and logs were saying music was playing but I couldn't hear. I don't really know how but I got it working now :S weird because I tried it on a different Mac with same OS version, same project was working on that machine...

This is all I can tell :/ Thanks again for this beautiful asset!

 

Hi HelloMeow - thanks for a great tool. Almost embarrassed to ask, but I am not figuring out how to get the volume at a given beat event. What's the preferred pattern for this?

 

Ok, I think I got it - let me know if there is a better way:

private void OnBeat(Beat beat)
{
List<Value> volFeatures = new List<Value>();
rplayer.rhythmData.GetFeatures(volFeatures, beat.timestamp - 0.1f, beat.timestamp + 0.1f, "Volume");
float adjustedIntensity = volFeatures.Count > 0 ? ( volFeatures[0].value / 2 ) : 0;

... etc

its kind of a pity to drag in the rplayer. I am hungering for an event ( on the beat ) that gives me the last recorded values of all other tracks in a convenient collection. Thanks again - great tool.

 

That works, but there is an easier way. You can use the beat's timestamp to look for the closest volume feature directly like this:

Code (CSharp):

1. Track<Value> volumeTrack = rhythmData.GetTrack<Value>("Volume");
2.  
3. int index = volumeTrack.GetIndex(beat.timestamp);
4.  
5. Value volume = volumeTrack[index];

 

Greetings! I've set this up and have my drum loop analyzed. How would you suggest I map beat data to a light intensity as a simple test? I'm not sure if I code a script, apply an existing script to the light, or add events to the RhythmTool track window?

 

Hello there. You'll have to write a script for that.

Here's an example script that uses a RhythmEventProvider to start a coroutine that fades a light's intensity every time a beat happens.

Code (CSharp):

1. using System.Collections;
2. using UnityEngine;
3. using RhythmTool;
4.  
5. public class LightIntensity : MonoBehaviour
6. {
7.     public RhythmEventProvider eventProvider;
8.  
9.     public Light light;
10.  
11.     void Awake()
12.     {
13.         eventProvider.Register<Beat>(OnBeat);
14.     }
15.  
16.     private void OnBeat(Beat beat)
17.     {
18.         StopAllCoroutines();
19.         StartCoroutine(Fade());
20.     }
21.  
22.     IEnumerator Fade()
23.     {
24.         float intensity = 1;
25.  
26.         while (intensity > 0)
27.         {
28.             intensity -= .05f;
29.  
30.             light.intensity = intensity;
31.  
32.             yield return null;
33.         }
34.     }
35. }

 

Thank you for this script example - which helps me get started understanding the RhythmTool -and "hello, HelloMeow".

I'm wondering if in the roadmap the Rhythm Analyzer UI development will allow for a track of game events (timeline of 3D object events synchronized with the audio analysis)?

 

Hello!
I was wondering about RythmTool's compatibility with WebGL Unity builds.
When trying to run just the example visualizer I am getting a null reference exception in Chrome and Firefox.
Seems like the error is either caused in Bindings or the track drawer.

 

Hello! I am now making a game for which it is necessary to analyze the entire music track before starting it. In previous versions, there was such an opportunity (judging by the previous posts) and the question is this. How soon will it be possible (using an event) to initiate a full track analysis? Yes, I saw how it was suggested to do it through the Coroutine and while (! Analyzer.isDone) {code}? It's just that before starting the game, it's important for me to get all the data about the track. Thanks in advance.

 

Hi,

Yes, this is how RhythmTool works. If you need data for the full song, just wait for the full analysis to be complete. This usually takes a couple of seconds depending on hardware.