RhythmTool - Music Analysis for Unity

Hello there!

I used to have an older version of RhythmTool that had some particular onset features that seem to no longer apply on the current version, to put on context:

My current project features 4 object spawners that would instantiate an object using the OnOnset() method, however back then OnOnset() featured onset types.

This is a bit of an issue for me because, I used different onset types for each of the spawners, so one of the spawners would fire if the type detected was low, another on mid, another on high, etc

I've been giving the documentation and the demo example a good view and working my way around instantiating objects on the newer version but so far I'm stuck in this part, (I'm already able to use onset strength as another conditional)

Is there a better way to handle instantiation for 4 different spawners in the newer version of RhythmTool now that onset types are gone?

 

Yeah, earlier versions detected onsets for different frequency ranges, but this approach didn't work very well. It wasn't a very reliable way to tell different sounds apart, because many sounds in music have components all over the spectrum. Because of this it would often detect onsets outside of the intended frequency range, while failing to detect other onsets.

A good alternative is to use the chromagram instead. This doesn't detect different percussion instruments, but it gives a rough representation of the song's melody.

 

I do like the asset but I did bump into a small prb for me.
I would like to use the Segmenter to detect the different segments of a song, but I don't manage to find it in the eventProvider. I'm mainly interested in when a new segment starts and if the overall volume of the new segment is higher or lower than the previous.
The best will be also to know if this segment has overall more volume than the average in the track, or if it's the most "calm" ...

 

Both volume and segments are stored in Value Tracks and you can find them by name. Segments are stored in a Value track with the name "Segments". A Value feature is just a float value and a timestamp.

Here's an example that compares new segments to the last. It subscribes to segment events with the track's name and then compares each new segment to the last.

Code (CSharp):

1. public RhythmEventProvider eventProvider;
2.  
3. private Value lastSegment;
4.  
5. private void Start()
6. {
7.     eventProvider.Register<Value>(OnSegment, "Segments");
8. }
9.  
10. private void OnSegment(Value segment)
11. {
12.     if(lastSegment == null || segment.value > lastSegment.value)
13.     {
14.         //new segment's value is greater
15.     }
16.  
17.     lastSegment = segment;
18. }

To compare segments to the average of a track, you could use the average of all segments. The segmenter does some filtering while the volume sampler does not, so you can't directly compare segments to volume features.

 

Thx a lot this totally help .
But I'm a bit unsure about how to get all the segments, could you please help me with that?

 

To get all segments, you can get the segments track from a RhythmData object like this:

Code (CSharp):

1. Track<Value> segments = rhythmData.GetTrack<Value>("Segments");

Then, you can go through all features and find the average like this:

Code (CSharp):

1. float average = 0;
2.  
3. for(int i = 0; i < segments.count; i++)
4.     average += segments[i].value;
5.  
6. if(segments.count > 0)
7.     average /= segments.count;

I should add that the analyzer should have analyzed the full song before trying to use features for the full song like this. Otherwise you will only find features for part of the song. You can check if the analyzer is done with RhythmAnalyzer.isDone and RhythmAnalyzer.progress.

 

It's working perfectly. Thx a lot.

 

Hello, this asset is definitely great. Now I'm coming back for more support

I'm doing the analyze of the songs in runtime, but I would like to save the result on disk to not redo the analyze the next time the song is played.

Do you have something already in the tool for that?

 

There's no feature like that currently included, but I made a serializer that can serialize RhythmData objects to JSON.

Please see this post for the script and an example. The example modifies the existing Audio Importer example to save and load analyzed songs instead of re-analyzing each song.

 

Thx a lot it just works out of the box

 

So I'm very new to Unity and taking it in school as part of a game dev major. I've read through the documentation and it still makes no sense to me. I've gotten it to analyze a song but I want to take the analysis file and read it to get a series of values from Chroma and volume that will procedurally generate a stage. Is there a way to convert the chroma values from the rhythm data to a readable array?

 

In addition, when trying to use anything that tries to use the RhythmDat type, I get the following error:

The type or namespace name 'RhythmData' could not be found (are you missing a using directive or an assembly reference?) Assembly-CSharp C:\Users\thefa\Documents\Ninja Runner\Assets\Scripts\rhythmTracker.cs 8 Active

--- Edit: Figured that error out. As I mentioned, I am very new to Unity.

 

The analysis results are stored in separate tracks, which can be used in a similar way as an array. You shouldn't have to convert anything.

The chroma features are stored in a chroma track called Chroma and volume features are stored in a value track called Volume. You can find these tracks like this.

Code (CSharp):

1. //Find the Value track with the name "Volume"
2. Track<Value> volumeTrack = rhythmData.GetTrack<Value>("Volume");
3.  
4. //no need to find the Chroma track by name because there is only one track with Chroma features
5. Track<Chroma> chromaTrack = rhythmData.GetTrack<Chroma>();

After you have the tracks, you can do whatever you want with their contents like this.

Code (CSharp):

1. //Loop through a track
2. for(int i = 0; i < volumeTrack.count; i++)
3. {
4.     Value volume = volumeTrack[i];
5.    
6.     //etc...
7. }
8.  
9. //Find a feature closest to a timestamp
10. int index = chromaTrack.GetIndex(123.4f);
11. Chroma chroma = chromaTrack[index];
12.  

 

Tool is gone from asset store, assuming it's a versioning issue. How much was this tool, when it was available?

 

It should definitely still be on the asset store, however, the link in the first post was broken. It should be fixed now.

 

Okay. Again, sorry as I'm very new to this...

So this was the base code which I've used to instantiate things on beats. How would I pull the current chroma value on each beat then:

------

using System.Collections.Generic;
using UnityEngine;
using UnityEngine.UI;
using RhythmTool;
public class rhythmTracker : MonoBehaviour
{
public RhythmData rhythmData;
public AudioSource audioSource;
private float prevTime;
private List<Beat> beats;
private List<Chroma> chroma;
public GameObject playerCharacter;
public GameObject enemyCharacter;
public GameObject GC;
void Awake()
{
beats = new List<Beat>();
}
void Update()
{
//Get the current playback time of the AudioSource.
float time = audioSource.time;
//Clear the list.
beats.Clear();

//Find all beats for the part of the song that is currently playing.
rhythmData.GetFeatures<Beat>(beats, prevTime, time);
//Do something with the Beats here.
foreach (Beat beat in beats)
{

playerCharacter.GetComponent<playerBehavior>().FireTestBall();
enemyCharacter.GetComponent<Enemycombat>().Attack();
// I want to find the current chroma value here.
//Debug.Log(chroma);
}
//Keep track of the previous playback time of the AudioSource.
prevTime = time;
}
}

 

You can find the chroma features in a similar way as the beat features, however, because chroma features have a length you should use Track.GetIntersectingFeatures() instead of Track.GetFeatures().

GetIntersectingFeatures will find all features where the timestamp and length overlaps with the time frame, while GetFeatures only finds features with a timestamp that falls within the time frame.

Code (CSharp):

1. using System.Collections.Generic;
2. using UnityEngine;
3. using UnityEngine.UI;
4. using RhythmTool;
5.  
6. public class rhythmTracker : MonoBehaviour
7. {
8.     public RhythmData rhythmData;
9.     public AudioSource audioSource;
10.     private float prevTime;
11.     private List<Beat> beats;
12.     private List<Chroma> chroma;
13.     public GameObject playerCharacter;
14.     public GameObject enemyCharacter;
15.     public GameObject GC;
16.  
17.     void Awake()
18.     {
19.         beats = new List<Beat>();
20.         chroma = new List<Chroma>();
21.     }
22.  
23.     void Update()
24.     {
25.         //Get the current playback time of the AudioSource.
26.         float time = audioSource.time;
27.  
28.         //Clear the list.
29.         beats.Clear();
30.         chroma.Clear();
31.  
32.         //Find all beats for the part of the song that is currently playing.
33.         rhythmData.GetFeatures<Beat>(beats, prevTime, time);
34.  
35.         //Find all chroma features with a timestamp and length that overlaps the part of the song that is currently playing.
36.         rhythmData.GetIntersectingFeatures<Chroma>(chroma, prevTime, time);
37.  
38.         //Do something with the Beats here.
39.         foreach (Beat beat in beats)
40.         {
41.             playerCharacter.GetComponent<playerBehavior>().FireTestBall();
42.             enemyCharacter.GetComponent<Enemycombat>().Attack();
43.            
44.             foreach(Chroma c in chroma)
45.             {
46.                 if((int)c.note % 2 == 0)
47.                 {
48.                     //do something if note is even
49.                 }
50.                 else
51.                 {
52.                     //do something if note is odd
53.                 }
54.             }
55.         }
56.  
57.         //Keep track of the previous playback time of the AudioSource.
58.         prevTime = time;
59.     }
60. }

 

Thanks a lot. I'm using this, if I hadn't mentioned it, for my class project. This should help me get to the point where everything else falls into place. Is there any way you'd like the tool attributed once our game is finished?

 

Hi @HelloMeow

How can I adjust the sensitivity of the beat tracker. I read the documentation but don't see this addressed. It seems to be picking up way too many beats. I tested with a song that only has a simple constant string instrument playing and it picked up beats all over the place, almost for the whole song. I must be doing something wrong here.

Code (CSharp):

1. //Find all beats for the whole song.
2.             print("checking for beats");
3.             rhythmData.GetFeatures<Beat>(beats, 0, audioPlayer.clip.length);
4.             print("there are " + beats.Count + " beats in this song");

 

The beat tracker tries to find the beat based on repetition in the song. Beats represent the tempo of the song. Like how someone would tap their foot. They don't represent any specific sound or instrument.

If a song doesn't have a clear enough rhythm, it will get confused and you will get beats that are all over the place. Ideally this shouldn't happen, but I'm not sure what would be the best way to deal with this.

 

Got a quick question: regarding volume.

Is it possible to use OnBeat but only take into consideration beats that go above a specified volume threshold?

I did read the analyzer does store this data, just not quite sore how to handle it around while inside of OnBeat()

 

Each type of analysis is stored in a separate track. You can find tracks and subscribe to events based on the feature type and (optionally) track name. Beats are stored in a track that stores Beat features. Volume is stored in a track that stores Value features with the name "Volume". Value features only contain a float value and a timestamp.

There can be multiple different tracks that store Value features (one track for volume and another track for segments), so when looking up volume, you have to specify the track name. There is only one track that stores Beat features, so you don't have to specify the track name.

You can combine beats and volume by looking up the corresponding volume feature at the timestamp of the beat. There are a few ways of doing that. The first is to directly find the volume feature at the beat's timestamp like this:

Code (CSharp):

1. public RhythmEventProvider eventProvider;
2. public RhythmData rhythmData;
3.  
4. private Track<Value> volumeTrack;
5.  
6. private void Awake()
7. {    
8.     //subscribe to Beat feature events
9.     eventProvider.Register<Beat>(OnBeat);
10.  
11.     //find the volume track
12.     volumeTrack = rhythmData.GetTrack<Value>("Volume");
13. }
14.  
15. private void OnBeat(Beat beat)
16. {
17.     //look up the index of the volume feature that is closest to the beat's timestamp
18.     int index = volumeTrack.GetIndex(beat.timestamp);
19.  
20.     //get the volume feature at this index and store the value
21.     float volume = volumeTrack[index].value;
22.  
23.     if(volume > 1)
24.     {
25.         //do something
26.     }
27. }

The second way of doing this is to subscribe to both beats and volume, so you can store volume in OnVolume and use it in OnBeat.

Code (CSharp):

1. public RhythmEventProvider eventProvider;
2.  
3. private float volume;
4.  
5. private void Awake()
6. {
7.     //subscribe to Value feature events from tracks named "Volume"
8.     eventProvider.Register<Value>(OnVolume, "Volume");
9.  
10.     //subscribe to Beat feature events
11.     eventProvider.Register<Beat>(OnBeat);
12. }
13.  
14. private void OnVolume(Value volume)
15. {
16.     //store the current volume
17.     this.volume = volume.value;
18. }
19.  
20. private void OnBeat(Beat beat)
21. {
22.     if(volume > 1)
23.     {
24.         //do something
25.     }
26. }

PS I haven't tested these examples and it's kind of early in the morning, so there might be typos or other errors.

 

That's alright, this more or less gives me an idea on how to get more consistent results when spawning notes, thank you very much!

For context's sake:

I'm reviving that one rhythm game I was building that used an older version of rhythm tool, I managed to get the newer version working but the way notes spawned in, while more accurate to the track BPM didn't quite have the same spread (in-game I have 4 objects that spawned notes, most of the time only 1 of them spawns any notes at all mostly due to the chromatic selection)

I've more or less got a similar result using OnBeat() it's just a matter of filtering out misfirings when there's little to no track volume and finding a way to "spread" the notes a bit more.

That being said, said example is a big help, I really appreciate it.

 

Edit: Used second method, works like a charm.

Forgot to place the Volume Sampler component on the scene which is why I couldn't detect volume at first.

 

Hello,

I'm trying to add a custom track to the RhythmData object to use the eventProvider and all the great stuff
But I'm a bit lost.

The track I would like to add uses the analysis of the chroma track to detect patterns in it, so I need to add it to the RhythmData after the initial analysis.

Any help will be awesome

 

You can add tracks to a RhythmData with RhythmData.tracks.Add().

 

Hello, I do have an issue with the rythmtool editor.

I did create a custom track that the editor sees and that I can edit with no prob, but when I quick unity and reopen the project, the track is gone. It looks like it's not saved in the Rythm data.

Also, I love the way I can directly add an onset for example at a specific value by double-clicking in the editor window, but my custom track seems to not work like that.

Any idea how to solve that?

 

Do you mean a custom track and feature type? Could you share some code?

 

Sure the track and feature are define like this:

public class patterntrack : Track<FeatureExted>
{

}
[Serializable]
public class FeatureExted : Feature
{
public float strength;
public float patternSumAverage = 0;
public int PatternID = 0;
public int AmountOfChromaInSegement;
public bool asMatch = false;
}

 

Thanks for sharing your code.

It seems to work for me, but only as long as the custom track is located in a script file with the same name. So in this case, patterntrack should be in a file called patterntrack.cs. Tracks are ScriptableObjects and Unity needs those to be in script files with the same name.

To get editor functionality for your custom feature, you'll have to implement a FeatureGUI and FeatureCreator for your custom feature. With FeatureGUI.GetRect you can define a rect based on your feature.

With FeatureCreator.Create you can create a new feature with a certain value. The value passed into Create is the normalized click position. So 0 is a click a the bottom of the track and 1 is a click at the top.

For example like this. This FeatureGUI and FeatureCreator are essentially the same as the one for Onsets.

Code (CSharp):

1. public class FeatureExtedGUI : FeatureGUI<FeatureExted>
2. {
3.     public FeatureExtedGUI(Track<FeatureExted> track, int index) : base(track, index)
4.     {
5.  
6.     }
7.  
8.     protected override Rect GetRect(Rect trackRect, EditorState state)
9.     {
10.         Rect rect = base.GetRect(trackRect, state);
11.  
12.         float height = rect.height * feature.strength * .1f;
13.  
14.         height = Mathf.Round(height);
15.  
16.         rect.y = rect.height - height;
17.         rect.height = height;
18.  
19.         return rect;
20.     }
21. }
22.  
23. public class FeatureExtedCreator : FeatureCreator<FeatureExted>
24. {
25.     public override FeatureExted Create(float timestamp, float value)
26.     {
27.         FeatureExted feature = base.Create(timestamp, value);
28.  
29.         feature.strength = value * 10;
30.  
31.         return feature;
32.     }
33. }

 

The prb was the script name, thanks so much for your help.

 

hello!
I want to use the user's real-time recording file to get the pitch and apply it to the game. Is this possible with your assets? If possible, could you give me the sample code?

 

This is not possible with RhythmTool. It can only analyze audioclips.