Search Unity

  1. Get all the Unite Berlin 2018 news on the blog.
    Dismiss Notice
  2. Unity 2018.2 has arrived! Read about it here.
    Dismiss Notice
  3. We're looking for your feedback on the platforms you use and how you use them. Let us know!
    Dismiss Notice
  4. The Unity Pro & Visual Studio Professional Bundle gives you the tools you need to develop faster & collaborate more efficiently. Learn more.
    Dismiss Notice
  5. Improve your Unity skills with a certified instructor in a private, interactive classroom. Learn more.
    Dismiss Notice
  6. ARCore is out of developer preview! Read about it here.
    Dismiss Notice
  7. Magic Leap’s Lumin SDK Technical Preview for Unity lets you get started creating content for Magic Leap One™. Find more information on our blog!
    Dismiss Notice
  8. Want to see the most recent patch releases? Take a peek at the patch release page.
    Dismiss Notice

RhythmTool - Make games that react to music with Unity

Discussion in 'Assets and Asset Store' started by HelloMeow, Sep 26, 2014.

  1. HelloMeow

    HelloMeow

    Joined:
    May 11, 2014
    Posts:
    168

    RhythmTool is a straightforward scripting package for Unity, with all the basic functionality for creating games that react to music.

    RhythmTool can analyze an entire song before playing it, or while it’s being played.

    There are a number of types of data it provides:
    • Beats
    • Onsets
    • Changes in overall intensity
    • volume
    This data can be used in various ways and is provided through collections and events.

    Use MP3 importer to import music in game.

    Supports Unity 5 and up. All source files included.

    Questions and feedback
    Any questions, feedback or features you would like to see? Post it here or send me an email at tim@hellomeow.net
     
    Last edited: Nov 24, 2017
  2. HelloMeow

    HelloMeow

    Joined:
    May 11, 2014
    Posts:
    168
    Good news everyone,

    In the last few weeks I have been able to make some big improvements to the tempo detection. Both the tempo detection and the synchronization with the song are much more accurate. It will also be more straightforward to use the tempo detection results. This really needed improving.

    The next update will bring a lot of changes and improvements:
    • Improved tempo detection
    • Improved the way tempo detection data is stored and how it can be used
    • Added BPM calculation
    • Analysis results can now be saved and loaded
    • RhythmTool is now derived from MonoBehaviour
    • Added OnReadyToPlay and OnEndOfSong messages
    • Renamed variables and methods to make more sense
    I'm currently cleaning up the code and re-writing the documentation. ETA is 2-3 weeks.
     
    elbows likes this.
  3. HelloMeow

    HelloMeow

    Joined:
    May 11, 2014
    Posts:
    168
    Version 1.6 has been released!
     
  4. -JohnMore-

    -JohnMore-

    Joined:
    Jun 16, 2013
    Posts:
    64
    Hi,

    Sorry, I already found the original thread and the documentation :)

    - deleted stupid questions-

    Thanks!
     
    Last edited: Feb 7, 2015
  5. HelloMeow

    HelloMeow

    Joined:
    May 11, 2014
    Posts:
    168
    Just a heads up,

    RhythmTool appears to work perfectly fine with Unity 5, but the Examples have a small issue where the music appears to be too low or inaudible. This is because Unity 5 doesn't convert Unity 4's 2d sounds correctly, basically turning them into 3d sounds.

    To solve this, just turn the "Spatial Blend" slider all the way to 0 for the AudioSources in the affected example scenes.
     
  6. sinetwo

    sinetwo

    Joined:
    Mar 23, 2015
    Posts:
    5
    Hi HelloMeow, this may be what i'm looking for. I'm effectively trying to create a metronome that has a looped beat, and play samples above that at decided times, and judge whether the person playing the game accurately hit it on those times or not.

    Would this be able to help with that workflow? it doesn't need to automatically calculate the BPM as I would do that, but I would need it to 'start' detection after an offset (e.g. if the song has an intro which should not be counted.)
     
  7. HelloMeow

    HelloMeow

    Joined:
    May 11, 2014
    Posts:
    168
    I don't think it will. This is mainly for analyzing songs on the fly.
     
  8. Izitmee

    Izitmee

    Joined:
    Jan 27, 2011
    Posts:
    3,225
    Hi @HelloMeow!

    I just discovered your package and am reeeeally interested in its BPM feature for a future project, so I'm gonna bother you with a couple of questions, if I may.
    1. If I understood correctly, BPM is calculated in realtime while the song plays. Is it possible for RhythmTool to parse bigger chunks of the song at startup (even if it requires some time) and return a more accurate BPM? This would also allow to use Rhythm tool in a "lighter" mode, without having to analyze stuff other than at the beginning.
    2. Out of curiosity, why no event system (or maybe I just missed it)? It would be nice to have an "OnBeat" (and so on) event that one could listen to, which would also prevent the "current beat for multiple frames" issue you mention in the docs.
    Cheers,
    Daniele
     
  9. HelloMeow

    HelloMeow

    Joined:
    May 11, 2014
    Posts:
    168
    Hi,

    It's possible to analyze the entire song at startup by just checking an option in the inspector. RhythmTool doesn't just calculate the BPM, it also tracks beats.

    Technically it would be possible to detect the BPM by analyzing a small part of a song, but not out of the box and not without making some changes. And if you do, you won't have the data for when the beats actually occur.

    It doesn't use Events. Mostly because I haven't had the need for my own games. It's on my list though.
     
  10. Izitmee

    Izitmee

    Joined:
    Jan 27, 2011
    Posts:
    3,225
    Thank you!
     
  11. joshvidamo

    joshvidamo

    Joined:
    Sep 12, 2013
    Posts:
    1
    Hi @HelloMeow!

    I've been looking for this for a looong long time now. This is worth the pay, thank you for this!

    I just want to know if you provide tutorials on a breakdown of using this Rhythm tool? Such as using beats as an interaction with the player's input and etc. (ex. Guitar Hero)

    If you do, that would be so cool 'cause I've been studying this tool for a while now and I think the documentation doesn't cover how to use the beats before, during, after they arrive on time. Or if it does, I'm having a hard time understanding it. :(

    Best Regards,
    Josh
     
    Last edited: Feb 19, 2016
  12. bahk007

    bahk007

    Joined:
    Sep 24, 2016
    Posts:
    4
    @HelloMeow
    Fantastic asset. I've downloaded and uploaded into my Unity project.
    I'm wondering if there is a way to code the data that the script is picking up and apply it to transformations of a set of objects. Kind of like a audio visualizer but without instancing objects.
     
  13. h_nob

    h_nob

    Joined:
    Oct 3, 2014
    Posts:
    2
    Hi @HelloMeow!
    Thanks for a good asset

    I have a question.
    If it is over bpm 180, it will be 90 in half, how can I solve this?
     
  14. HelloMeow

    HelloMeow

    Joined:
    May 11, 2014
    Posts:
    168
    Hi,

    Unfortunately that's a limitation of the method used to find the most likely beat length. It's limited to a certain range, which is between 80 and 160 bpm. I'm still trying to improve this.
     
  15. h_nob

    h_nob

    Joined:
    Oct 3, 2014
    Posts:
    2
    Thanks for the reply.
    I'm looking forward to the update!
     
  16. smnerat

    smnerat

    Joined:
    Sep 23, 2012
    Posts:
    35
    @HelloMeow I tried out the demo and it was pretty awesome, I was getting solid onsets and beats for the several songs that I tried. My only question before purchasing is the number of bands that you have. Am I limited to just the low, mid, high and beats? Or is that just for the best detection?

    I need to use between 12 and 24 bands, with 16 probably being the average, so at 16 bands would I still be able to get solid detections?
     
  17. HelloMeow

    HelloMeow

    Joined:
    May 11, 2014
    Posts:
    168
    Hi!

    There are 4 default analyses with different frequency ranges for onset detection. These ranges are somewhat arbitrary, but they appeared to give the most useful results. You can change these ranges, or add more analyses with different ranges, although I'm not sure how that will affect the results.
     
  18. smnerat

    smnerat

    Joined:
    Sep 23, 2012
    Posts:
    35
    As a heads up to anyone else, it's pretty easy to setup analyses for bands besides the defaults. The onsets I get lineup pretty well with what I expect and hear. I haven't tested out using more bands yet, but I doubt the performance would drop very much. Solid purchase.

    Code (CSharp):
    1. int FrequencyToSpectrumIndex (float f) {
    2.     var i = Mathf.FloorToInt (f / AudioSettings.outputSampleRate * 2.0f * 1024);
    3.     return Mathf.Clamp (i, 0, 1024);
    4. }
    5.  
    6. void SetAnalyses() {
    7.     float[] frequencies = new float[]{ 31.5f, 63, 125, 250, 500, 1000, 2000, 4000, 8000, 12500, 16000, 20000 };
    8.     float bandwidth = 1.414f;
    9.  
    10.     for (var i = 0; i < frequencies.Length; i++) {
    11.         int a = FrequencyToSpectrumIndex (frequencies[i] / bandwidth);
    12.         int b = FrequencyToSpectrumIndex (frequencies[i] * bandwidth);
    13.         rhythmTool.AddAnalysis (a, b, i.ToString());
    14.     }
    15. }
     
    Last edited: May 30, 2017
    Akshara likes this.
  19. HelloMeow

    HelloMeow

    Joined:
    May 11, 2014
    Posts:
    168
    RhythmTool 2.0 has been released. This new version brings a number of improvements and big changes:
    • RhythmTool, RhythmEventProvider and the documentation have been rewritten from scratch
    • Removed RhythmTool.IsBeat and RhythmTool.IsChange. ContainsKey or TryGetValue for both the beats and changes collections can be used instead
    • Replaced RhythmTool.NewSong() with RhythmTool.audioClip
    • Renamed RhythmTool.calculateTempo to RhythmTool.trackBeat
    • Renamed RhythmTool.preCalculate to RhythmTool.preAnalyze
    • Renamed RhythmTool.storeAnalyses to RhythmTool.cacheAnalysis
    • Added RhythmTool.Reset and RhythmEventProvider.Reset, an event that occurs when a song has restarted or a new song has been loaded
    • RhythmEventProvider now needs to be given a RhythmTool Component instead of using all RhythmTool components in the scene
    • RhythmEventProvider no longer uses UnityEvents. Instead it uses c# events, for a more consistent api. In the previous verions c# events and UnityEvents were used in different cases. This means the events are no longer available in the editor
    If you're using a previous version and plan to use version 2.0, you will need to keep an eye on the following:
    • Use RhythmTool.audioClip instead of RhythmTool.NewSong()
    • Use the new C# events in RhythmEventProvider instead of the old UnityEvents
    • Give every RhythmEventProvider a RhythmTool component to use for it's events
    All the examples and the optional AudioImporter have been updated as well.
     
  20. yozzozo

    yozzozo

    Joined:
    Oct 7, 2013
    Posts:
    5
    Hi @HelloMeow,

    I've bought RhythmTool and integrated it into my game, but running into a couple issues. The game uses a short song that loops, as opposed to a single longer song that starts and ends. I'm detecting the end of the song and then simply calling Play again to loop.

    One issue is that the beat events from the RhythmEventProvider seem to only get sent until the song loops, then they are never sent again.

    Another thing is, I'm relying on BeatTime() to calculate normalized beat time (to synchronize GameObjects), and that seems to work fine even as the song loops. However, there's a delay right when as it loops -- where I'm assuming the song is being re-analyzed by RhythmTool, despite me having the "Cache Analysis" option turned on. Eventually, the tracking "catches up" to the playing song, and it works fine, until it loops again. When things loop, shouldn't it just set its internal tracking indices to 0 and re-use the existing analysis if the song hasn't changed?

    Thanks!
     
  21. HelloMeow

    HelloMeow

    Joined:
    May 11, 2014
    Posts:
    168
    Hi Yozzozo,

    The issue with the RhythmEventProvider is a known bug and has been fixed in version 2.0. Please contact me at tim@hellomeow.net if you want me to send you a fix for an older version.

    The other issue (with BeatTime) might be caused by the analysis itself. The synchronization of the beats can be off near the very start of the song. I'm trying to improve this for the next update. It's not likely that it is re-analyzing the song, because it doesn't do that when song hasn't been changed.
     
  22. yozzozo

    yozzozo

    Joined:
    Oct 7, 2013
    Posts:
    5
    @HelloMeow Thanks. Regarding the RhythmEventProvider issue, I'm actually using version 2.0, so this may be something new...
     
  23. HelloMeow

    HelloMeow

    Joined:
    May 11, 2014
    Posts:
    168
    I've found something that I hope might be causing the issue.

    Unity doesn't make a distinction between an AudioSource that has stopped playing and an AudioSource that is paused. RhythmTool did not handle this correctly. When a song reaches the end and stops playing, it assumes it's paused. So when Play() is called and the song restarts, it does not reset the loop that checks for some of the events.

    Adding the following lines in RhythmTool.cs should fix it.

    In OnSongEnded() around line 500 add Stop();

    Code (csharp):
    1.  
    2. private void OnSongEnded()
    3. {
    4.     Stop();
    5.  
    6.     if (SongEnded != null)
    7.         SongEnded();
    8. }
    9.  
    And in OnReset() around line 520 add currentFrame = 0;

    Code (csharp):
    1.  
    2. private void OnReset()
    3. {
    4.     lastDataFrame = 0;
    5.     currentFrame = 0;
    6.  
    7.     if (Reset != null)
    8.         Reset();
    9. }
    10.  
     
  24. yozzozo

    yozzozo

    Joined:
    Oct 7, 2013
    Posts:
    5
    @HelloMeow This fixed the issue for me! The song now sends beats reliably on every loop. Thanks!
     
  25. HelloMeow

    HelloMeow

    Joined:
    May 11, 2014
    Posts:
    168
    @yozzozo That's good to hear. A new update including this fix has been released today.
     
  26. lawrencetran

    lawrencetran

    Joined:
    Mar 4, 2015
    Posts:
    4
    Hi can you explain this part of the code in the Visualizer example?
    Code (CSharp):
    1.         foreach (Line line in lines)
    2.         {
    3.             Vector3 pos = line.transform.position;
    4.             pos.x = cumulativeMagnitudeSmooth[line.index - rhythmTool.currentFrame] * .2f;
    5.             pos.x -= magnitudeSmooth[rhythmTool.currentFrame] * .2f * rhythmTool.interpolation;
    6.             line.transform.position = pos;
    7.         }
    It seems that you are using the cumulative magnitude as a measure of distance for each Line. But why do you multiply by the interpolation of the past frame? It seems to work just fine without it.

    Also is this the best way to move these Lines towards an endpoint? Why is using cumulative magnitude better than just counting the number of frames between and interpolating between frame indexes?
     
  27. HelloMeow

    HelloMeow

    Joined:
    May 11, 2014
    Posts:
    168
    Interpolation is used to move lines smoothly. Otherwise the lines would only appear to move when currentFrame changes, which is only roughly 30 times per second.

    In the example the lines move based on the loudness of the song, which is why cumulativeMagnitudeSmooth is used. The position of each line is based on the sum of all magnitudeSmooth from currentFrame up to the line's frame index.

    This is how to move the lines at a constant speed, only based on the line's frame index.

    Code (csharp):
    1.  
    2. foreach (Line line in lines)
    3. {
    4.     Vector3 pos = line.transform.position;
    5.     pos.x = line.index - rhythmTool.currentFrame - rhythmTool.interpolation;
    6.     line.transform.position = pos;
    7. }
    8.  
     
  28. jashan

    jashan

    Joined:
    Mar 9, 2007
    Posts:
    2,958
    Hi Tim, first of all: Awesome tool, thank you for sharing it via the Asset Store! We're currently using it to add procedural beatmap generation into Holodance and results so far are very promising!

    I ran into one thing I found a little odd and just wanted to check with you about this:

    Code (csharp):
    1.         _low = new Analysis(0, 30, "low"); //0hz - 645hz
    2.         _mid = new Analysis(30, 350, "mid"); //645hz - 7500hz
    3.         _high = new Analysis(370, 900, "high"); //7500hz - 20000hz
    4.         _all = new Analysis(0, 350, "all"); //0hz - 7500hz
    To me, this looks like there's a gap between 350 and 370, and, more importantly, _all ignores anything above 7500 hz. Is this done this way intentionally, or maybe just an oversight?

    Obviously, making a change here could throw off a lot of things relying on the current ranges, so I want to make sure this is correct as early as possible ;-)
     
  29. HelloMeow

    HelloMeow

    Joined:
    May 11, 2014
    Posts:
    168
    Hi,

    Yeah, that's intentional. The ranges are kind of arbitrary and happened to work the best for most songs I used while testing. A gap like that between the different ranges doesn't really affect the results, because most sounds are made up of a wide range of frequencies.

    Using different ranges like this really only allows you to get a very rough idea of the kinds of onsets that are detected. Recently I've been looking into alternative or better methods that could provide better data.
     
  30. jashan

    jashan

    Joined:
    Mar 9, 2007
    Posts:
    2,958
    Thank you, that clarifies it. And can't wait for better data, even though the results are already a lot of fun to play and work perfectly in many cases.

    The greatest issue I see at the moment is tempo-recognition and BPM. While it does work decently in some cases, we often get "half beat" (or maybe "double beat"): The song is actually fairly fast, like 150+ BPM but RhythmTool thinks it's 75+ BPM. While with certain kinds of music this is a bit fuzzy (Dubstep would be an obvious example), I have seen this with 4 to the floor techno / trance tracks.

    Another issue is that most current music actually never changes tempo but might start with little percussion, so at first the tempo detection is way off (simply because there's no onsets that would give you any reasonable way to guess the tempo) and then later slowly adapts. In other words: I see a lot of tempo changes where there aren't any. The way I do this when I manually sync a song is by guessing the BPM, and then comparing the bars in Cubase with the waveform and make sure the "ones" line up throughout the song.

    Would it be possible to add a checkbox for this assumption ("assume the song does not change tempo"), and then add an heuristic like that? Basically something that tells RhythmTool "look, this song doesn't change tempo, so when you make the right guess, it will work throughout the whole song, which you can test by getting the 'dominant BPM', and then checking if that matches most beats when you create them under that assumption".

    Closely related: For quite a few features we actually need to know where the bars start. So while it's great to know "here's a beat", what we often need is "this is the first beat of that bar". With constant BPM and an offset (song might actually start after 0.378 seconds intro), this would be easy to calculate (we actually even have that calculation, in other words, we build our own ticks based on offsets and tempi in various sections of songs where we know those for sure, in the case of procedural mapping, I'd be quite happy to have it work at least with constant-temp songs).
     
  31. HelloMeow

    HelloMeow

    Joined:
    May 11, 2014
    Posts:
    168
    Currently the beat tracking is limited to a range of 80-160 BPM. Anything outside of that range gets halved or doubled.
    This is an unfortunate side effect of the way competing BPMs are handled. Otherwise the detected BPM could fluctuate between half or double the actual value.

    Optionally looking for a static tempo is a great idea that I will add to the list. This would only be possible when pre analyzing songs.

    Bar detection, like a lot of music feature extraction, is not easy. At least not to do fairly consistently for most songs. It might seem easy if you know certain parameters, but these depend on the song and are not trivial to detect, even if you already have beat locations. It is one of the features I want to add, along with the detection of pronounced held notes and better beat tracking.
     
    jashan likes this.
  32. Jos-Yule

    Jos-Yule

    Joined:
    Sep 17, 2012
    Posts:
    248
    Looking at the data that is retuned by the tool, is it possible to subscribe to the Beat event, and trigger other sounds to match/sync to that beat? ie. I'd like to cut/crossfade 2 tracks, with the same beat, can I use the beat event to start the other clip in sync?

    Thanks!
     
  33. HelloMeow

    HelloMeow

    Joined:
    May 11, 2014
    Posts:
    168
    Hi,

    That is possible, but I think that the current version has two problems that could make it more difficult.

    The first is that it can be tricky or messy to handle multiple songs. You might need 2 RhythmTool components. The second is that beat tracking prioritizes smooth transitions when a different beat length and offset are detected. This is nice for gameplay and can mask incorrect beat tracking, but it will stand out when you sync a sound to it.

    I'm currently redesigning RhythmTool and these problems are some of the reason why.

    Instead of using the Beat event, I would recommend looking at the most common BPM and offset for the whole song and using those to sync the songs.
     
  34. Jos-Yule

    Jos-Yule

    Joined:
    Sep 17, 2012
    Posts:
    248
    Yes, that was my "fall back" option. I'm pretty sure that the samples/tracks I'm going to be using will have consistent BPM, so if I can detect the BPM (or know it before hand) I can figure that out from there.

    Thanks, good luck on v2!
     
  35. RonTang

    RonTang

    Joined:
    May 20, 2017
    Posts:
    4
    Hello @HelloMeow
    Thanks for your very good asset.
    I have a special case.
    I want to detect much frequent onsets , so I modify the var "frameSpacing" in RhythmTool to 666...
    But I think bpm broken , all the music bpm show 199.I believe this is not an easy issue for me.
    So I am looking forward your help.
    Thanks HelloMeow , good luck on v2!
     
  36. RonTang

    RonTang

    Joined:
    May 20, 2017
    Posts:
    4
    OK, I fixed it by my self. Many other var need modify due to "frameSpacing" changed. Thanks.
     
  37. HelloMeow

    HelloMeow

    Joined:
    May 11, 2014
    Posts:
    168
    Yes. I wouldn't recommend changing frameSpacing because most of the algorithms are tuned to use the default value.

    In version 3 this shouldn't be an issue.