Koreographer - Audio Driven Events, Animation, and Gameplay

Bring your music to life with Koreographer
Asset Store Links:
Koreographer | Koreographer Professional Edition


​

Koreographer simplifies the process of synchronizing gameplay and animations to music and sound in your game or application. It’s simple editing interface allows you to map events in the game to rhythms, beats, notes, volume, and other dynamics of the music.

• Enhance the cinematic quality of your game by tying visuals to your game's music score
• Create a rhythm game, enable lip syncing or add subtitles to audio with minimal effort
• Precisely synchronize in-game events to different points in your game's music
• Rapidly create new music-based gameplay, animations, and effects

“I am very excited about what Koreographer means for game development.... Koreographer really does represent a paradigm shift in how audio and visuals interact.”

- Adam Creighton, Creative Director, Panic Button​

"It's so good... it gives me that 'chills' effect every time. Totally crazy."

- Erik Asmussen, 82 Apps, developer of Disco Dodgeball​

Koreographer driven lighting effects in Disco Dodgeball​

Simplify the synchronization process​

The visual authoring interface and in-game event system make it fast and easy to tie object and code parameters to audio and [Pro] MIDI events.

Enable new music and gameplay experiences​

Explore game mechanics driven by music and make music a core part of your game. Or simply ratchet up that cinematic feel.

Enhance the music, don’t sacrifice game performance​

Koreographer’s approach to music synchronization eliminates realtime music processing, leaving your game running at peak performance.

Keep synchronization on time, always​

No more time spent debugging scripts and tweaking algorithms to resolve latency and other audio bugs across platforms.

Features

• View the music in the Koreography timeline and precisely add game events and other outputs
• [Pro] Import a MIDI file into Koreographer to quickly map notes or instrument data to game events
• Gain live access to your music's timeline from within your game: power animations with beats and musical events rather than built-in system timing
• Tag and sync to any musical event, even those not picked up by automated music analysis algorithms. Tag tempo changes, notes, musical phrases, lyrics, percussion, crescendos, section changes, and more
• Eliminates realtime music processing, leaving your game engine at peak performance
• Drive animations, textures, particle effects, gameplay actions, and more with music
• Perfectly execute subtitles, lyrics, and lip syncing - even includes a Karaoke demo! (demo requires Unity 5.0+)
• Send customizable payloads to your game along with the music event, including curves, text, numbers, [Pro] Unity assets, [Pro] colors, and [Pro] frequency spectra
• [Pro] Automatically generate events from the music using RMS and FFT analysis
• Create Rhythm gameplay with ease - Koreographer even gets you started with a built-in Rhythm Game demo! (demo requires Unity 5.0+)

Compatibility

• Supports Unity 4.5+.
• [Pro] Integration with Audiokinetic's Wwise 2017.1.0+ (new in 1.5.0!)†
• [Pro] Integration with Firelight Technologies' FMOD for Unity 2.0+ (new in 1.6.1)†
• [Pro] Integration with Tazman Audio's Fabric 2.2.2+ (new in 1.1.0!)
• [Pro] Integration with Procedural Worlds' SECTR COMPLETE (new in 1.3.0!)
  • [Backwards compatible with Make Code Now!'s SECTR Audio 1.2.1+]
• [Pro] Integration with Dark Tonic's Master Audio 4.2+ (new in 1.4.0!)
• Integration with Hutong Games' PlayMaker 1.7+ (new in 1.3.0!)
• All Unity platforms supported.
• All Unity scripting languages supported.
• Supports all music files supported by Unity including .mp3, .ogg, .wav and others.
• Any notification or event within Unity can be tied to the audio.

† Contact us for access.

Need more information?
Website | Documentation | Tutorial Videos | Contact Us | Discord | Request a Demo
Buy Now:
Koreographer | Koreographer Professional Edition​

* Items marked with [Pro] are limited to the Profession Edition.​

 

This look really cool , i'm looking forward to buyng it but the price is a little bit too expensive for me..
any plan on a discount?

 

Thanks!

We don't currently have plans to discount Koreographer or run a sale. That said, it is something we are considering for the future! Just nothing currently on the radar!

 

That is a very cool demo!

BTW, your "showcase" link is not working.

 

Woah, thanks for pointing that out! We've submitted a fix for it: now we play the waiting game!

In the meantime, this link should work!

 

How well does this run on mobile?.... I am interested in specifically, Android. I have been toying around for a while with ideas related to a musical/tap game.

 

We haven't run it on mobile ourselves, but some early beta users were very pleased! One of our beta users was even using the event system to control music transitions for a VR app on mobile (Samsung GearVR). It's actually thanks to her that we have a configurable system for delaying event processing to maintain crisp synchronization (see the "Special Note About Event Delay" in the Koreographer User's Guide).

 

This might just be the asset I need to create that Bust a Groove sequel I always wanted

 

Yup! Koreographer makes that pretty simple, actually!

 

The midi features are for reading midi files? Or can it receive and send midi data?

 

The MIDI features are for reading MIDI files. Koreographer includes a MIDI Converter that takes a MIDI version of an AudioClip and allows you to convert MIDI information (like note value or velocity) into Koreography data.

We have used our MIDI library internally to receive MIDI data, but that required a separate, custom app or plugin that connected to the host OSes MIDI-framework (all OSes provide different interfaces for MIDI which complicates things).

 

Imunderstand. Ive been trying to implement it myself and that has also been the only viable solution for me.

Thank you for the fast response.

 

Going to keep an eye out on this one. It is a little expensive for me right now, but I might take the plunge if I decide on doing a music based game. If any official tests are done via mobile platforms, I would love to hear about it.

 

We'll keep you posted!

 

Hey Everyone! We just released version 1.1.0 to the Unity Asset Store! Here's a list of notable changes from version 1.0.1:

• [NEW] Integration with Tazman Audio's Fabric audio engine!
• [NEW] HUGE fixes for runtime memory performance. We tracked down a slew of minor allocations in the API that could add up quickly over time. We think we got most of them but if you notice any allocations occurring, please let us know!
• [NEW] AudioSourceVisor component. Enables simple Koreography Event triggering based on an associated AudioSource component (does not enable the Music Time API - use a Music Player if this is needed).
• [NEW] Detect when Unity's Audio System is disabled when using the Koreography Editor; offer to re-enable it.
• [NEW] Core adjustments to make extending Koreographer easier by abstracting most of the "glue" visor code into the "VisorBase" class and removing AudioClip reference requirements on Koreography - matches now occur by name!

For the full list of release notes, click the version number on the asset store page.

Now back to adding more cool stuff!

 

Hey Folks! Version 1.2.0 just hit the Unity Asset Store! Here's a handful of changes from version 1.1.0:

• [NEW] Support for all Windows Phone 8 and Windows Store build targets! With these platforms joining the herd, Koreographer supports all platforms that Unity does (in Unity 4.5+)!
• [NEW] Massive simplification and optimizations to the KoreographyTrack serialization system! Hurray performance!
• [NEW] MIDI Converter can now Append/Replace Koreography Events to/in a Koreography Track!

The big news here, of course, is full support for the non-desktop Windows platforms. Those can be tricky due to C# API incompatibilities but we were able to wrangle things together to get a solid build moving!

For the full list of release notes, click the version number on the asset store page.

So many cool features to add, so little time! Back to it!

 

will there be playmaker connection in future? is it possible (for noncoders like me)

 

Yes! This is something that we have planned and will be adding very soon!

 

The tool looks great.
Can any portion be used with streaming audio (e.g. system audio or a large track) with out pre-processing the file?

 

Thanks!

Koreographer does not do any runtime processing of audio data: the Koreography events are added in the Unity Editor either using the Koreography Editor or via the MIDI Converter. That said, Koreographer does support Unity's Streaming AudioClip assets!

Does this help?

 

Thanks for the quick reply! Great to hear it supports Unity Steaming AudioClip assets.

 

Happy to help! That's what we're here for!

 

Just to be clear, if I (player) load some track inside a game dynamically (not predefined), would it still have full Koreographer support?
It would also mean that I could define some template (beat/speed etc.) which would then apply to any loaded track. Is that the case?

 

Koreography data consists of a Tempo Map and Koreography Events. Koreography Events are usually song-specific, meaning that you'd align an event to a beat drop, or something.

If your dynamically loaded audio files had identical musical structures you could swap out the AudioClip reference in the Koreography at runtime and it would work fine.

If what you're looking for is the Tempo Map (information about beat timing) and you either A) know that all your audio has the same BPM, or B) can ask your users to add the BPM information themselves (simple for single-tempo songs, harder for multi-tempo), then you can similarly swap out pre-defined Koreography or create new Koreography at runtime and set it up with the AudioClip reference and the tempo information.

Does this help answer your question? Can you go into a bit more detail about what exactly you have in mind?

 

Any suggestions on how I could integrate it with SECTR Audio? (https://www.assetstore.unity3d.com/en/#!/content/15325)

I'm not familiar with Fabric and my game is already using SA cues.

 

Thanks for reaching out! We have reached out to the makers of SECTR Audio to discuss how to best integrate the two plugins!

It would be very helpful for our conversation to get a better understanding of your use-case. Are you hoping to integrate Koreographer with the SECTR Audio's music playback? Sound effects? Ambient/environment effects?

 

Ideally I'd be able to link AudioCues and Ambient environment effects to Koreography (which I just bought yesterday, so I'm still working on wrapping my head around it in general). It might be difficult because of the way it can have subclips and whatnot... though they have audioclip sources at some level so idk.

Maybe some kind of hook to when one of the random sub-elements is triggered? I'm not sure how that would work though, since I think Koreographer does it's work before runtime. I might not even really need any direct integration to get what I'm after, maybe just some workflow method I'm not thinking through yet.

Basically right now I have dynamically generated level built out of "room" prefabs and it assigns an ambiance to play when the player enters various rooms. I'd like that random song selection to also affect various things within the room like that disco youtube video.

 

As you point out here, SECTR Audio uses Unity's AudioSources for audio playback at some level. This means that integration should be pretty trivial.

The way the Fabric integration works is that it watches over any AudioSource component used by the Fabric audio system. Presumably a SECTR Audio integration could work in an almost identical fashion. We just need to investigate with the actual package. We've reached out to the SECTR Audio developers to get the conversation rolling!

Depending on how SECTR Audio handles AudioSources, this should be a perfect use-case for an integration!

Could you perchance send a screenshot of your hierarchy in a live game? Specifically one showing an expanded SECTR Audio hierarchy?

 

Does the plugin already support or are there any plans to support the WebGL platform? WebGL has just been removed from 'Preview' state in the latest 5.3 release of Unity, and I would love to use this plugin for some web content. A quick attempt at building and running on the platform with content using this asset was a no go.

 

Yes, the plugin supports the WebGL platform. We have not yet had a chance to test a build in 5.3 but will fix any issues there ASAP. To our knowledge, WebGL has been supported since the initial preview in 5.0.

Are you hitting specific issues? What settings are you using to build? Which browser are you testing in?

 

I just loaded up the included "FullDemo" scene that is included with Koreographer and did a build with 5.3.0f4. Everything appears to work great in Safari 9.0.1 and Chrome 46 (on Mac). See the attached file for a screenshot!

Perhaps there's an issue with your test project? If you'd like us to take a look at your project directly, please feel free to reach out to us and coordinate a project handoff so that we may better assist you!

 

Good to hear! I will test this further when I get a chance tonight.

 

Awesome! Best of luck and let us know if you run into any issues! We're here to help!

 

Just tried a fresh rebuild, and all is well! Officially psyched to experiment with the toolset some more. Thanks for the quick reply!

 

Fantastic! Would love to see what you make if you're comfortable and can share the link when you're done!

 

Hello!
I started using the tool and I am loving it! But I am running into a problem that may not be covered by Koreographer.

I will just put an example:
I need to fire a bullet at some sound event, and this bullet needs to trigger a sound that should be in sync with the music. I get the Koreographer callback in the update phase, when the audio may have kept playing, then I play the sound, which may start with some delay, and it gets out of sync. Same thing with explosions and all the sound effects. I would like the music to drive their timing but also play the sounds perfectly

Is there a way I can play a sound in sync or queue to be played next event perfectly in synch?

Thanks for your time!

 

Hi @Eduardnaut! Thanks for reaching out! Very glad to hear that you are enjoying working with Koreographer

You are correct: this is not directly covered by Koreographer. The issue here, as you point out, is that there is some delay between when you ask an AudioSource to play an AudioClip and when that AudioClip is actually played. This is due to the underlying audio system's buffering mechanism: when you say "Play this file", there is already some amount of audio data that's been queued up and sent to the audio mixer/driver. The Played file will get mixed in the next time the audio sub-system requests more data. And if your audio file has any build up in it; if the "attack" doesn't start with sample one, then you get that sound's inherent lag added in, too. Please see the writeup in the Description section of the AudioSettings.GetDSPBufferSize documentation for more detail.

Yes. This is doable with a little extra work. In order to do this you will need to make use of the AudioSource's PlayScheduled feature. Here is an outline of how you might accomplish this:

1. In Start(), initialize a "scheduleSampleOffset" value such that:
   scheduleSampleOffset = dspBufferSize * (numDSPBuffers + 1)
2. In Update(), call levelKoreo.GetTrackByID(shootEventID).GetEventsInRange() with a startSample location of lastFrameSampleTime + scheduleSampleOffset and an endSample location of currentFrameSampleTime + scheduleSampleOffset.
   • "levelKoreo" is a reference to your Koreography data that contains the "shooting" events.
   • currentFrameSampleTime can be grabbed from Koreographer.GetSampleTime(). At the end of Update() you would set lastFrameSampleTime to this value.
3. If any KoreographyEvents are returned by the above call, do the following for each KoreographyEvent koreoEvent (assuming OneOffs):
   1. Calculate the time-in-seconds of koreoEvent in the future:
      eventTimeFromNow = (koreoEvent.StartSample - Koreographer.GetSampleTime()) / levelKoreo.SampleRate.
   2. Calculate the precise time on the DSP Timeline at which to start the audio:
      timeInTheFutureToPlay = AudioSettings.dspTime + eventTimeFromNow.
   3. Grab an AudioSource from your pool and ensure the AudioClip is loaded into it.
   4. Call AudioSource.PlayScheduled(timeInTheFutureToPlay).
4. Check your scheduled sound effect AudioSources to see if they've completed playback. If they have, return them to the pool (you actually may want to do this before steps 1-3 above - this would free them up for immediate reuse).

Note that the above algorithm is the quick-n-dirty way to accomplish what you're looking to do and there's plenty of space for optimizations and handling other edge-cases:

• GetEventsInRange() runs in O(n) time: it iterates over the entire list of KoreographyEvents in the KoreographyTrack until it reaches an Event greater than the specified range. You can use the KoreographyTrack.GetAllEvents() API to get your own local copy of the event list and then iterate over that while "remembering" the index of the "next event to check." This turns the algorithm into near-constant time - O(1).
• This may not work quite right with a non-steady pitch - I haven't spent the time to convince myself that it isn't a problem here. If you don't intend to change pitch at all, then this isn't a problem anyway (Pitch changes the rate at which audio data is consumed by the audio subsystem.)
• As hinted at in the algorithm above, you'll likely want to build a simple AudioSource pooling system as each sound effect (AudioClip) would require its own AudioSource to play from. PlayOneShot doesn't work as they can't be scheduled.

This is fairly involved but it isn't actually that difficult. The key insights are that you can use PlayScheduled to set the sound effects to play at the right time, use KoreographyTrack APIs to "peek" ahead at specific KoreographyTracks, and that you need to do some basic time conversion from "sample time" into "solar time" (seconds) in order to set the schedule properly.

And that's it!

Is this helpful? Please let us know if we can be of further help

 

Hi @FascistDonut! We have some good news for you! Koreographer 1.3 will include official SECTR Audio integration We've been hard at work on a lot of great new features and this is one that made the cut for the upcoming release! Using the included integration, all audio played through SECTR Audio can be detected by Koreographer and synchronized with!

 

Thanks @SonicBloomEric !!! That was indeed VERY helpful!
I will prepare such a system for queuing audio clips, at the same time I queue the koreo events (I made a system to queue "riffs patterns" at any matching time, and get some beat track of koreo events matching the riff pattern) Can´t wait to sync all my audio!

 

Great! I'm glad I was able to point you in a useful direction

That sounds really cool! I hope you can share it (video or WebGL example or the like!) once you get something moving!!

 

Hey everyone, just wanted to quickly announce that Version 1.3 has been submitted to the Asset Store and is currently pending review.

That said, I wanted to also mention that we are splitting Koreographer up into two separate versions: Koreographer and Koreographer Professional Edition. All current owners of Koreographer will be automatically updated to the Professional Edition once it launches. Importantly, the price of the package will also go up! That means that if you haven't already purchased it, you can save $18 by purchasing it now. See the following graphic for more details!

We have no clue when Unity will get to reviewing the submission, so get it now while the getting's good!

 

Aaaaand that was fast. Unity went ahead and pushed Koreographer Professional Edition live to the store! Still no word on the new Koreographer non-professional edition, but it should be available soon!

 

Hey everyone! Koreographer is undergoing a little change and we thought we'd take a minute to let you all know what's happening and why we're doing it!

For almost 6 months Koreographer has been a top Unity Asset Store Audio Extension. We’ve had fantastic developers use our technology to build games that we’re proud to have been a part of, including Panoramical and Disco Dodgeball.

The one thing we’ve been troubled by is the accessibility of Koreographer for developers that are tinkering or prototyping an idea in Unity. Some developers simply can’t justify the entrance fee, and they let us know the $116 price tag is a bit steep.

We at Sonic Bloom put our heads together and devised a way to provide people access to the core Koreographer experience at a price that makes sense for hobbyists, while taking nothing away from the brilliant developers that have already purchased Koreographer.

To that end, we’re proud to announce Koreographer and Koreographer Professional Edition.

Koreographer Professional Edition will be the platform for people looking to get the latest features, integrations, and workflow enhancements for their projects. It includes source code access, 3rd party tool integration (like Fabric and SECTR Audio, with more to come), and MIDI conversion. The best part is anyone who purchased Koreographer v1.2 or lower have already been upgraded to the Koreographer Professional Edition license.

So, what about the new, lower priced version of Koreographer? It will provide the core experience and functionality for those that want to check out what Koreographer can do, but don’t need the fancy bells and whistles. Just like the Professional Edition, it provides sample specific event timing in both the Koreography Editor and in our custom runtime system. To top it off, we gave both versions PlayMaker integration for visual scripting compatibility.

Thanks to everyone who has supported us thus far! We’re grateful to our fans, and we hope to support you with more great updates and new features soon!

See below for our point by point feature breakdown!

 

This is a really great tool and Pro edition is really robust! It's great to see the added integrations from the first release.

I was wondering if it would be possible to include a very basic example of a rhythm based game in a future update? Something like a Rockband or Audiosurf where notes are moving towards the player and the player is rewarded with points or an audio cue for hitting them on time.

I realize that might be a bit of an undertaking, but if you were to include a super simple version of that it would be an incredible aid to learning, and I think a lot of potential customers might be interested in using it in that way.

 

Thanks! We're excited that you find the integrations useful! We put a lot of work into making sure that everything works as smooth as possible.

We're excited to say that the next release should include precisely this kind of demo! "How to create a Rockband/Guitar Hero-style experience with Koreographer" is one of most frequent requests that we get. We don't have a specific timeline for the next release yet, but rest-assured this is at the top of the "new features" we're working on!

 

Hello, Koreographer team! Some piece of work you've made!
I have a question - is it possible to create Koreographer events from script and not from editor?
I'd like to prepare a level based on music - I need events for beat and melody of specific pitch and I'd like to create it procedurally. I know exactly what values I need etc. I just need processed audio file dumped into table of int/float values.
Is it possible to generate it using your library? Do I need pro version for it?

 

Once you test on mobile, I'd be interested in as well. I can't take the plunge until you do.. Also, I read the docs on Fabric integration and don't really understand the integration. Perhaps a little more detail or some kind of video walk through would help.

 

Thanks!

Yes, this is very easy to do. You can create your own Koreography, KoreographyTracks, and KoreographyEvents - nothing about this is locked behind Editor code: the APIs for those classes are available in both the Editor and Runtime. The Koreography Editor (an EditorWindow) simply uses these APIs to make event generation easy!

If you wanted to do your own event creation you would simply create a new KoreographyEvent (a pure C# object) and specify a Start (and optionally End) sample time, optionally create a new Payload (if I'm reading your question correctly, you'd create a FloatPayload - also a Pure C# object), add the Payload to the Koreography Event and then add the KoreographyEvent to a KoreographyTrack. The KoreographyTrack would then be added to a Koreography which you could load into a Koreographer component for playback. You could prepare the Koreography and KoreographyTrack beforehand or even generate them at runtime if you didn't need the data serialized.

Yes! Easily possible. And no, Pro is not required for this!

 

We have tested the engine on iOS and have not come across any issues whatsoever. Users have reported success on both Windows Universal (including mobile) as well as Android (although we have had to point out that Android devices have serious audio subsystem latency issues that you need to be aware of during development - you basically need to provide a UI to allow users to configure a delay that you feed to Koreographer).

Here is a quote from a user in one of the reviews from Pro (no Pro-specific features are referenced):

I took the liberty of bolding the pertinent part.

We have not reported any "mobile tests" as we have not done any deep testing across platforms. We have tested the tech on an iPhone 6 internally and worked with users on Windows/Android. This is why we have not responded further [yet!] to the previous request!

Note that if you do use Koreographer and run into mobile performance issues we are extremely aggressive at squashing them. There is no reason that Koreographer should be heavy at all and we have done a lot of work to ensure that we do not needlessly allocate garbage memory at runtime.

We have plans to create some walk-through videos but unfortunately won't have them ready very soon. We'll see about getting some stuff put together sooner rather than later!

The documentation is somewhat written for people already familiar with Fabric. Have you used it before? Are you reading the docs as a new user?

I hope this is helpful! Please let me know if you have further questions!

 

Thanks for the quick response. I'm rereading the fabric integration stuff. I'm familiar with Fabric, a few days worth anyway and I've included it into my project already.

Related to one aspect of my current project, I have some music that I want to sync up, manually placing note values at specific points in the timeline of a song and want to use these events within Unity. I was thinking of just getting something like Cinema Director for $40 and plop in an audio file and then attach events at the proper place, as that asset also shows a visualized wave form to help that process. I understand that Koreographer deals with a more precise sync to the audio file samples that is less dependent on varying game fps, but I'm wondering how applicable it will be in my case, especially considering the sample sync technique may not be accurate on android or possibly less powerful ios devices. Anyway I'll likely buy your asset for some other projects that I have in mind.

I'll spend some more time with it. In general I watch tutorial videos at about 1.5x. Yours would be good at about .75x

edit: also, it says in the fabric integration docs that it's built against 2.2.2, though the current fabric is at 2.2.4.