[RELEASED] Make music in Unity with a full-featured synthesizer and sequencer - Audio Helm

Audio Helm is Released!
https://www.assetstore.unity3d.com/#!/content/86984

Audio Helm is a live audio synthesizer, sequencer and sampler for Unity. With these tools you can create musical sound effects and generative music for your game.

Links
- Manual
- Standalone Synth Editor (Free/PWYW)
- Video Tutorials

Synthesizer
The synthesizer generates dynamic audio live, no samples or recordings required. It runs as a native plugin to ensure low latency, high performance, mobile ready audio. Download the standalone synth now (free or pay what you want) to browse and create synth patches you can import into your game.

Native Synth Editor:


Sequencer
The sequencer is a tool for creating musical patterns and rhythms by playing synthesizer or sampler notes over time. You can create your own patterns inside Unity's inspector or create them live from code to generate procedural music. The sequencer supports dynamic tempo so you can increase/decrease the bpm as your game progresses.



Sampler
The sampler takes an audio sample or recording and can play it back at different speeds to create musical pitches. Using different keyzones you can create a full spectrum piano sampler. Audio Helm comes with 4 drum machines each with a separate sample bank.



OS Support
- Windows 7 and higher
- MacOS 10.7 and higher
- Linux, e.g. Ubuntu Trusty and higher
- iOS 8 and higher
- Android 5.0 (Lollypop) and higher

 

I am very interested in this. I will post again once I've had time to check out the standalone sequencer and gather my thoughts.

 

Count me in too. Extremely interested

 

I'm in too!!

 

I'd be willing to pay money for this if it was in the Unity Asset Store by Monday August 14th. (Unfortunately the deadline for my project is soon)

 

Hey good to hear you're interested but it won't be finished until sometime in September!

 

I might be able to make that work... will I be able to change the settings of the synth programatically at runtime?

 

Yep, I have code hooks to change synth params during gameplay. There are some simple examples in the assets that will show how to use them.

 

These Synth Assets are finally released! They contain a sequencer and sampler as well.
https://www.assetstore.unity3d.com/en/#!/content/86984

 

Very useful demo scenes, nice tutorials, the synth sounds great and it was worth the wait

 

@elbows If I could get that as a review I would die and go to heaven!

 

Sure, I will write a short review imminently, though I won't mention the wait since thats more relevant to long-time observers of this thread than on the asset store where it is fresh & timely

Small obscure request: Would you consider increasing the maximum pitch-bend range to 48 in the synth engine/standalone/presets? I've got a roli seaboard block and although I can turn down the range of the instruments pitch in their dashboard utility, 48 is the default for those devices. Yes this is an obscure request but the seaboard block is a lot cheaper than the previous seaboards so over time I kind of expect more people to use these devices.

Speaking of which, I will be using multiple instances of the synth plugin, one per note, so that I can use the additional dimensional controls of the seaboard to control synth params in a per note/per voice/multi-timbral way. As a quick starting point to testing whether this will work, I quickly tried 2 instances by using 2 audio mixers with helm and 2 helm controller scripts. Seems to work, but before I start adding more voices and more lines of code to my script, I thought I'd ask if you had any tips/best practice ways to handle multiple instances.

Unnecessary detail: I'm using the following to get live midi input which I then handle with my own code which takes account of the voice-per-midi-channel way the roli seaboard works (MPE).

https://github.com/keijiro/MidiJack

 

Thanks!!

I just changed the pitch bend range to 48. It'll come out in the next release cycle.

Multiple synth instances work in Unity. There is a small caveat where the channel slider needs to be different for each synth if you want to control them separately. Then in the HelmController/Sequencer, the channel needs to match the Audio Plugin. I was trying to find a automatic way to do this (still am) but Unity makes it really hard to access an audio plugin instance from a MonoBehaviour.

Are the second and third seaboard dimensions just like second and third aftertouches? Ideally I think the best option is for me to add these to the current single aftertouch setting so you could use a single Helm instance and just map those to modulate whatever params you want in the synth patch. I'll look into this for the future.

I'll have to try out that MIDI plugin as I'm thinking about adding full MIDI support in the future. Ideally, MIDI would bypass any MonoBehaviour script as I think it's tied to the game refresh rate. Need to think on it.

EDIT: That review! My heart...

 

Nice one!

Cheers, yes I already spotted the channel slider and used it for my quick 2-voice test.

I'll bore you with that detail in a later post but I suppose the main reason I jumped to using multiple instances was because I didn't see much detail about what the multi-polyphony is like in the helm synth. eg are any parameters (other than the obvious like initial note velocity) per note/per voice or do they apply to all voices that instance of the synth is producing?

 

Nice. Let me know how this goes and if you have any feature requests.

Aftertouch is the only per-note setting right now and you can change that after the note is played. That said, you would only get one dimension, e.g. pressure.

It should be pretty easy to support all three seaboard dimensions in a future release though. I'm using the JUCE framework in the standalone editor which is *owned* by ROLI, so they definitely make it easy to support the seaboard.

 

Cheers, yeah I just spotted that SetAftertouch accepts a note value so that all makes sense.

Cheers. Actually they go on about 5 dimensions but one of those is initial velocity. There is also lift, which I don't use often, which is sent as midi release velocity. Moving finger left and right sends pitch (which is why pitch bend range is so large because you can bend across the full octave range of the keyboard), stroking a finger up and down a key sends cc74 by default although this can be changed to any other cc with newer seaboard block (but not with older seaboards). And yes, pressure after initial hit is sent as aftertouch, but it is sent as a midi channel aftertouch message, not a midi polyphonic aftertouch message. Most of this midi detail is only really applicable to the standalone/daw plugin version and whilst JUCE will certainly make it much easier for you its probably still quite a bit of work at this stage. And I don't want to keep going on about this stuff on this thread as its rather a niche area and I already have a workaround that should be more than good enough for me for ages.

 

Can I change which patch an instance is using programatically or is that hard to do due to the above?

 

I'm planning on making a flexible version of this where you can load whatever patch you want from script. The dynamic patch loading isn't difficult, it's getting Unity to serialize non-native patch files. I'm still not sure how you are supposed to have files that don't have standard extensions in your game.

Right now the only way to do this is have two separate audio mixers with different instances of Helm. Then switch the audio mixer group Output parameter to the other audio mixer group. The MIDI File Loading example does this so you could take a look there.

I'll put this on my TODO list though.

 

Hey Matt, is there any kind of Timeline integration? Would be cool to compose the track over timeline

 

Oh! Yeah that does sound cool.
There's no Timeline integration right now, but I'm currently adding features useful for very long sequences.

I'll take a look at how sync them up and get back to you.

 

cool

 

Your Audio Helm for Unity Asset sounds incredible!
As a passionate audio guy and developer of Block Rocking Beats this is really an amazing Asset.
I created some real-time audio synthesis in Block Rocking Beats but Audio Helm is definitely something else.

I tried out the helm sequencer in Unity v2017.2 and adding notes via script works great!
I am having a bit of trouble removing notes via script.

The call's below do remove the note's in the helm sequencer editor window but the native plugin still seems to play them.

RemoveNote(Note note)
RemoveNotesContainedInRange(int note, float start, float end, Note ignore=null)
RemoveNotesInRange(int note, float start, float end)

Any thoughts?

The Clear() call does remove the notes in the helm sequencer editor window and native plugin

 

Thanks!

Ah! Yeah there's a current bug where you can't remove synth notes while in play mode. I have a fix that’s coming out in the next release soon.
It’s really simple if you want to fix it yourself now though. In AudioHelm/Scripts/Sequencer.cs, you just want to swap line 296 and 297. The order should be:

note.TryDelete();
note.parent = null;​

Checking out Block Rocking Beats now! Looks amazing.

 

New version is out! Version 1.1

New fetures:
- Dynamic BPM! Increasing/decreasing tempo, etc.
- Dynamic synth patch and MIDI file loading. Change patches on the fly.
- Added inspector tools and animation hooks for synth parameters
- Long sequences scroll horizontally for better full song viewing/editing.
- Inspector events for note on/off and beat events.
- Performance improvements for native synth
- Pause audio support

Bug fixes:
- Fixed bug where you couldn't delete notes from synth sequencer while in play mode
- Fixed sequencer synchronization problems. Now still in sync after pausing, on mobile, after loading hiccup etc.

Other
- Expanded pitch bend range to +/- 48 semitones

Dynamic BPM!

NOTE: If you get an "Access Denied" error when updating, you have to close Unity, delete the AudioHelm folder in your project. Then open Unity and install the new version. Unity doesn't release native plugins while running, so this is unfortunately the only way to update them.

 

Awesome update maestro, I especially like:

Keep up the good work!

 

Hey Matt,

I am working on the integration of the synth and sample sequencers in our vr experience.
The synth sequencers work great, especially with dynamic synth patch loading.

For the sample sequencer, it looks like it skips the first beat once in a while (4 loops or so).
Got the same result in your "DrumMachines" Demo.

Changing value below in SampleSequencer.CS solved it for me:
const float lookaheadTime = 0.1f;
into
const float lookaheadTime = 0.125f;

Not sure if this is the correct way

 

That should work fine!
I'll take a look at why this is happening though. Might have to do with the faster framerate in VR apps?

Either way the fix will come out in the next version. Thanks for the report.

 

Audio Helm 1.2 is out!

Here's what's new:
- Added spatialization plugin compatibility (Resonance, Steam Audio, Oculus, etc)
- Fixed drag/drop in sequencer not redrawing on Windows
- Fixed extra note off events in sequencer when deleting notes
- Fixed first beat in sample sequencer not triggering sometimes
- Fixed iOS archiving error

And if upgrading please delete AudioHelm/Plugins folder in your project *before* opening Unity and downloading the new version. Otherwise, Unity won't update the native plugins.

 

Hey Matt, I've been using Audio Helm for a few different projects now and am still loving it!

I have a small issue though where I don't seem to be able to pick up events on an "early" note of a loop, I'm using the HelmSequencer and adding notes in via a script. Loop length is 16.

The audio always plays but the event never gets sent out for the opening note. Eg, note.start = 0 (note.end = 1 in my case).

If I add around 0.1 to the start time of the note then I receive the events.
If I add on 0.05 I get the events about 50% of the time.

The audio always plays though. I've been looking for what could be picking up the notes and playing them but can't find the bit of code.

Thanks in advance

 

Thanks for the report. I've got a fix coming out in the next version!

 

Brilliant, thank you, great news there's a new version on its way!

 

Hi, first of all, thank you for this great plugin.

Unfortunately I have just been having some issues and couldn't find any information about it.

I am trying to load a patch at the beginning of a level with LoadPatchData. It works as it should in the editor but on android I'm having some issues. It seems to put everything into a jar and therefor it can't load the file with a path.
This is what I am using: https://docs.unity3d.com/Manual/StreamingAssets.html

On android you need to use something like WWW to unzip the data.

 

Yeah Unity wants to serialize everything into the build and they make it difficult to access raw files on different platforms.

I created a tiny script that will serialize a patch into a game object so you don't have to load the actual file. In the LoadPatchesAndMidiAtRuntime demo, you'll see three patches in the hierarchy each with a single patch inside. You can then pass this to LoadPatch in a Helm Controller.

Hope that helps!

 

Thank you for you quick response mtytel.

It works great. Took me a little while but I managed to use the example to add it to a scriptable object.

 

Hi,elbows.Did u mean u have handle how to load the MPE signal from the roli seaboard in Unity? I don't find a lot resourses about that.Could u give me some suggestion or reference?

 

Hi,

I dont remember every detail of how I did it, but it certainly involved a little bit of code writing by me, and understanding MIDI messages and how MPE MIDI messages work.

I know I used this to handle midi input:

https://github.com/keijiro/MidiJack

But it does not seem to support midi channel aftertouch messages (which are an important part of MPE). So I must have added that feature to Midi Jack myself. I will see if I can find my code and will post again about this bit later.

Then, I wrote my own code to do things with incoming midi, by using the Delegates, and by assuming that each incoming note will be on its own midi channel. How that affects your code etc in Unity depends on exactly what you want to do with the MPE messages, what you want them to control etc.

 

Thanks for your reply.That would be very helpful I also find this "MidiJack" project.But as you said,it can only get the midi on off and velocity when i press a single key.I think it's very import to get the aftertouch message like the pressure and the pitch between different note.I am not sure do u have a plan to put it in AssetStore.If u do I would certainly support it.

 

No, I dont have any plans to do my own midi plugin and put it in the asset store.

Anyway I dicovered/remembered that somebody else added pitch bend and aftertouch support to Midi Jack. Its not been merged into the master branch on github, but all the code is in the pull requests:

added pitch bend interface and examples

support after touch

Standard Midi Jack already tells you what channel the messages are sent on (and with MPE each note is on its own channel) and it already supports cc messages, so the cc74 part of MPE is already covered. So with these and the above code additions, you should have everything you need to read MPE messages in Unity.

 

Thanks elbows.The link it's very helpful and easy to use!

 

Hi! This asset looks really awesome. Just a question before buying... do you support Midi Out to external devices? In other words, could it be used to build a sequencer for external midi devices on iOS? By "external" I mean hardware devices (synths, drumboxes and so)

thanks!
Javier

 

how can i get the normal piano preset
i just want piano sound，so i can play piano