G-Audio Tools( GAT ) - Releasing soon, suggestions welcome

New clean support thread here:
http://forum.unity3d.com/threads/223729-SUBMITTED-G-Audio-2D-Audio-Framework

Webplayer: Little tour of the 6 example scenes provided with G-Audio. All scenes feature abundantly commented code. Built with Unity Free.

What is GAT?

G-Audio Tools is a 2D audio framework that enables much lower level control over audio playback than Unity's audio API.

What does it do that Unity doesn't?

-Real time filters for Unity Free: low and high pass and shelf, peak and notch, distortion and LFO already implemented. Filters can be controlled in realtime, per playing sample or per track, or applied to audio data for cached pre-processing of samples.

-Sample pre-processing: fade-in, fade-out, reverse, normalize or pitch shift whole samples or chunks, accurate to a single sample.

-Stop playback of a sample without adjusting volume : smooth stop in less than a frame.

-Full panning control: need a sound to travel from one speaker to another in a 7.1 system? GAT has you covered.

-Next to zero garbage collection: creating and destroying AudioClips on the fly can create heavy GC spikes ( and framerate drops ). GAT does not use AudioClips and pre-allocates memory so that garbage collection is kept to a minimum.

-Automatic mixing: all played sounds are mixed on one single AudioSource. Just tell a sample to play, it will never cut playback of another.

-Route samples through tracks ( inspector friendly, apply an effect to more than one sample at once ), or play them directly.

-Pre configured FFT for spectrum analysis


Who is it for?

-Intermediate to advanced programmers who need more control over audio playback

-Anyone who's making an iOS app where audio is at the forefront

-Anyone who's ever tried to make a sampler or a sequencer in Unity

When is it coming out?

-v1.0 is waiting for review by the asset store team.

GAT is fully compatible with Unity's Audio API: you may use it along with standard Unity audio playback.

 

Sample processing sounds interesting for use in tracker music to keep size down, is that possible?

 

It sounds like a really useful tool. I've had a lot of issues with audio in unity, mainly because my project requires me to load audio files from memory, which is something unity has little to no support for. My current setup uses NVorbis and feeds in through a set of AudioSource filters, but it's very unreliable and I'm definitely looking for alternatives. I imagine this isn't the purpose that GAT was designed for, but I'm interested if it would work for this situation.

So I'll ask: Which platforms does GAT run on? Additionally, can you play a clip (like an .ogg or similar) from memory or disk via a memory/file stream or byte array?

 

Can it's work with any 7.1 or 5.1 devises.

 

Hi Woodlauncher,

Unity already supports tracker modules - http://docs.unity3d.com/Documentation/Manual/TrackerModules.html. Do you mean to ask if it would be possible to add sample processing to playback of .mod files? If that is your question, I really don't know, I'd have to look into it.

Cheers,

Gregzo

 

Yes it can, plus you can easily control how much of a mono sample is fed to any channel.

 

Hi Doddler,

At launch, GAT will only support uncompressed pcm samples. It's purpose is real-time sampling / sequencing etc... so compressed audio would mean too much overhead for these use cases. It does have classes to load and unload pcm samples from Resources or from disk, and manages a configurable memory buffer so that all allocations are super quick and the garbage collector never kicks in.

Platforms: all, as GAT is fully written in .net.

 

Sounds cool! We would like to know - will this compete with our product Master Audio? Or is this something where people might use both at the same time? So far it seems the latter. We don't really add features to Unity audio that aren't already there.

 

Hi Jerotas,

In many ways, GAT sits at the opposite end of the spectrum compared to Master Audio. It is lower level, doesn't feature any event-like tools, and is aimed at music making in Unity. I wouldn't use it in a tradditional game audio setting, but couldn't do without it for all my generative music projects.

So no, not a competitor, and yes, both could be used together. GAT does all it's mixing on one single AudioSource via OnAudioFilterRead, and provides the client with reading callbacks allowing real-time processing or analysis. Samples can be pre-processed on the main-thread and cached, or manipulated directly on the audio-thread. It is about more control, not more ease of use!

 

VERY awesome! Definitely on my watch-out list.

 

Yes, that is what I meant. I haven't looked into it very deeply so I don't know if maybe it's already supported in one of those formats, but if you can pitch shift samples that would really help keep filesizes down.

Unless there is some big flaw with the tracker formats/tracker music in general that I'm missing I do not see why anyone wouldn't use tracker music for games).

 

From the unity docs ( link I provided earlier ): Tracker module files differ from mainstream PCM formats (.aif, .wav, .mp3, and .ogg) in that they can be very small without a corresponding loss of sound quality. A single sound sample can be modified in pitch and volume (and can have other effects applied)

So it looks like your covered already.

GAT is more meant for real time stuff: instruments, generative music, etc...

 

This is grand as a concept, certainly a pal would make this and i see no reason why music generation cant be game based. One of my fave asset buys was for audio analysis, and driving values based of frequency and so on, would this be suiable or is it chicken and egg anyways as you're driving the creation yourself. I havent seen much updates to the old asset which is sad and i'd love something that was aimed towards synthesis (although not so much a personal definite buy, my friend though, hed go mad for it) but is there any aspect of analysing the spectrum for driving values? I have an example whih i wont post unless asked that reflects it and my pal on seeing it said 'wouldnt that be good in a slightly different situation mixing synthesis too, we made plans but bugger all idea how to synthesize the sounds as it stood. So yeah, i dont know if its in your remit or the product or its even relevant but just wondered if it would be considered s future feature or it would be expected for the user to supply such functionality

 

Hi lazygunn,

I've already dabbled with spectrum analysis ( FFT ), and have got a pretty fast C# implementation already running. Any sample that you play, you can ask for a copy of the data before it plays and draw it, fft it or whatever suits your needs.

FFT is quite resource intensive, though. Better to do it only on the final mix, which certainly is doable.

 

I'll be checking it out for sure when it's out. Thanks for the info!

 

Ahh you know, i dont think it was even as heavy as that but thats interesting, well assuming FFT is the same FFT driving my 3D ocean waves in concept (i googled just then haha, fast fourier transform it is) - and in that regard (Given i've seen already GPU processing becoming big in audio analysis), does it make sense, given any bottlenecks, to have this on the GPU - i'm only just learning all this stuff after a lifetime of keeping my 'art' away from heavy programming concepts, so forgive ignorance, but I am curious, especially using and learning about compute in different fields.

 

Moving fft calculations to the gpu is not really doable without severe headaches and compatibility issues in Unity...

If what you need is frequency band values in realtime to have graphics react, a simple FFT of the final mix is plenty enough, and is perfectly acceptable performance-wise even on mobiles. Bear in mind that as the audio buffer is, by default, 1024 samples long, fft will not give you a super high frequency resolution. But enough for most use cases...

Cheers,

Gregzo

 

Righteo! Was just curious, main compatibility thing I think atm is its a no go with anything cept directx (And definite headaches getting anything to the cpu) but I find the topic of interest anyways. Next time my pal pops up i'll suggest he look at this thread anyways

 

I'm currently making a vid to showcase the basics of the API. Should be available tomorrow at the latest...

 

Quick update:

Working like a mad lemur today, implementing a higher level API with nice and comfy Unity components for users who don't want to join me in the rustling of leaves among the intricated foliage where I dwell.

Good news, it's really useful!

-Sounds can now be routed through tracks, which have gain, pan and effects applied to before mixing.
-As many tracks as you can muster without crashing your machines!
-Sounds can still be played on their own, with their own filters and pan control. You can also have a sound with one filter routed through a track with another.
-Found a cute little simple lemury distortion on the web, implemented it. Nothing fancy, but super fast.

Comments welcome,

G

P.S.: vid will come when I have the higher level API ready, probably tomorrow. Made one during the night, only to realise I really needed to work on some higher level controls...

 

Hi gregzo

Looks like an interesting audio framework you are developing.

Can I ask a couple of questions...

Does it support 3D audio specialization (i.e. attenuation curves, Doppler) or is it only 2D?

Any idea on the number of voices that will be reasonable to have active and what is their CPU usage?

Is there any voice virtualization available?

Just out of curiosity, why do you think that Unity's existing audio is not good for music?... I am adding music support to Fabric and so far I haven't experienced any issues when using PlayScheduled/SetScheduledEndTime ... I may be missing something so it would be interesting to know why you think that and how this framework will make the process more easy to implement.

Thanks

Taz

 

Hi Taz,

Had a peek at Fabric, looks great!

One by one ->

-3D : Nope, Unity already does it quite well. Audio in GAT is played via a GATPlayer, which itself feeds an AudioSource. Distance calculations are still performed by Unity, so you can use reverb zones or distance adjusted lowpass ( Pro ) on the whole player, but panning is unaffected by player position. This could easily be done, though... simply adjust the GATPlayer's global pan to the angle between the AudioListener and itself.

-Performance: To remain fully cross platform and compatible with Unity's audio engine, I'm doing all processing in .NET. Optimizing as much as I can, but of course it's more cpu intensive than the most probably C++ backend Unity relies on(FMOD).

But! If you need to chop up or process samples on the fly, GAT can do it with 0 garbage collection, using a pre-allocated buffer: no AudioClip.Create + no new float[] + no GetData or SetData = no garbage collection.

Tested mixing 50 44.1khz samples on iPad 3 yesterday, 30% of cpu usage imcrease. It's an extreme case, 50 simultaneous, independently panned mono samples, but there you go. In typical use cases ( 10 overlapping sources ), I'm getting much more reasonable usage ( 5-10% cpu ). Will have more detailed results soon. I've also tested adding 40 mixed samples to Angry Bots, no frame rate drop - a lot of the expensive calculations( rendering ) are done on the GPU so processing audio on the CPU doesn't affect performance so much.

-Unity : Do try to build a sampler or a sequencer using AudioSources and AudioClips! You'll see quickly it simply isn't feasible.
You mention using SetScheduledEndTime - it's the perfect example : it simply cuts plaback of the sample, resulting in a click. What if you need to stop a clip in less time than it takes to fade the AudioSource's volume to 0? You can't just fade the data in the AudioClip. You need a copy of that data, patch the fade in, and restore the original data once you're done. But what if the clip is being played simultaneously by different sources, which don't stop at the same time? Endless headaches, lots of spaghetti code, and fragile, bug prone results. Unity's audio is great for game development, not so much for music making.

Another example: you need to draw audio data on a screen in a virtual room, just a line representing the floats - if you simply grab the data from OnAudioFilterRead and draw it, it will be already distance attenuated and the amplitude of the drawn curve will decrease the further away you are from the AudioSource. And what if you want to draw the output not of one, but of 5 combined AudioSources? You'll have to mix the data anyway to draw it, whereas GAT provides you with thread safe copies at any level ( per sample, track or master ).

And last, multi-threading issues: for most Unity users, all the implications of OnAudioFilterRead being on a seperate thread are not clear at all. GAT provides thread safe access to the audio data, wether for analysis or processing.

To sum up: for most game development situations, GAT is not the right tool. It's the right tool for me, and I hope it can be for a few audio minded devs out there!

Cheers,

Gregzo

 

awesome.
can't wait. When you release this tool.
Plzzzz make one sample or Example for 5.1 or 7.1.

 

Sounds interesting. A demo or video highlighting the specific advantages this has over standard Unity audio processing would be useful.

 

Hi gregzo,

Thanks for answering my questions..

Considering its written in .NET 5-10% CPU usage for 10 sources sound reasonable to me... I get similar usage with Fabric's sample player and modular synth components... have you tried using Mono.Simd to improve performance?

I agree that Unity's existing audio isn't capable of making a sampler or a DAW like sequencer (though its there in FMOD just not exposed through the API) but creating an interactive music systems that requires sample accurate transitions (on a beat/bar or even marker points for wavs) between music segments its been relative easy to implement... Perhaps this is because Fabric already provides a number of audio features that I don't have to worry about (i.e. event system, different types of components, parsing wavfiles, audio source pooling and component instancing)

I like what you are trying to achieve with this framework and when its out I will certainly take a look at it.

Cheers,
Taz

 

Hi Tazman,

Thanks for the Mono.Simd tip, wasn't aware of it. I've researched it a little, and from what I've found might only work on standalone builds... Do you have any experience in integrating it to Unity? If it's not Pro only and is deployable on mobile platforms, I'll definitely give it a shot.

Just spent a few hours debugging a very annoying Unity issue - drop down menus in the editor ( enum fields in the inspector, settings, etc... ) suspend the main thread but not the audio thread, and this suspension doesn't fire OnApplicationPause nor OnApplicationFocus... Anyway, GAT now comes with a work around. Yippee!

Demo video very soon, just finished a set of modular components that enable doing a bit of music without code, more fun in a demo than arid code.

Tomorrow, I hope.

Cheers,

Gregzo

P.S.: Are you Tasmanian? Just went there last year, amazing place.

 

Can it recognise multiple sound devices, display them and allow the user to mix the signal from these seperate devices?

 

This sounds like a great plugin, am looking forward to trying it out. One thing I could imagine using this for is to produce music and visuals in real time. I noticed that you have a Multi-player plugin, any chance of a Multi-player example for making music cooperatively?

 

Nope, sorry! This is native plugins territory, and my first aim for GAT is to be fully multiplatform and Unity free compatible.

Cheers,

Gregzo

 

Visuals: yes indeed! I've already got that factored in. You can derive from GATChunkCopyClient and get realtime copies of audio data as it is played, per sample or per track. FFT to come in v1.1 - got it working, but need to draw a line somewhere for release.

Multiplayer: I've done a lot of work in synching iOS devices for cooperative music. For now, I'll keep that to myself, as my next project heavily relies on it...

Doing the first vid tonight, hopefully edited tomorrow.

Cheers,

Gregzo

 

Hi to all,

Very happy to report that, after a week of sleep deprived nights and sadu like eating behaviour, I'm ready to show GAT.

-Performance is surprisingly good: 5% increase in CPU load on an 3.1ghz i5 iMac whilst mixing 20+ mono 44.1khz samples to stereo output, routing them through 4 seperate tracks, each with a different filter( LowPass, HighPass, LFO and distortion ) and some live resampling too. And making copies of buffers for drawing an fft of one of the tracks, and an audio curve of another.

-Still fully Unity Free compatible, still fully multiplatform.

-Demo vid is being uploaded as I write and features a peak at the API.

-Will post webplayer of the scene in the vid if requested.

Looking forward to your comments,

Gregzo

 

Hi gregzo,

Its simple to integrate especially if your plugin comes as a DLL... Haven't done much with the library though, just tested it on a few examples to see how it works but they were all done on the PC and from what I can see as well its not supported on mobile platforms. Still you can get some good performance improvements if your main platform is the PC.

Haha no I am not Tasmanian... just a much easier nickname for people to remember than my full Greek name

 

Video took ages to upload, and I fell aleep...

Here it is ( and on the OP ):

 

Little heads up:

Will submit to the asset store hopefully before next week.
Cleaning up: completely reworked the inheritance to allow playing of any float[]. Simpler, slightly faster, more extensible.

Now's the time for last minute comments / requests!

Pretty happy with benchmarking so far, looks like GAT might be a viable 2D alternative to Unity's audio API.

Summing up the GAT Package:

-2d audio player, play any float array through tracks or directly, with panning helper objects or mixing callbacks

-Filters: LowPass, highPass, notch, gain LFO and distortion, which you can add to individual samples or tracks, and which you can control in real time, via code or inspector components

-Main thread pre-processing of samples: chop up, resample, fade in, fade out, reverse and normalize chunks of a sample on the fly. Automatic caching of processed samples for high efficiency.

-Memory management: minimize garbage collection to next to zero with GAT's GATDataManager. Simply set how much memory you'd like to set aside for audio, and GAT does the rest.

-FFT: configure fft components for easy spectrum analysis of your data, per track or per sample. Hamming and Hanning windows available, frequency ranges can be set to optimize calculations.

-No dlls, no native code : fully compatible with Unity Free and with all platforms.

Introductory price : 30$

All comments appreciated,

Cheers,

Gregzo

 

Added webplayer to the OP:

Webplayer: 7 piano samples processed in real time.
-The number at the top right shows how many samples are currently mixed.
-Press the green button to trigger a distorted and pitch shifted piano sound - sounds may overlap. The black line draws an fft of all currently playing sounds.
-Drag the blue thingy to change the lowpass frequency on the bass ( biquad filter ). The blue line draws the audio buffer for that track.

The package will now also include a realtime adsr envelope.

Asset store images done, releasing very soon.

G

 

Nearly done, some flu related delays, and some more to do things properly!

Asset Store key images are ready, documentation is progressing, 6 example scenes are being commented.

Regarding documentation, the plan is to provide the following:

1-Online Doxygen generated html ( with detailed comments ).
2-Six simple example scenes illustrating key aspects of the framework, with abundantly commented code acting as tutorial.
3-A UnityAudioTrivia.txt to reference little known audio facts and prevent GAT unrelated support.

Let me know if you feel there's anything I must consider!

Many thanks,

Gregzo

 

Looks good! One technique I use quite a bit in Reason is using low frequency oscillators as a trigger for other sounds/tracks or effects. I would imagine this is possible by looking at the audio buffer being drawn on the bass. If you had an example implementation, that might would be nice! If not, I am sure that is one of the first things I'd work on.

Planning to purchase this for sure! I can't stand the 'popping' that I always seem to get when using Unity's audio, and it sounds like your implementation addresses this.

 

Hi Voronoi,

LFOs are a great idea. I've already implemented a gain LFO, and will look into affecting other parameters too( filters, pitch etc... ). It's on v1.1's todo list, for now, I have to focus on releasing otherwise I'll just keep implementing new stuff for ever.

About the 'popping': should not happen, at least not in builds. In which circumstances are you getting pops?

I'm also including a Unity Audio Trivia .txt in the package, that references little known facts about Unity Audio. Nice to have them all in one place.

I will post a new webplayer tonight that shows the 6 example scenes acting as tutorial of the framework.

Edging nearer!

Cheers,

Gregzo

 

Webplayer: Little tour of the 6 example scenes provided with G-Audio. All scenes feature abundantly commented code. Built with Unity Free.

OP updated. Submitting this week-end.

Cheers to all,

Gregzo

 

My reason for being interested in GAT would be sample accuracy. I made a sequencer in Unity which plays samples on regular audio sources, but PlayScheduled has an audible glitch that gets in the way (reported, so hopefully it'll get fixed), and it's problematic that volume/pitch/pan changes can't be scheduled which prevents e.g. tempo-synced arpeggios reacts to what happens in the game.

If every operation can be sample-accurately scheduled, I'll definitely get GAT.

 

Hi Nifflas,

I think you'll find that G-Audio will solve your sequencer issues, and make your code much simpler and more readable.

Scheduling sample accurate gain/pitch/pan changes is a use case I hadn't thought of, good idea! It's doable with the classes G-Audio provides, but not out of the box. I'll put it on the future features list users can vote for.

Cheers,

Gregzo

 

Okay! I might still be interested if I can just sample accurately start a sound with a specific pan/pitch/volume. Still, sample accurate modifications would be amazing

 

You can accurately start a sample with any pitch, gain per channel or pan( if stereo ), envelope and filter you want. You can even have GAT automatically cache resampled copies of your samples if you need to. Example code:

Code (csharp):

1. public GATResamplingSampleBank sampleBank;
2. public string sampleName;
3. public bool cacheData;
4. public double pitch;
5.  
6. public void PlayScheduled( double dspTime, float stereoPan, float gain )
7. {
8.     IGATProcessedSample sample = sampleBank.GetProcessedSample( sampleName, pitch, cacheData );
9.     GATFixedPanInfo panInfo = new GATFixedPanInfo(); //DynamicPanInfo also exists if you need pan to change whilst playing )
10.     panInfo.SetStereoPan( stereoPan, gain );
11.     sample.PlayScheduled( dspTime, panInfo );
12. }

In most cases, one would play samples through a track in one line of code, but G-Audio is very flexible and can let you in at many different points in the playback process.

 

Doesn't that code process the entire sample and applies the pitch before it starts playing, instead of resampling in realtime? Isn't that a bit heavy for long samples? Plus, I do a lot of realtime pitch slides, and would need that ability in GAT too for it to replace Unity's audio sources.

...or maybe I'm just misunderstanding how it works.

 

Yes, absolutely. If you do not need to cache the sample, you can resample in realtime too:

Code (csharp):

1.    
2.     public GATSampleBank sampleBank; //No need for a resampling capable bank if you're not caching
3.     public string sampleName;
4.     public double pitch;
5.     GATRealTimeSample _realtimeSample;
6.      
7.     public void PlayScheduled( double dspTime, float stereoPan, float gain )
8.     {
9.         GATData data = sampleBank.GetAudioData( sampleName ); //unlike AudioClip.GetData, no copy or conversion occurs here.
10.         GATFixedPanInfo panInfo = new GATFixedPanInfo(); //DynamicPanInfo also exists if you need pan to change whilst playing )
11.         panInfo.SetStereoPan( stereoPan, gain );
12.         _realtimeSample = new GATRealtimeSample( data, panInfo, true ); //true for canPitchShift
13.         _realtimeSample.pitch = pitch;
14.         _realtimeSample.PlayScheduled( dspTime );
15.     }

Note that pitch can then be adjusted in realtime. You can also querry the sample's playing status ( waiting to play, playing, done playing ), add realtime filters, and stop the sample elegantly without clicking.

 

Cool! Thanks for clarifying!

 

No problem! I worry that people might not understand what G-Audio does that Unity's audio doesn't, so it's nice to be challenged and to provide examples.

 

I think people like me who really need this will understand its purpose too I'd be happy to upload some GAT usage examples when I've got my hands on it.

 

Great!

Just need to polish documentation and format the code like a maniac, and off we go. It is a big project, so just that will take at least a couple of days. Then, a few days waiting for review - hopefully releasing before next Friday.

Cheers,

Gregzo

 

A user has kindly reported severe audio glitches in Chrome, Windows 8 64 bit.

Unfortunately, I will not have access to a PC to test on until the 27th of January, and the glitches have never happened on my mac, standalone or webplayer, Chrome, Safari or Firefox...

If you've experienced glitches, or if you have a PC and are not experiencing any trouble with the WebPlayer, I'd be very grateful to hear about it!

Webplayer link.

Many thanks,

Gregzo

P.S.: Docs nearly done, will submit Sunday for sure. G-Audio might be out by the end of next week.