[WIP] resonance.RT

hey all, just wanted to show a brief teaser about something i've been working on.

by all means, ask questions or leave comments. let me know what you think

UPDATE:
This is a new video using 4.6 free and resonance. It shows what the jet airplane, propeller airplane and car sounds like using resonance. If you have any questions or comments etc, please don't hesitate to post them

btw, if you just want to skip past the explanation of the parameters etc, the first actual demo starts at 8.00 minutes in..

 

Hey folks

If you're new to the thread, please read this first before watching the video above. Thanks

It occurs to me, that I haven't done my best explaining what resonance is, or what it does. I will attempt to rectify that now. First, a few points:

• If you are happy with your current process of finding decent, loopable sections of an audio asset, in some cases could be as many as 20-50 loops/cuts per 1 finished asset, as well as finding an appropriate editor, and cutting up, implementing and maintaining those assets just to end up with something that has to be pitch shifted and fading in/out, to approximate the frequencies between them, then read no further.
• If you are happy with the current triggering semantics, such as they are, and don't have a case where it'd be nice just to have the audio literally fire an event to let you know you have hit a marker or region in the audio to perform some other process, then read no further.
• I liken the audio part of a project to a bass player in a band. You really don't hear a great bass player, you just hear the great bass in the song, but a bad bass player? You hear him immediately. Audio, sound effects and music in general in games is often an after thought in my humble opinion. Folks focus primarily on the visual eye candy or physics, and then pitch shift their audio assets so that it looks good, feels good, but sounds like a bad 1980's Mario game. But hey, if that's your thing, then read no further.

If you're at this point and still reading this, then the assumption is that I've pique'd your curiosity a bit. Good

Now for some technical stuff. If you don't care about this part, then you might wanna skip this section

#region Tech Stuff

From here on out, I will explain things from the point of view of creating a racing game and creating believable/realistic engine sounds and my odyssey of trying to solve it. This topic, believe it or not, is actually extremely difficult to achieve and there are only a handful of games/developers that actually pull this off in a believable manner in my opinion, and even then, their solutions are mostly in-house or cost major dollars to license, and that's even *if* they license it. CodeMasters and their F1 series comes to mind. Another might be iRacing etc. There are others, but i won't list them all. There are a few different current solutions that I've tried:

• The basic "find 3-4 loops method". This was my first approach, and while it works, man does it sound nasty to me. When i stomp on the accelerator as-it-were, or just accelerate in general, i want to *feel*/hear the power while I'm doing it, and that definitely wasn't the case.
• FMOD's engine designer which I initially tried to use, and was cool, but I found it extremely fiddly, time consuming and ultimately frustrating, trying to find decent loopable sections in the source asset, at the frequencies I wanted, for the durations that I wanted, without sounding like they were cut. And believe it or not, there are just some audio assets where this is next to, if not actually, impossible. And even then, I imported them into the designer and had to spend yet more time adding them, fiddling with lining them up, fade in/outs etc etc. I must've spent several hours/days tweaking and trying to get it all sounding right in the editor. Funny thing is? Up until that point, I never really thought about how to get all of that work into Unity and fully realized. I found a few small non-detailed tutorials, and after looking at what the requirements would be, and the licensing: http://www.fmod.org/sales/, I ultimately gave up. I like to consider myself fairly intelligent, but man, don't developers try to actually *use* the stuff they intend other folks to use?
• Rev by CrankCase audio. Cool little demo, and it says that it's for Unity on their site, but I got no responses to my emails asking for an actual Unity demo or trial or anything else. Add to that, the fact that as of this writing, the licensing is daunting. https://www.audiokinetic.com/licensing/pricing/ There was licensing the to consider and additional packages to install, namely WWise from AudioKinetic, and all with limited documentation and tutorials on how to do so in a relatively easy manner. Maybe it's been updated now, I'm not sure, but even if so, the licensing makes it a complete non-starter for me.
• AudioMotors by AudioGaming. Pretty cool, and very similar to Rev, however again, nothing on how to actually get anything into Unity or even if they support it. Evidently, AudioGaming and FMOD are somewhat partnered, but i didn't even see AudioMotors offered for Unity at all on FMOD's site. The demo is cool, but extremely fiddly, especially when trying to set the ramp. Factor in the price/licensing, and it was another non-starter for me.

And that's it. It was all i could find after *literally* weeks of searching. Now, that all being said, I think each offering have their good points and their bad points. And I'm not naming them to trash them in *any* way, it's just that they didn't live up to what I wanted/needed in my personal experiences.

So. What to do now? I just couldn't let it go. There *had* to be a way to:

• Play an audio asset, in this case a car engine, *without* losing it's fidelity. Every engine has it's unique individual harmonics, and I don't want to lose those by having to pitch shift loops and thus ruin it in my opinion.
• I didn't want to have to spend umpteen hours prepping and cutting and searching to find acceptable loops, only then to have to try to manage those individual assets not only during development, but deployment and updating later as well as the many known and unknown limitations of playing them back in Unity.
• I wanted to somehow, synchronize an 'accelerator pedal' or physics engine to the audio. Especially when the pedal is 'flat' or at a sustained speed, how to loop *that* at *any* place in the audio without compromising the first few points?
• Needs to be *fast* and accurate.
• I don't want to have to lug around a bunch of monolithic audio assets. Is there a way to easily dynamically generate or play just certain sections of them as well? And if so, it'd be really cool if i could somehow get events from the audio itself to use as triggers for other things.
• K.I.S.S. (Keep It Simple Stupid) or at least as simple as possible. This is also known as developer laziness, of which I am fully guilty. I've spent weeks writing an application to perform an automated process, that only takes 5 mins to do, because in my mind, "I might need to do it again at some point in the future". I think all developers have that mindset to a certain extent as well as the abhorrence of creating assets to begin with. *Any* kind of asset. If that weren't true, there wouldn't be a name for it. It's called "developer art". LOL. Be honest. We tend to think in terms of place holders instead, right?

Still with me? Because if you're anything like me, a technophile, *even a little bit*, then I'm about to blow your mind. I'm going to reveal what I think is one of the greatest kept secrets in gaming, let alone audio. And believe me when i say, I *know* I'm not alone with my list above. I've seen all of the posts on this very forum, let alone other places, asking how to do each thing.

Ready? Asynchronous Granular Synthesis. Yeah, I know. It's a mouthful. And I'm sure at this point, some folks's eyes just glazed over, but please, bear with me. It's about to get good

What is it? Well, basically it's a way of defining a specified FFT (Fast Fourier Transform) 'window' or section in the audio that incorporates a group of minute sized individual sections called 'grains'. It then layers these grains based on some parameters such as density, length etc etc. The following links are part of the many, MANY whitepapers I've read over the last year, and will explain it with much more detail and clarity than I could offer:

http://www.media.aau.dk/~sts/ad/granular.html
http://www.sfu.ca/~truax/gran.html
http://www.music.mcgill.ca/~gary/307/week4/granular.html
http://courses.cs.washington.edu/courses/cse490s/11au/Readings/SynthesisChapt5b.pdf
http://www.cs.au.dk/~dsound/DigitalAudio.dir/Papers/BencinaAudioAnecdotes310801.pdf
http://www.camil.music.illinois.edu/Classes/404A2/2/granular.html

Cool. Did you get a better understanding of what the tech actually is? The last link explains it best in my opinion because it's a bit more graphical

#endregion Tech Stuff

Did you also notice the mind blowing secret?

*This technology has been around forever* AND it meets ALL of the criteria on my list, and ALL while the audio is *actually* playing. Now, I've been not only a developer for about 30 years, but a musician as well. I've played guitar for almost 35 of my 50 years. I've used Sonar as well as ProTools to record. and engineer and they *always* had the coolest patches and plugins, but I never made the connection. Even though they were running real time, because I was recording, I never thought of them as 'real time' in the same sense that I do when talking about games. And then it hit me. *No one*, was really taking advantage of this technology. Not really. And the *ONE* place that could benefit the most, would be games. If you trust it to record with, then why wouldn't you trust it to play audio in your games with?

The possibilities started to hit me. Separating time stretching from pitching. Dynamically changing bpm in just a section of the playing audio without changing the rest. Not just looping, but dynamically define them while the audio is playing. On and on. Literally. For a few days, I simple let my mind gorge on the possibilities.

My absolute first thought after my epiphany? How the hell can folks not know about this? Am I really the only one that didn't know? Why hasn't someone built something actually *usable* in a game engine with it? And if they have, why the hell isn't there an API/SDK to take advantage of it? Are my google skills seriously *that* bad?

The answer to one of those questions: is a big emphatic 'NO'. The big boys have not only known about this tech, but remember those in-house solutions I mentioned above? Remember the big licensing prices? Some of them use it. I'm pretty sure the World of Tanks guys use it. Listen to the tanks next time you play. I'm fairly certain that the CodeMasters guys use it. Listen to the engine sounds of F1 2013. How are they able to cycle, and sustain, the corresponding engine audio to *exactly* the positions of the throttle? Even if I hold the accelerator at the same spot? I hear no discernible pitch shifting. you're trying to tell me that they prepped 100's of clipped loops? *per car*? But here's the thing: After spending a year researching, writing code and testing. It doesn't matter what they use. Shrug. who knows, maybe i stumbled upon something unique. Again shrug. All i know is that it works. And it works well. And not just for engine sounds. This tech works on all audio. Ambient, cyclical, you name it.

I've spent a considerable amount of time fine tuning, performance tuning and testing. Writing code to get it into Unity. Writing code to get around limitations or problems IN Unity.

Let's be clear.
resonance.RT *isn't* a clip player. It *isn't* a full featured audio solution. It *isn't* a mixing solution. It *isn't* really a fliter either. What it is, is a multi-threaded DSP audio component built specifically to BE a granulator. Not as an afterthought, or as some squeezed in capability, but from the ground up. It performs it's calulations, in a multi-threaded ring buffer, and then when an audio source, be it FMOD, G-Audio, MasterAudio (great assets by the way) or any other source or asset, requests a buffer, they get back a fully granulated buffer for that position in the audio.

I built this with the intention of using it with ambient effects such as wind, or ocean shore lapping sounds, or explosions, or engines; be they car, race, motor cycles, tanks, helicopters, propeller planes, jet engines etc etc ALL with the thought that I would want them to be able to be varied, in code, at run time, in response to what happens in game, without having all of the clipping, looping, prepping, managing, deploying or updating potentially 100's of monolithic files.

Try doing all of that with static audio assets.

Ok, so now you know what it is in that video above. If you haven't watched it, shrug, give it a shot. Few more minutes won't kill you

Again, my apologies for not explaining or being clear. It was my wife, this morning, that explained it to me and made me realize



Btw, kudos and thanks must go to gregzo. You've helped tremendously whether you know it or not. Your posts, comments and answers have helped me to not only fix, but avoid or work around tons of issues.

 

I don't get it. Am i posting in the wrong forum section or something? LOL bad breath mebbe?

100+ views, no comments, questions or even flames?

 

Hi there! that's cool! It's a grain sampler!!! woah! nice once. When a video isn't engaging people, then, probably have to make different video! yes it's true the video started with some boxes and it was grey, the mic could be higher frequency and crazy voice . I'm an audio fan and saw "i have made a grain sampler audio effect" i would be right on it!!!

So... it's a grain sampler / granulater with lots of parameters and a full on wave editing display! awesome!

Humm... your follow up post is perhaps very long! mostly from what i know a grain sampler has random position jitter for where playback position is, random grain length, pitch, resynth, so on, very nice
http://en.wikipedia.org/wiki/Granular_synthesis
http://www.native-instruments.com/index.php?id=graincloud2

They are very good at ambient background noise as well as controlling engine sounds, they can do reverbs, and grain delays, which is the favourite effect of squarepusher. Grain FX are Awesome for all audio tasks, and there isnt any grain things on the asset store. They make amazing ambient sound.

Ok... one thing i dont agree on is reverberation... reverb in audio processing terms is most commonly known as the diffuse echo that results form singing in a cathedral and making a sound in a drainpipe. While your program can also do reverberation i think it's a grain sampler.

Perhaps just rewrite the second post to an online theory of granular synthesis and it's many virtues and uses, i am sure there are many pages online already written in a more terse format!

The forums are very busy with info and projects, i also have had a project announced and not noticed! your one is awesome.

 

Grain Resynthesis Sound Effects Generator sounds more crazy and futuristic as well for a title Oh yes sorry i m wrong i thought resonance was reverberation. http://en.wikipedia.org/wiki/Acoustic_resonance

 

Hehe doode, thank you for the kind words and the heads up. I definitely lack advertising skills ; (

resonance is the name of the offering

I am in the process of putting together some actual demos and possibly a collaboration with some other folks to show what it can really do

The absolute best thing? It uses the source audio's harmonics. While you *can* pitch shift, the granulation component uses the source audio 100%

 

Bump for updated video demo

 

Was this ever released? Have searched high and low in Asset store and google.

Have an immediate need for such a tool.

 

I put it on hold since there didn't seem to be enough interest, so I'm using it for my own project

 

This is really cool, I would use it in my game.

 

This asset sounds great. This might be a big time saver for all racing games.. Although the presentation surely needs some K.I.S.S.

 

Hi Marionette.

I left you a private message in the forums, not sure if you received notification about it or not.

Look forward to hearing from you.

 

agreed, admittedly i'm a bit long winded lol

 

Any news?

 

Actually yes

I have recently delved back into the code and have been adding some algorithms to handle some issues I was unhappy about, namely phase unwrapping etc and some additional FFT stuff to increase the audio quality, and am literally just finishing up on the preliminary testing. If all goes well, I plan to submit to the unity store within the month. Stay tuned

 

Good to hear. Could you make a asset store thread at some point? And explain the asset in one sentence. I think that would garner some additional interest.

 

Hi! Good job, man!!!
I've been interested in granular sounds for vehicles for some months. The only granular-related technology I've tried is the standalone version of REV. I found it great but after reading its licensing page I felt... poor.
I really hope your system will be as good as it. I'll wait for the Asset Store package!!!

 

Any news?

I would be more than pleased for if what I saw in the Youtube video was released. I don't need perfection. And frankly, I couldn't find any flaws.

 

First, i must apologize for not keeping you all updated.

While I have been quiet, I haven't stopped working on it. I've updated the api to be more performant, usable in 5+ and have implemented a host of new things including an automated RPM detection algorithm which allows for a reduction in phasing for variable speed audio.

I'm currently implementing a much better, completely in unity editor solution for resonance's granulation editor and with the use of some excellent waveform tools from FC Audo tools, which is almost complete. I'm just finishing up the editor GUI wire up.

I'll post a bit more over the next couple of days including the need for a few folks to beta test.

Thank you for your patience and bearing with me.

 

as promised, a few teaser screens of the new editor GUI etc...

screen1:



screen2:



as i get closer to beta, i'll need some help testing if anyone is interested?

 

Absolutely.

 

Excellent

Getting closer. Just squashed some nasty memory leaks in the new editor. Can't wait until we have a solid GC. All kinds of weird memory allocations with strings and other things. I had to go back through the code and manually destruct as much as I could to help the GC out. Took a few days to sort out, but I'm back on track.

Still have some wire up to do with the audio regions etc. Also have been fooling with the idea of storing the regions in the source audio Metadata instead of having to 'carry around' markup files that store settings for external use. Internally, those settings are serialized however it's per component instead of per asset. I'm liking the idea of being able to simply drag and drop/load an audio asset with the settings and regions contained in the audio metainfo. I'll keep everyone posted.

I've gotten a bunch of requests to beta and it's truly appreciated, however for this first round I'd like to keep it a small closed beta. I'll need folks to test not only functionality and issues, but as many different platforms as possible. For those interested and have the bandwidth to test, I'll be in contact

If you are interested in testing, please send me a private msg with your specs, primary platforms that you use and a little development background so that I might get a feeling for your testing comfort level as well as your availability to test.

I'm also going to need help with some demos etc if you're so inclined, once I get closer to release.

For this help I'll be offering a discount for those testers.

Thank you truly for the interest folks, it's appreciated

 

small update. I've added the ability to serialize resonance's settings and user regions into the wave file itself via custom chunks that do not impact the source wave at all other than to add some bytes to the overall file size, less than 1k etc. The RIFF spec states that any unknown chunk(s) in the wave file are to be ignored, which i've tested and all is well. since these chunks have no bearing on the actual audio data itself, i'm very pleased with it and seems to simplify things a lot

why i think this is the right way to go, is that it means you can bring in your audio asset, modify the resonance settings as well as create/modify user regions (which are used for a number of things, least of all events), update/embed them into the wave file and now all you have to do is worry about deployment of the audio asset itself instead of additional settings files etc etc. if you need to give it to another dev to modify, everything is contained in the audio asset. boom, ready to go.

I had initially intended to release 2 dlls, the core resonance dll and a separate editor dll, but to be honest, while it made certain aspects of development a little easier, it also made adhering to my proposed license a bit 'iffy' (more on that later), because i do not want the editor to be accidentally deployed along with the core runtime. So... i've merged them both into the core and will handle it another way. It also makes it easier for deployment for me as i'd have to maintain separate versioning among other things etc.

i've been working on this for about 3 years now (off n on) and i want it to be right. i don't want to be known as 'that' guy who puts out crap to be patched later, so even tho it's taking me a while to sort this out, hopefully you can understand my reasoning.

thank you again for your patience

 

Hi! Could You tell me how our project-team can take participant in closed "beta-testing" ?
Thanks!

 

sorry for the late reply, literally just got done testing OS X Yosemite deployment and functionality, which i'm pleased to say has passed

i'm beginning android build testing tonight.

there are still a few small things to look at on the OS X implementation, but functionally, it works

so windows & mac so far

i'll be sending out additional emails probably tomorrow

thanks

 

hi Dmitriy, just send me a private message with contact details and i'll add you to the testing

 

Hi!
I've send private msg.)))

 

This is still in development?

 

Yes it is. There were some issues that required that i refactor the code extensively. Namely, working around stupid things that i didn't realize would be a factor, such as asset management for builds etc. It is still being worked on but i do not have a definitive timeframe yet. Soon, is the projection.

 

Good and sad to know it, because resonance.RT is exactly what I need for my current project that involves making music with a granular synth.
It's IOS and Android, and by the way, would be a great practical demo for resonance.RT
So, any chance to have it in a few days?