Reading a whole audio file and FFT it

Hi there!

I'm planning on making a game that would use the music from the player to generate a world, and then let the player play this world. Kinda like audiosurf but not a rythm game.

Is there a way I could read the whole file from A to Z at start, in some sort of preprocess ?
I know I could use GetOutputData or/and GetSpectrumData if I wanted to do something live (like this guy's Music Visualizer).

But that's not what I wanted to do. Here is a 2 years old threat that was discussing about it.
I guess that's something achievable because Audiosurf is making its sequel, Audiosurf Air, using Unity.

Do you have any ideas please?

 

Does anyone knows if it's planned to be released in Unity 4 ?

 

Hi there. I got here from your post in iOS dev, so I'll assume you want to use songs from the player's iTunes library to generate your world, which will be much more work than using pre-defined music (AudioClip).

You will need:
1) access to the player's library (search for plugins)
2) an elegant way for the player to choose which song generates the world
3) to decompress the music file in a float or int array, removing header data
4) implement your own fft, as a plugin too if you want to optimize speed

I'd suggest you make sure each of these steps is within your skills or has already been done before you get started!

On the other hand, using pre-defined clips would be a piece of cake. AudioClip.GetData() and then fft (C# implementations are easy to find).

Good luck, and do let us know of your progress!

Gregzo

Edit : just read the post you mention, pretty much what I wrote above, and with just as little detail. You'll have to do some research, and write plugins, that's for sure.

 

Audio buffer access has been around since 3.5 via OnAudioFilterRead(), do check posts on that (docs are very sparse).

3.5 also brought us Microphone class, as well as AudioClip.GetData and SetData. And a few annoying audio bugs, some of which I've listed here.

Gregzo

 

Thanks a lot for your answer gregzo!

It looks a bit brighter now on how and what I'll need to do ^^
For the FFT, I found a C# implementation, I think I'll try it out, and I'll look into plugin if it's too lagy

I've still a question : could you please be a little more specific on the difference between a pre-defined clips and a clip I'll get from the user via a plugin ?

For exemple, I could use prime31 plugin to get the file as WAV ; nope ?

 

Hi Baz,

Indeed, prime31's media player plugin should work ( exportSongFromLibraryAsWav method). As this method decompresses and exports to the Documents folder, it might be better to call it when not much else is happening. From the top of my head, here's what I think you'll need to do:

1) Export the user picked song to wav
2) Use System.IO to load the data in memory, excluding header data (you'll need to figure out how long is the header)
3) Optional : convert the data to a float[] ranging from -1 to +1 would allow you to set it to an AudioClip, which you could then add effects to in real time.

The problem I foresee is memory : 44.1khz 16 bit stereo wav, 180 seconds song = 31.752 mb, that's a lot of ram. Depending on how much is needed for graphics, it might mean you're game will not run on 256mb devices (iPad 1 for example).

Two posts of mine might help you :
a) Loading audio data from resources (to read from disk, you'll need to adapt using system.IO)
b)Writng a wav file (might help you understand the format in case you're not familiar with it. Some IO as well in that script.)

Last, what I meant by pre-defined clips are clips already imported in Unity before building. That would save you all the IO and conversions, but would of course not allow the user to choose a clip from his own library.

From my experience, loading and converting 30+mb of data on older iOS devices will be troublesome. Tread carefuly!

Gregzo

 

Hi again,

just thought of a much simpler way :

1) exportSongFromLibraryAsWav (method from prime31's plugin)
2) userClip : AudioClip = WWW.getAudioClip() (the WWW class can be used to read data from a file using file:// instead of http:// )
3) floatsToProcess = userClip.GetData()

Pseudo code, do ask if you need more details.

FYI your app's Documents folder's path (where the wav will be exported) can be retrieved using Application.persistentDataPath;
The URL you'll use will be something like

Code (csharp):

1. userClipURL : String = "file://"+Application.persistentDataPath+clipName;

Hope it helps!

Gregzo

 

Thx you A LOT gregzo, you're helping so much!

right now I'm testing the AudioClip.getData (doesn't seems to work), I'm trying to figure out what is what in this FFT implementation, and finally I'm trying to figure out how to convert user's audio file into WAV on Android.

Thank again, I'll keep in touch

 

With Android I have no experience. Regarding SetData, I replied in answers. FFT implementation, will check it out when my 10 day old daughter gives me time!

Glad to help,

Gregzo

 

This is a fantastic thread -- would love to hear if there's been any updates on getting this functionality to work and how

 

Hey not sure if this will be helpful but my app requires reading a whole audio file and FFTing it — vid for example

I published a plugin for iOS that makes accessing, loading, and playing songs easy. I simplified the functionality of the published asset so accessing the data isn't as straight forward as in the app but I could rework it if it would help you. Ill be happy to answer any questions.
edit: it doesn't support android