Search Unity

  1. Welcome to the Unity Forums! Please take the time to read our Code of Conduct to familiarize yourself with the forum rules and how to post constructively.

Reading a whole audio file and FFT it

Discussion in 'Editor & General Support' started by BazLevelUp, Sep 21, 2012.

  1. BazLevelUp

    BazLevelUp

    Joined:
    Jun 27, 2012
    Posts:
    36
    Hi there!

    I'm planning on making a game that would use the music from the player to generate a world, and then let the player play this world. Kinda like audiosurf but not a rythm game.

    Is there a way I could read the whole file from A to Z at start, in some sort of preprocess ?
    I know I could use GetOutputData or/and GetSpectrumData if I wanted to do something live (like this guy's Music Visualizer).

    But that's not what I wanted to do. Here is a 2 years old threat that was discussing about it.
    I guess that's something achievable because Audiosurf is making its sequel, Audiosurf Air, using Unity.

    Do you have any ideas please?
     
    Last edited: Sep 21, 2012
  2. BazLevelUp

    BazLevelUp

    Joined:
    Jun 27, 2012
    Posts:
    36
    Does anyone knows if it's planned to be released in Unity 4 ?
     
  3. gregzo

    gregzo

    Joined:
    Dec 17, 2011
    Posts:
    795
    Hi there. I got here from your post in iOS dev, so I'll assume you want to use songs from the player's iTunes library to generate your world, which will be much more work than using pre-defined music (AudioClip).

    You will need:
    1) access to the player's library (search for plugins)
    2) an elegant way for the player to choose which song generates the world
    3) to decompress the music file in a float or int array, removing header data
    4) implement your own fft, as a plugin too if you want to optimize speed

    I'd suggest you make sure each of these steps is within your skills or has already been done before you get started!

    On the other hand, using pre-defined clips would be a piece of cake. AudioClip.GetData() and then fft (C# implementations are easy to find).

    Good luck, and do let us know of your progress!

    Gregzo

    Edit : just read the post you mention, pretty much what I wrote above, and with just as little detail. You'll have to do some research, and write plugins, that's for sure.
     
    Last edited: Sep 22, 2012
  4. gregzo

    gregzo

    Joined:
    Dec 17, 2011
    Posts:
    795
    Audio buffer access has been around since 3.5 via OnAudioFilterRead(), do check posts on that (docs are very sparse).

    3.5 also brought us Microphone class, as well as AudioClip.GetData and SetData. And a few annoying audio bugs, some of which I've listed here.

    Gregzo
     
  5. BazLevelUp

    BazLevelUp

    Joined:
    Jun 27, 2012
    Posts:
    36
    Thanks a lot for your answer gregzo!

    It looks a bit brighter now on how and what I'll need to do ^^
    For the FFT, I found a C# implementation, I think I'll try it out, and I'll look into plugin if it's too lagy ;)

    I've still a question : could you please be a little more specific on the difference between a pre-defined clips and a clip I'll get from the user via a plugin ?

    For exemple, I could use prime31 plugin to get the file as WAV ; nope ?
     
  6. gregzo

    gregzo

    Joined:
    Dec 17, 2011
    Posts:
    795
    Hi Baz,

    Indeed, prime31's media player plugin should work ( exportSongFromLibraryAsWav method). As this method decompresses and exports to the Documents folder, it might be better to call it when not much else is happening. From the top of my head, here's what I think you'll need to do:

    1) Export the user picked song to wav
    2) Use System.IO to load the data in memory, excluding header data (you'll need to figure out how long is the header)
    3) Optional : convert the data to a float[] ranging from -1 to +1 would allow you to set it to an AudioClip, which you could then add effects to in real time.

    The problem I foresee is memory : 44.1khz 16 bit stereo wav, 180 seconds song = 31.752 mb, that's a lot of ram. Depending on how much is needed for graphics, it might mean you're game will not run on 256mb devices (iPad 1 for example).

    Two posts of mine might help you :
    a) Loading audio data from resources (to read from disk, you'll need to adapt using system.IO)
    b)Writng a wav file (might help you understand the format in case you're not familiar with it. Some IO as well in that script.)

    Last, what I meant by pre-defined clips are clips already imported in Unity before building. That would save you all the IO and conversions, but would of course not allow the user to choose a clip from his own library.

    From my experience, loading and converting 30+mb of data on older iOS devices will be troublesome. Tread carefuly!

    Gregzo
     
    Last edited: Sep 24, 2012
  7. gregzo

    gregzo

    Joined:
    Dec 17, 2011
    Posts:
    795
    Hi again,

    just thought of a much simpler way :

    1) exportSongFromLibraryAsWav (method from prime31's plugin)
    2) userClip : AudioClip = WWW.getAudioClip() (the WWW class can be used to read data from a file using file:// instead of http:// )
    3) floatsToProcess = userClip.GetData()

    Pseudo code, do ask if you need more details.

    FYI your app's Documents folder's path (where the wav will be exported) can be retrieved using Application.persistentDataPath;
    The URL you'll use will be something like
    Code (csharp):
    1. userClipURL : String = "file://"+Application.persistentDataPath+clipName;
    Hope it helps!

    Gregzo
     
  8. BazLevelUp

    BazLevelUp

    Joined:
    Jun 27, 2012
    Posts:
    36
  9. gregzo

    gregzo

    Joined:
    Dec 17, 2011
    Posts:
    795
    With Android I have no experience. Regarding SetData, I replied in answers. FFT implementation, will check it out when my 10 day old daughter gives me time!

    Glad to help,

    Gregzo
     
  10. sarith

    sarith

    Joined:
    Feb 12, 2013
    Posts:
    2
    This is a fantastic thread -- would love to hear if there's been any updates on getting this functionality to work and how :)
     
  11. hunterleaman

    hunterleaman

    Joined:
    Oct 2, 2014
    Posts:
    18
    Hey not sure if this will be helpful but my app requires reading a whole audio file and FFTing it — vid for example

    I published a plugin for iOS that makes accessing, loading, and playing songs easy. I simplified the functionality of the published asset so accessing the data isn't as straight forward as in the app but I could rework it if it would help you. Ill be happy to answer any questions.
    edit: it doesn't support android