Search Unity

Feature request: real-time audio support

Discussion in 'Scripting' started by Pi_3-14, May 20, 2014.

  1. Pi_3-14

    Pi_3-14

    Joined:
    May 9, 2012
    Posts:
    168
    I'm designing a musical trainer game that processes microphone input and create appropriate speaker output in real time.

    I'm doing this by (ab)using AudioClips.

    Its actually possible to do this, but the code is horrible. It is a complete hack, and moreover has quite high latency. I have to create one audioclip for microphone input, and another for speaker output. I have to set both of them spinning. Then I have to periodically poll the microphone one, observe how far the writehead has progressed, pull out necessary samples, process them and write them.

    And even then it is not so straightforward. If I want to avoid dynamic allocation. i.e. if 617 samples have arrived, I need to create a new float[617] array, and pass it into the audioclip's GetData method. The workaround is to preallocate PoT buffers; length 1,2,4,...,4096, and then keep filling the largest possible, and processing that.

    It's horrible.
    HORRIBLE!

    And then writing, I have to make sure I am writing suitably far ahead of the output audioclip's read-head. So that it doesn't splutter. But I have to minimise the distance, because that distance is latency.

    HORRIBLE!

    I don't like working against the grain like this.

    As Unity moves to incorporate a C++ layer, I petition and beseech the powers that be to provide a realtime audio IO object.

    Code (csharp):
    1.  
    2.     bufLen = 1024;
    3.  
    4.     // callback
    5.     void floats_in_out( float[] mic_in, float[] out_L, float[] out_R ) {
    6.         // yes yes YES!
    7.     }
    8.  
    Just that single callback would give the flexibility for doing wonderful things.
    Unity will eventually allow C++ files to be dropped into an assets folder and have them automatically compiled (http://forum.unity3d.com/threads/24...-forum-topic?p=1634492&viewfull=1#post1634492)
    This opens up the possibility of real-time DSP.

    There is at least one existing multi-platform C++ library that provides real-time audio IO that works on OS X, Windows and Linux. It's maintained as a single file.

    There are several similar libraries.

    Is this the best place to request such a feature?
     
  2. Zaladur

    Zaladur

    Joined:
    Oct 20, 2012
    Posts:
    389
    Unity 5 will come with a complete overhaul of the audio system, but I do not know if that will solve your specific issue or not.

    I would think that the feedback section would probably be a better place to request features. The Scripting subforum doesn't get as much Unity Team visibility.
     
    Last edited: May 20, 2014
  3. Tenebrous

    Tenebrous

    Volunteer Moderator Moderator

    Joined:
    Jul 25, 2011
    Posts:
    99
  4. Pi_3-14

    Pi_3-14

    Joined:
    May 9, 2012
    Posts:
    168
    From:

    http://blogs.unity3d.com/2014/03/18/unity-5/


    '... Unity 5.0 is introducing a completely overhauled sound system, which is much more robust and has visual tools for sound designers. The entire audio pipeline was ripped out and reconstructed to be more flexible and efficient.'


    I can only hope they are addressing real-time audio IO, which is currently very clunky.

    I've put in a feature request here: http://feedback.unity3d.com/suggestions/real-time-audio-io-access-1

    OnAudioFilterRead doesn't synchronise between input and output streams. Currently there is no accurate way of processing microphone data and putting it out through the speakers with optimal latency, or anywhere close. Also it is pro only, which is restrictive.
     
    Last edited: May 21, 2014
  5. gregzo

    gregzo

    Joined:
    Dec 17, 2011
    Posts:
    795
    Hi Pi_3.14

    My audio framework, G-Audio, can pipe mic data to a custom mixer. Do give the looper demo a go, see what you think.

    G-Audio adds tons of functionnalities to Unity's audio, whilst remaining compatible with Unity Free and cross platform. The new update adds I/O to the mix, enabling routing audio streams to disk, to cache, to G-Audio tracks, etc…

    Cheers,

    Gregzo
     
  6. gregzo

    gregzo

    Joined:
    Dec 17, 2011
    Posts:
    795
    I forgot to mention:

    G-Audio's mic module only allocates a 1 second clip.
    Also, G-Audio has a custom memory manager to create and destroy audio data objects on the fly with zero garbage collection.
    Very useful for granular synthesis, for example, where hundreds of grains of sound can be allocated, processed, and ditched every second.

    Gregzo