AudioStream - an audio streaming solution for all and everywhere

The main driving factor behind this asset was Unity suboptimal streaming capabilities of Unity on mobiles, especially on iOS.
The audio stream there is being stored in memory in its entirety during streaming, thus leading to out of memory crashes after very short period of time.
This can be worked around to some degree by periodically restarting the stream, but leads to very poor user experience among other things so I decided to look for a proper solution.
Since I coulnd't find any, this asset was made.
It gained many few other capabilities over time -

• AudioStream plays network streams, as well as local and remote single audio files (such as podcasts)
• Supported are all audio formats FMOD can handle including, e.g. FLAC or modules and MIDI files
• handles direct stream links to audio as well as M3U/8 and PLS playlists - which are common distribution format of vitually all webstreaming radio stations.
• allows testing streams directly in the editor
• Uses FMOD Studio low level API for streaming, decoding and - optionally - playback and recording.
• any available system audio output device (not just a system default one) can be used for output - this component can be used independently for any Unity AudioSource, and can target even just a specific channel of the output
• includes 'AudioStreamOutputDevice' Unity AudioMixer native effect which enables redirection right from the mixer/group to user selected output device - (Windows x64 and macOS *only* currently)
• support for proper 3D and ambisonic audio playback using Google Resonance libraries provided via FMOD integration, including any system input (but it's possible to use any other external spatialization library)
• supports multichannel input devices
• can act as a source for Icecast server in PCM and OGGVORBIS formats.
• streaming over LAN between two instances (1- and 2- channel OPUS encoding, using NetMQ for transport)

Tested so far on Windows, OS X ( still OS X ), iOS and Android.

AssetStore link: https://www.assetstore.unity3d.com/#!/content/65411
(semi)current docs should be here: https://github.com/r618/AudioStreamDocs

There is single package with all source code included and demo scenes showing the components usage and handling of events.

NOTICE: supports FMOD Studio Unity Integrations 2.01.04 and up.

NOTICE: -* AAC format *-
there's no special setup or any special considerations regarding this format. The *ONLY* actual difference compared to other formats is that the media *must* be played on an actual physical device (e.g. iOS/Android phone) which has hardware decoding capability for this format. That means that you will not be able to test it in e.g. Editor or standalone builds.

CAVEATS: Tizen platform in previous version of Unity (until support for it ceased completely) was poorly supported by FMOD native part resulting in FMOD not loading at runtime, at all.

 

V 1.0 is live.

 

Damn i did only find this today! Sadly spent some money on other solutions using naudio in the back.. terrible performance.. So found this one about 10 minutes ago : D haha Asset store is getting to big .. : d Cant find what you need .. hehe

Anyway. im using the 5.4 rc works fine. only thing you need to do is nuke experimental in UnityWebRequest : )

 

right, thanks @nrlnd
will adjust for the beta soon, haven't looked at it yet

 

Unitys regular stream seams to perform better on standalone. But then again there is some weird stuff going on with 5.4.
Like.. I think it has something to do with unitys editor to do., If i run both streams the minimal and standard. it's on around 71 ish fps. If i only run minimal it's around 71 as well. Ok now to the freaky stuff. If i run the standard stream and got it selected in the editor it runs around 157 fps. ok.. If i select some other object like the camera. It drops down to 75. EH ..So..I run the minimal stream. But Select the standard asset in the editor. it's not running now. the minimal stream is up in 157. haha.. If i run both streams .. select the standard asset in the editor it's up on 157. If i fold down the standard editor. not visible.. the streams are back to 75 : d think this is some general freaky stuff going on in the 5.4 editor. About to kickstart 5.3.

 

for more consistent fps in the demo scene in the editor you can remove

Code (CSharp):

1. [ExecuteInEditMode()]

attribute of the AudioStreamDemo script

you will loose just the nice preview in the game view while not running
apparently editor has some trouble with immediateGUI stuff always running for some reason

 

as for the performance compared to unity stream : there might be some unnecessary mutex locking required only for releasing the channel, i'll look into it
AudioStreamMinimal is more performant than 'standard' one, but has less control, so is unfortunately not suited for all circumstances

 

Think this is an 5.4 issue. Fps runs fine in 5.3. Oh.. And congratulations! Most important part. Your solution is capable to grab Soundcloud streams with out any issues.

 

cool
- but be careful: I'm not entirely sure SoundCloud's TOS allow streaming for 3rd party applications, but then again I haven't studied it in detail - it might be perfectly fine

 

Nah thats fine using my app id and following the tos.

 

V 1.1 is live

- update for FMOD Version 1.08.09 - Strongly recommended to update, if previous version was used as it contains a bugfix related to network loss condition
- update for Unity 5.4
- improved startup for variable buffer lenght streams
- improved stream loss detection and recovery
- improved stream finishing for finite streams ( hosted files ) and locally streamed files for AudioStream component
- improved state reporting
- few performance improvements - AudioStream is now even more low profile

Cheers!

 

As mentioned in http://forum.unity3d.com/threads/selecting-the-audio-output-device.306897/#post-2741181
it is possible to use other than system's default audio output for playback.
There is a catch currently though that Unity's AudioSource can't be used this way - therefore it would work only for fmod's own separate playback - i.e. no usual effects, filters, pitch changes etc. available for scripting on AudioSource/editor - working case would be simple playback of a file / stream on a chosen system device with e.g. volume change available.

I'll probably look on how to expose this to be as simple as possible, if there is any preliminary interest please do let me know.

Cheers.

 

Hello,
Thank you for making this asset. I want to use it to stream ogg files on Android (not possible with Unity right now).
I'm making a small android game and I would like to keep the size small as much as possible.
So, I would like to know if adding FMOD studio will increase the size of APK a lot.

 

Hello @servusdei ,
I have just taken a peek into apk Unity builds with demo scene; for my phone it includes lib/armeabi-v7a and lib/x86 folders with respective fmod binaries in each, armeabi-v7a being ~ 9 MB and x86 ~ 9.7 MB in Packed size. So in total it seems fmod plugins take cca 18.7 MB compressed in the archive.
I hope that helps and best of luck with your game!

 

The whole apk with demo scene only is approx. 27 MB so that means the engine itself is slightly above 8 MB, which is neat.

Also note that I don't know exactly how multiarchitecture builds are handled e.g. on the Play store - chances are that arch not corresponding to user device is stripped from the archive, similarly to the Apple's store, but I am really not sure.

 

V 1.2 is live on the store

- audio can be played on any available system output*

* this is atm restricted to non AudioSource audio only.

 

Hello,
I'm interested in your plugin but I'm not sure I can do what I want with it.
I have a sound playing on the main output and I want to have a synchronisation signal played on another output every time the sound ends.
The 2 sounds have to be perfectly synchronised and for that I'm using the OnAudioFilterRead() method in Unity to check the position in my generated sound. But for now, i'm unable to play the 2 sounds on 2 different output.

If I understand, with your plugin it's possible to change the output but only to non AudioSource audio. Does that means, I can't use the OnAudioFilterRead() method if I want to change the output?
Is there a way to do it with your plugin ?

Thanks

 

Hi @rekerukuru89 ,

yep that's unfortunately correct : OnAudioFilterRead is Unity callback and as such Unity still decides what to do with the audio/data if used directly - since it has no option to play it on non default device atm the output always ends up on the same default system device.

You can use FMOD directly for what you describe, I think:

if you know the exact length of the initial sample you you can use

Code (CSharp):

1. setDelay

on a manually created fmod

Code (CSharp):

1. channel

with its

Code (CSharp):

1. system

set up to use desired output driver.

if not you can still set up fmod as described above, but play

Code (CSharp):

1. channel

directly from OnAudioFilterRead instead at proper time. ( since this would invoke playback not from the main thread, unexpected issues might arise, but it _should_ work ).

another option would be to use fmod studio events, but i have, unfortunatelly, no experience with it; moreover, since you generate the sound it's even hard to tell overall if its feasible for this.

For the fmod to work in unity you need their Unity package and look for fmod documentation in general - their unity c# wrapper is practically one to one mapping to their c/c++ runtime.interface.

So the functionality you describe is not in my plugin atm; I'll see whether it is feasible to make it work 'automagically' for AudioSource somehow, right now I'm not entirely sure, but I'll probably do some testing.

Cheers!

 

Thanks for the complete answer, let me know if you find some way to make it work.
I'll try with the fmod plugin !

 

so things are looking very optimistic for an output to any present device from AudioSource:



all the usual AudioSource stuff including effects is working like it should, the component has to be last in the chain ( since it silences the original signal and reroutes it - with a necessary slight delay )

will have to actually make it seamlessly work with the streaming part and do proper demo (scene?) for it, but otherwise it seems like it could be submitted very soon!

One fmod bugreport later I'm really glad that it worked and the approach proved to be feasible.

Cheers!

( /cc @rekerukuru89 )

 

@r618

Nice, I've done some test with FMOD but it seems to complicate for what I wanted to do.
I found a c# audio wrapper called NAudio : http://naudio.codeplex.com/

And I did it with the wrapper, using the asio driver so the latency is very low.

When the new version of your plugin will be out I will try it to check the latency !

Thanks for your research !

 

Is it possible to choose a channel on the selected output device ?

 

@rekerukuru89

yeah, NAudio is a widepsread .NET audio solution, which I have used occasionally too; from what I understand it's main bottleneck is that it's doing en/decoding in the managed code and lack of it's native audio interface wrappers for various paltforms. ( the latter maybe being progressively patched )

FMOD has more consistent multi platform coverage including consoles, has better platform specific support and has no noticeable footprint when running ( footprint in the profiler is lower than other unity components when running e.g. empty scene, and has 0 GC Allocs ).

As for latency - I did some testing measuring AudioSettings.dspTime difference between 1st audio frame of an AudioSource and 1st callback requested by FMOD for routing the audio to selected output.
The difference is somethings like 60 - 100 ms, give or take some logging involved.
So - not utterly terrible and it might be probably possible to lower it by offloading FMOD initialization to Start and calling the setup for routing from OnAudioFilterRead thread. ( which I did initially, but it seems like this had problems on Mac, so I left it on the main thread for now )

This atm works for _any_ AudioSource but I've yet to come up with some easy to use and understand solution for AudioStream too, maybe leaving this as a separate component so it can be used independently.

 

I'm not sure what do you mean, can you be more specific in what are you trying to do ?
You can request some non standard speaker setup directly from FMOD, if that's what you mean :



This would atm work only when not using AudioSource with this plugin.
(and might be actually worth implementing for AudioSource with the redirect component too)




... other than that probably not automatically without some explicit sw channel filtering ( or having the original signal already have only required channel/s )

 

We are currently looking for a solution to be able to select an output device and specify channel on that device per audio file.
So i.e if a soundcard has 8 channels we would be able to play 8 audio files at the same time.

 

unless the driver does create and expose e.g. virtual device/s mapped to respective sound card physical channel/s i'm afraid there's no general way to utilize them
( sound is played on a 'driver' - which is a representation of a single device in the system in FMOD - then it depends how that concrete device is configured, i.e. what outputs are connected, how the signal is split up is necessary and so on )

 

After a bit of delay the new version is live

V 1.3 092016
- new AudioSourceOutputDevice component - enables redirection of AudioSource's output buffer to any audio output present in the system
- update for FMOD Version 1.08.11 <- at least this version is needed for AudioSourceOutputDevice to work as it contains a bug fix formerly preventing so.
- fixed tags reporting on track change
- ( refactored common functionality into a new source file )

So it's possible to direct output of any AudioSource to any present system output driver via the new AudioSourceOutputDevice component (more info and details in readme).

/cc @rekerukuru89 if you still want to try this out.

Cheers!

 

Hi there,

Your plugin looks great but before I purchase I wanted to check if it can do what I'm after. I'd like to direct a real-time live audio input coming into my sound card to play from a Studio Event Emitter with 3-D spatial positioning applied. The only processing that is needed is the spatialisation which I have working through FMOD at the moment but I'm unsure if it is possible to do the same with a live input?

Many thanks,

Phil

 

Hi @phillee53 ,

First of all - AudioStream does not, unfortunately, stream audio in at the moment.
I was thinking about adding audio / microphone input for some time now, but it is not ready yet.

Secondly - technically it does streaming using FMOD, but not using any FMOD Studio functionality - I suppose that Studio Event Emitter is part of that.
I can only recommend looking for live in input in the Studio itself, but being not very familiar with it myself, I am not sure if it's feasible at all.

That being said - I'll probably try to figure something out - stay tuned.

I hope this helps!

 

Hi @r618,

Thanks for your response. Great to hear your investigating the audio/microphone input. That's interesting to hear, I was unsure if it actually was possible. The spatialisation does work through an FMOD studio plugin so would need to pass through that at some point in the processing. However, the same plugin that I'm currently using (The Oculus Native Spatializer) also works straight into Unity without the need for FMOD, so any standard Unity audio source can also be spatialized. Would this make any difference to the possibility of using your future audio/microphone input function attached to a moving object then spatialized?

Cheers!

 

Hi, it shouldn't:
AudioStream can be used with AudioSources transparently - it passes audio data to Unity's implementation for spatialization and all things Unity such as effects etc.
I might provide you with testing build with the audio input to test the processing speed and moving objects in VR, but it will be at least couple of days until I will be able to look at it and figure whether it's actually feasible it or not.

 

Hi! I can use this plugin for streaming play audio from default URL link of music file? (just playing until downloading)

I cannot use default WWW and audio source for streaming because it's crash with "Error: Cannot create FMOD::Sound instance for resource H%BB, (Operation could not be performed because specified sound/DSP connection is not ready. )".

Unfortunately it looks like this is an engine bug. There is a bug on the IssueTracker out now, however it appears that Unity has marked it as fixed and users are reporting it as still an issue.

 

That would be great to test out thank you. It would be important for my particular project to keep the latency to a minimum. Would be interesting to see how it performs.

 

Hi,
yeah, streaming - and optionally playback - works independently of Unity implementation, so current Unity streaming bugs should not affect it.

It's possible to stream either from webradio playlist, direct URL link ( a hosted audio file), or file from local filesystem.

Hope that helps!

 

@VictorKrasovsky just to clarify:
if you want to physically download the file and use it later you'd have to do it by yourself.
AudioStream just streams the content and immediately plays on audio output without storing it.

 

A lot of thanks. I think that's what I need.

 

Small update regarding streaming from audio input devices and spatial sound / stereo panning -

After initial testing of audio input - thanks @phillee53 for testing! - I've discovered that stereo panning and spatial left/right position is - unfortunately - ignored by the AudioSource. I've reported a bug to Unity, but it's hard to say when this will be corrected.
- just felt obliged to warn users wanting to stream audio to 3D positioned objects - this will not work for now

I will probably release an update which will allow to stream any audio input ( microphones, line-in devices and so on ), but without spatial functionality - if I won't find a workaround.

 

Can this plugin import a sound file from the android SD Card into unity to play in the game?

 

Hello @hzqtkxel,
depends how you structure your audio assets, but since you mention SD card, I would assume yes.

This plugin does not import anything into your project - it only needs a full file path to an audio, which then can stream - so if you put your files separately somewhere on the filesystem on Android outside of Unity application and that location is accessible, you are good to go.

The situation is more complicated if the audio is part of the project, e.g. in StreamingAssets folder - since the plugin needs full file path and the audio is distributed as part of the resulting jar/apk, you would first need to extract it from the jar, supply its file path to the plugin and only play it afterwards.

I hope this makes sense and helps!

EDIT: cleared up the StreamingAssets folder. (Resources has nothing to do with this.. )

 

Hi,

I'm having some issues with the 'Audio Source Output Device' component. When I use this to route the audio through a unity audiosource, the audio becomes 'crackly' and sounds clipped / distorted. If I use the standard AudioStream behaviour and do not route through a Unity audiosource, the sound is perfect quality. This happens even in the demo scene.

Is there a way to resolve this issue?

EDIT: the issue seems to resolve itself if I enable 'Bypass Effects' on the audiosource. However, I can hear the audio 'stutter' every so often while streaming a local song off my harddrive. I assume this is something to do with the streaming process.

 

Sorry to hear that @JonDadley , and thanks for letting me know!

- it is also worth checking DSP buffer size in the audio settings of the project and set it to Default, if possible - this affects the streaming data, I'm hoping to find a solution meanwhile.
It is also recommended to restart the editor after the DSP buffer size change - especially with older versions of Unity.

Do let me know if there's some problem still.

 

I want to get some audio files from the android memory and assign those audio files to a script and the script can then play, pause and analyze the audio as necessary.

 

Thanks for the info, I'll try it.

In the short term I used the standard audiostreaming and it worked flawlessly. Super impressed with the asset, it allowed me to add .flac streaming to my game extremely quickly. Thanks for the excellent work!

 

So, suppose you have a reference to AudioStream component in your script, lets say

Code (CSharp):

1. var astream = this.GetComponent<AudioStream.AudioStream>();

.
Once you obtain the location of the file on the filesystem, you can then assign its filepath:

Code (CSharp):

1. astream.url = _the_filepath_;

,
play it:

Code (CSharp):

1. astream.Play();

,
pause it:

Code (CSharp):

1. astream.Pause();

,
and you will be notified by events in both cases once the operation completes, if the events are setup e.g. in the Inspector.
When the track finishes playing, you will get notification, too. (again, if the event is setup e.g. in the Inspector).

For audio analysis you can use Unity's builtin GetSpectrumData of AudioSource, this is a working example:

Code (CSharp):

1. using UnityEngine;
2. using System.Collections;
3.  
4. // this script assumes AudioStream url is assigned in the Inspector pointing to an audio file and 'Play on Awake' is checked
5.  
6. [RequireComponent(typeof(AudioStream.AudioStream))]
7. public class AudioAnalysis : MonoBehaviour
8. {
9.     AudioSource asource;
10.     float[] buffer = new float[512];
11.  
12.     void Start ()
13.     {
14.         this.asource = this.GetComponent<AudioSource>();
15.     }
16.  
17.     void Update ()
18.     {
19.         if ( this.asource.isPlaying )
20.         {
21.             this.asource.GetSpectrumData(this.buffer, 0 , FFTWindow.Hanning);
22.  
23.             // do something with the spectrum data from the buffer ...
24.         }
25.     }
26. }
27.  

Note: this approach for analysis with GetSpectrumData does not work in Unity 5.1 due to bug - I unfortunately don't know up to which version - but does work in later versions such as 5.4.4 which I've tested it on.

I hope this helps and let me know if there's still something not clear.

 

Sorry if you have already answered this question. But for my game I want to be able to stream whatever is in the system's audio output, that being anything that comes out of the speakers, and from there turning the the audio into a spectrum that will be used as music visualization.

Will this asset allow me to do this?

Thanks

 

Hi @AveryBartlett ,

not with this asset only, and I'm not sure how exactly everything should be configured since I've never tried it properly, but -
if you are on Windows, you can try to enable loopback device - look for “Stereo mix”, “Wave”, “Wave Out Mix”, or “What You Hear” in Recording devices ( you might enable 'Show disabled devices' and 'Show disconnected devices' in the list of devices first ) and (probably) enable checkbox 'Listen to this device'. - your soundcard must support it
On mac look for Soundflower, or Rogue Amoeba offerings.

Since AudioStream does not have recording / audio in features released yet, I'd probably then setup local audio streaming server with the above loopback device as a source and point AudioStream to its (localhost) url - look for e.g. shoutcast/icecast servers.

Either way - there's not an easy way to automate everything inside Unity application itself without manual ( or some external, at least) setup first at the moment.
Once the audio is being streamed, you can, of course, continue to process it further, such as in the example in my previous post.

I hope that this at least pointed you in the right direction.

 

Hi, I'm interested in your asset. : )

I have some question for your assets...

I want to use android for local & streaming(optional) player for Language listening utility.

- Is this asset has a-b loop functions?

For example, from 0:57.32sec to 1:23.02sec loopback play. (infinite loop playback)

- Pitch for octave change. (Optional)

- Playback speed modification. 0.5x~3.0x would be good.

Is there some example access local mp3s (outside form app) with APK?

It will be good for me.

Thanks. : )

 

Hello @Bay ,

no, unfortunately not - this asset is rather streaming oriented indeed, hence I don't have support for advanced playback features - same e.g. for looping - you'd have to loop yourself if playing a local/single track, but no plans for a-b looping for now.
I'll add it to lookup/TODO list, though.

- you can modify the Pitch property on AudioSource directly

- see the first point - no plans for now, unfortunately

no complete online example, but if you have the location of the file somewhere on the filesystem and that location is accesible for the application, you can pass this as url for the AudioStream - see the answer before: https://forum.unity3d.com/threads/a...n-for-all-and-everywhere.412029/#post-2986446

 

just to reiterate and followup for accessing files from unity application on adroid / @Bay
basically there are two situations regarding playing local files on android -
either you have access to the file which is somewhere on the filesystem - meaning your application has permission to access it; - pass its filepath as url to the AudioStream in that case

or the file is part of the application as a file in StreamingAssets folder - you'd have to extract it from the jar/apk and copy somewhere else prior to playback
at least that's my understanding of the current state of things

AudioStream does not support playing Unity imported assets - i.e. assets residing in the Resources folder, or common assets such as audio clips residing in the scene or prefabs.

 

I streamed Audio from android SD Card using audiostream.cs but the sound is laggy.
Streaming using audiostreamminimal.cs does a good job in streaming and the sound is good.

But I need to analyse the stream and I need to use AudioSource.GetSpectrumData()

Is it possible?