AudioStream - {local|remote media} → AudioSource|AudioMixer → {outputs}

Thanks for the reply.
I will share the streaming URL once I get permission from the team.
My use case for audio streaming is
Users in my application will listen to the live streaming of audio. My current source of streaming is Url and the url is generated with the help of the streaming app Wowza. Live streaming of audio is working When I am running the Wowza generated Url in the VLC media player.

Thanks.

 

send me the link (feel free to PM) once you have it - I'll check the format / if its'possible to use it

 

Hi @r618 , I found out that my app defaults to earphone audio when playing any Audio Clip. I would like the default audio to be through my phone speaker, and to be able to let the user switch to a bluetooth speaker or Ear Pods . Can AudioStream can help in this case? Thanks!

 

Hi, the output selection is more restricted and only changeable by user on iOS:



I could display this sheet from the plugin but it's not implemented currently - you can access this by pulling control center:


moreover, when application is in recording mode (Unity has setting for this in iOS player), the earpiece/speaker can be selected:


BT outputs/inputs are tied together by default (when BT mic is selected for recording, the connected BT headset output is activated automatically) - this can be overridden to a degree by the plugin, and by Unity with 'Override speaker' setting for iOS Player
you can change outputs between speaker/phone at runtime when in recording mode with the plugin by changing the session, but it needs to be called manually (it's not in the demo I think) and usually the defaults in iOS Player settings work for most users for the lifetime of an application

Hope this helps!

 

I'm leaving here link of a testing APK build for Android I did in hopes to compare Unity/direct FMOD playback latency:
https://www.dropbox.com/s/jfnkq4xx2vi4j6m/PlayAudioLatencyTest.zip?dl=1

unity audio settings are set to Best Latency, the clip is trimmed at the beginning to start immediately
first button plays via AudioSource
second via fmod with DSP buffers set to the values below
you should try to enter as small values as possible and press 'Update DSP buffers' ( e.g. 128 / 2 on my testing phone, defaults are populated on start )
there is very small noticeable difference, but I can't tell how much this means also compared to other assets

-- would require much of a rewrite too if gone via FMOD route and abandoned Unity audio/clips, so leaving this here so it might be useful for comparing for an Android developer
but overall the latency on Android is not the best to say the least and FMOD just uses what's available

 

Thanks, apparently my app defaults to earpiece output because the recording mode was set on my app. I tried setting the 'Override Speaker' option for iOS Player, but this prevents the Audio from playing on my TV when my iPhone is connected to it. So I need better solution to switch between audio output in my iOS app...

Will AudioStream plugin support this feature for iOS any time soon?

Another question- What does this plugin add to Unity-iPhone application if we exclude this feature?

Thanks!

 

What happens when the app is not in recording mode and you pull up current outputs in control center ? Does is allow you to switch to your TV (I suppose it's via AirPlay) ?
If it doesn't I'd guess the default session set by Unity is not the correct one (see also below *; unfortunately I can't test with a TV with AirPlay support myself)
- 'Override' - as the name suggests - just overrides whatever current setting OS thinks is optimal for given situation -

* This feature (updating and changing AVAudioSession) is implemented in few objc/++ source files (as iOS plugin in Unity) and C# script
Apart from that iOS FMOD native libraries + accompanying AudioStream scripts, demos and resources are included in Xcode project by default
You can isolate it and use only the above sources if you remove everything else - except probably AudioStreamInput_iOS and demo which use it and possibly some common C# support script/s -
but the whole functionality is isolated from FMOD and Unity

 

My app requires recording mode to be set. Regardless, I was able to connect to Airplay and Speaker through the control center. It did not seem to be able to detect and play audio from Bluetooth speakers and AirPods though, and I had to use this hack to do so.
Does the AudioStream support something similar to this- maybe one or two files from the plugin that I had to include in my app in order for the app to be able to detect and play Audio on Bluetooth devices through the Control Center?

I'd prefer to eventually do the audio switching through Unity UI, but of course Control Center is a nice temporary solution for now...

 

Right, yes it supports playing on BT connected devices - it's little bit buried in the description, but the screenshot of the recording scene running on device is there too:

(the device 1 is AirPods)

For just playback alone (e.g. in a separate scene from recording) you'd have to call the included AVAudioSession wrapper and update the session (it's enough to do this once somewhere near start) - afterwards it's possible to select BT connected device from the OS outputs selection and Unity will play all audio on it
(- this is not included by default in AudioStreamBase, but it's worth adding I think - so it's not automatic right now - sorry! -)

 

btw that 'hack' is not that horrible really - there is cca no other way to update audio session Unity is using on iOS otherwise
(it's "just" a matter of doing it cleanly and in a way friendly to the user and to engine)

 

I just submitted small update:

v 2.3 032020 Corona+

Updates/fixes:
- AudioStreamOutputDevice : moved automatic output devices changes notifications to separate component (see below)
- AudioStreamInput* : moved automatic intput devices changes notifications to separate component (see below)
- AudioStreamInput2D : fixed incorrect handling of Mono output, corrected input signal retrieval so this further improves latency, Speex resampler output is improved as consequence, too
- AudioStreamInput2D_iOS : updated same as above
- AudioStreamBase, AudioStreamLegacyBase: automatically update audio session at start to include Bluetooth devices as possible system output on iOS

New Components:
- AudioStreamOutputDevicesChangedNotify : optional component can be placed in the scene to receive system's output devices changes updates
- AudioStreamInputDevicesChangedNotify : optional component can be placed in the scene to receive system's intput devices changes updates


So separate notification components (mainly useful when using just mixer plugin but want to be notified of system changes) and rather nice input/recording fixes

However, I noticed that AudioStream - main streaming components - stopped working on iOS some time ago
Legacy still works with the above improvements, so please use that if you don't need secure HTTPS transport for the time being
Please contact me otherwise if you need streaming & playback on iOS to speed things up - I'll be most likely replacing FMOD - for streaming playback only, everything else - such as recording / device handling - is fine with it
But streamed decoding with current approach is borderline unusable, especially when all supported various codecs are involved
- I intent to use NAudio/cscore most likely, but am open to suggestions if there are any ( I personally am a fan of BASS/.NET but it's not free for commercial use unfortunately so I won't be including support for it by default )

 

Thanks @r618 ! Bluetooth Input/Output works like a charm.
By the way, is it possible and do you have any plan to support Audio Output selection from iPhone Control Center any time soon? Would be awesome to be able to switch to one of those devices in app as well!

 

It's.. problematic; iOS doesn't provide user space with all available outputs and all interaction is handled on the OS presented selection sheet exclusively - so I can't just display outputs directly in the app (like when recording mode)
The maximum I could do is to call device picker programmatically, but at that point there's little difference between this and users actually pulling control center by themselves
It requires manipulating Unity View from inside the plugin - it's not impossible, but complicates things a bit - and due to the above not entirely worth it
I recommend sticking to iOS way of handling things for the time being - I hope it's not that big of a problem

 

Hello, I'm looking to understand if this is love : is your plugin the one I've been looking for?
I wish to output 8 different audio sources from unity, going each to their own speaker (passing through a sound card in order to route to the 8 speakers I suppose). This is not for surround sound. Each audio source will play their own sound.

What do you reckon?

 

hi, it should work without problems when you set a custom mix matrix for each sound
- have a look at the demo application (downloads are on the store page) and 'MultichannelOutputDemo' scene where a mono sound is directed on selected channel on selected output and simple automatic custom mix matrix is used
you will have to setup your scene similarly and each sound separately in it
(the matrix is simple for mono sounds - the one from the demo can be reused - but you can e.g. mix multichannel sounds to a single output channel with different one)

 

Hello,

I have a really weird issue, sorry if it came up in this thread but I haven't found that as I browsed through it.

Only on some machines, when using an audio source that has a AudioSourceOutputDevice.cs component from Audiostream attached, the sound is played with a high pitch even though the clip is not pitched at all. If I disable the AudioSourceOutputDevice component, the sound plays normally without the pitch but the output device source is no longer the one I previously set (the one I want).

I've been down in the dumps for more than a week trying to figure this out and it's eroding my morale

This issue happens also in the demo scenes.

Also on machines where this issue happens I get spammed with log messages like this

PCMReadCallback 1222032920 underflow - requested 38400 while having 37888 (providing 37888), time: 5352.08533333333
UnityEngine.Debug:LogFormat(String, Object[])
AudioStream.FMODSystemOutputDeviceCMReadCallback(IntPtr, IntPtr, UInt32) (at Assets/Vendor/AudioStream/Scripts/AudioSourceOutputDevice/FMODSystemOutputDevice.cs:85)

On machines that do not have this issue, this message also appears but much more rare (like 40, 50 times as opposed to over 1000).

Also I keep geting this error even on the PC where everything works fine and I don't have installed FMOD studio:
FMOD Studio: The FMOD Studio project path must be set to an .fspro file.

Is there a way to avoid this error without having to install FMOD studio separately ?


I used unity 2018.3.5.f1 and after that I upgraded to 2018.4.19f1 so I can update to the latest fmod package and latest Audiostream package. The results were the same.

It is very weird that this does not happen on my pc, it works perfectly fine but only happens on my laptopt and a colleague's laptop.

Thank you in advance for your quick replies.

If there is any other info that I haven't specified that you need, please let me know.

Best Regards,
Paul

 

Hello Paul,

first for FMOD Studio:
you can delete most of the stuff from FMOD package after importing it (or alternatively select just needed stuff in the import dialog)
the needed parts are (with package's default import location):
'Plugins/FMOD/lib'
- here you can select only platform you want support for
and
'Plugins/FMOD/src/Runtime/wrapper'
you can delete everything else FMOD package related from the project, and all FMOD Studio messages will vanish immediately ;o)
[ this is mentioned in README also if you want to have a look ]

for the main issue:
I will have a look at your device first - please run editor on problematic machine, select DEBUG log level on the component (it's in [Setup] section) and send me the console log ( you can open it directly from console in editor:

)

the sound might play with severe problems due to performance, but the needed info should be there
(you can PM the log if needed)

Thank you !

 

Thank you for the very quick reply,

We managed to solve the issue by setting the Default Speaker Mode to Mono (in Unity Editor: Edit->Project Settings->Audio)

Unfortunately we didn't have time to do the logs as you requested but if another issue appears or this returns we will promptly provide the logs.

Thank you again for the assistance.

 

Hm, I guess somewhere somehow is wrong # of channels set - by syncing unity default with the problematic output you fixed it that way; if you can stick with unity output being mono then it's good enough solution for now
: will have to test this
Logs would still help though )

Just to be clear : are you using latest version currently on the store ? - should be 2.3. released cca week ago

Anyway, thanks for letting me know !

 

We are not using the latest version of Audiostream since we have to use Unity 2018 3.5f1 and the new FMOD requires Unity 2018 4.18 or higher. Anyway, if I ever get a chance to get the logs I will definitely send them to you asap.

Thanks again.

 

hm the '2018.4.18' or higher is rather specific - I would expect them to support at least LTS - which means all 2018.4 releases - but I might be wrong
I recommend upgrading to LTS - should not be difficult going from .3 release - but I know these things can be problematic - just a suggestion -
I was fixing wrong mono handling in latest - but for input, not for this component - there are no functional changes for output device component going couple of versions back - will look at how to replicate this if possible

 

Hello there, I have problem with reapiting lag with live stream, when after starting stream in first "soundItem" I create another "soundItem" and put link to file from PC. After this I start audio from PC, but previous item with live stream has reapiting lag(like last 2 seconds repeats alltime and stream is not continue). When I put another live stream, then previous works fine.

 

Hi @kotor322 , which version are you using ?
In one of the previous ones a local file playback went through wrong codepath and didn't use local file system
Are you on latest (currently 2.3) ?
If yes give me some time to replicate and fix it
Thank you

 

Yes, I use latest version and Unity 19.2.21f, also testing it with AudioStreamMinimal recently and it's works fine, but I need AudioStream because must have my audiosource with customize range and controlled sound curve. And problem with live stream when i set in another "audiofile" audio from pc, but audio from pc works fine.

 

And also when it happend I don't have any errors or event warning, if I have at least error when it happend, I just make replay "audioitem" with this problem.

 

Thanks, I'll investigate

btw meanwhile: you can still use AudioStreamLegacy ( it sits in 'Scripts\AudioStream\_LegacyUsingFMODNetworking' ) for playback from network <- if you don't use/need HTTPS that is

 

Thanks, I try AudioStreamLegacy and it's works. Also I had reapiting lag with AudioStream in live streams and videos from PC when I use OpenFileDialog for selecting file. I temporary solve it with pause "audioitems" when panel was opened and restarting "audioitems" when OnApplicationFocus is triggered. But in AudioStreamLegacy I don't have this problem too. I hope you fix this in AudioStream, because I need HTTPS not asap, but eventually.

 

Thank you for the report

I want to replace current network stream implementation - see also: https://forum.unity.com/threads/aud...ll-and-everywhere.412029/page-12#post-5562682 , needs some time to be done

the issue you were getting with opened OpenFileDialog is most likely because main/UI thread is paused at that time and current component gets its updates from it - one more reason to try different approach -

I'll update thread once I have something usable,
Thanks !

 

Hi there,

I was hoping that someone could tell me if this asset pack could work for my project. Specifically, we are having a producer client in Unity which will stream audio and generate data for when certain lights flash to the audio being streamed, which would need to be streamed out with the audio in sync. We would then stream that through ShoutCast or IceCast. That would then be received by a different Unity consumer / client, whom would receive the stream and associated metadata to reproduce the music + light show on their end, in sync.

Does AudioStream enable encoding/decoding of audio with attached metadata? This would need to be done with a high accuracy < 100ms.

I'm a bit new to the subject of audio streaming, but looking for a solution and appreciate any help that I can get. Thank you!

 

Hi, you have access to all audio as it's being produced or played back, but you would have to code all support for any custom metadata - inserting it into PCM stream and reading back (and skipping them on audio output)
This can be done either on current network transport level (using NetMQ packets), or directly in OnAudioFilterRead callback/s. This way you can have <100 ms resolution, but note that since it would break any compression you would have to use PCM encoding only which would consume considerable bandwidth -
The final latency when going through Icecast (difference between producer and final client output) will be very high though - you would have to use provided server/client components (and having extra client application) to skip Icecast to reduce it - if possible
If you have an Icecast instance running you can test 'IcecastSourceDemo' scene in the demo application (download is on the store page) with it

Hope this helps!

 

Hi there, any news about AudioStream update? It wiil be really cool to have this feature asap for my project.

 

Hi, sorry to keep your hopes up, but this won't take days - this is range of weeks / months (I can't work on it exclusively - unfortunately!)

If you're about to release definitely don't wait just for this

Sorry about that !

 

Just submitted small but rather important update for cases where Unity and actual HW speaker modes differ (hopefully for good)
@paulsz you might want to give it a try some time

===============================================================================
v 2.3.1 042020 Corona++

Updates/fixes:
- AudioStreamOutputDevice : fixed wrongly determined Unity output channels count when Unity set speaker mode != driver capabilities (hopefully for some time at least)
this caused erroneous playback on some machines and when using other than default (Mono/Stereo) speaker mode
- mix matrix : (for recording and output device) - cleaned up displaying and simplified automatic mix matrix creation

Demo applications:
- built with 5.1 default speaker mode, actual mode is determined at runtime based on HW, requested and actual speaker modes are both displayed

General:
- tested with 'FMOD for Unity' package which is now available on the Asset Store
(and is identical to FMOD Integration available on FMOD website)

 

Hello- interested in using this nice looking tool in my project-

My project is using 2017.4.32f1 LTS - I saw some mention in this thread about FMOD requiring newer versions of Unity etc-

Will I have any issues/limitations using this asset with my older version of Unity? Thank you.

 

Hi, no it's not true
the package is submitted with 2017.4.1 and tested on all main releases from 2017.4.1 to latest 2019
FMOD Unity Integration _parts which are needed_ don't have any incompatibilities with 'reasonably' new Unity versions (2017 and up; it should work with pre 2017 too (so 5.x) , possibly with some tweaks needed - haven't tested this for some time)

If Asset Store doesn't allow you to download 'FMOD for Unity' on 2017 you can get the same package from https://fmod.com/download - called 'FMOD Studio for Unity' there, be sure to pick latest *not* Early Access version (atm 2.00.08)

I recommend picking only needed parts in Import dialog when importing FMOD package - it's described in the README hopefully clearly enough -

lmk if you need assistance or have questions

 

Thank you for the detailed reply- I will purchase the asset today. Thank you.

 

The asset works great- just getting an error on game quit-

At game start I have the AudioStream scripts disabled and they are only used when the game is connected to Twitch, which then they are enabled and used, works great- but if they are never enabled I'm getting this error when the game quits

Code (CSharp):

1. InvalidOperationException: Handle is not initialized.
2. System.Runtime.InteropServices.GCHandle.Free () (at <96a2980540ac4006b7a04455c3e3ad39>:0)
3. AudioStream.AudioStreamBase.OnDestroy () (at Assets/AudioStream/Scripts/AudioStream/AudioStreamBase.cs:1916)
4.  

It doesn't affect the game at all, was just wondering how to get rid of them if possible. Thank you.

 

Thank you *very much* for the report !

It's enough to check if it's allocated before releasing it, so:

Code (CSharp):

1. if (this.gc_thisPtr.IsAllocated)
2.     this.gc_thisPtr.Free();
3.  

You're right that it shouldn't affect anything since it's fortunately the last call when releasing it, but if you want to get rid of error message just stick the condition there and it should be gone

Great to see how people can use it in various ways !

 

pushed one more small fix:

v 2.3.2 042020 +++

- fixed issue with loading Resonance plugin in Windows player on 2019.x
- demo application should displays build time, not current time

 

Hello- how do you recommend checking the validity of audio files set to play using the AudioStream class? I was doing a play test when a url was submitted for an .mp3 file that crashed the game. It stopped on line 174 of the AudioStreamSupport class and threw an error saying the audio file was unsupported.

I am checking to make sure the url's are valid and that they have an .mp3 extension before they are sent to AudioStream, is there a way to check if the audio file is supported using AudioStream before calling Play?

This is the url that was submitted https://archive.org/details/aesop_fables_volume_one_librivox/fables_01_25_aesop.mp3

I can see part of the issue is that it is a valid url with an .mp3 extension but it links to a playlist instead of a single mp3 file- how would you recommend I check this url before submitting it to AudioStream? Thank you.

 

first of all - hook up your events @mdotstrange :
look at how events from the component are handled in AudioStreamDemo scene - you need mainly OnPlaybackStopped (and possibly OnError)
- e.g. if you enter that url in that demo OnError will be called multiple times as it's trying to open the file, followed by single OnPlaybackStopped when it gives up
- so you can catch at least unsuccessful playback (and if there was OnError call before you'd know it's because of a problem)

- if you don't know what the file format is prior playback I recommend turning off 'continuosStreaming' on the component too - otherwise it would be trying opening the file constantly ( having this on is useful when you know your stream and want to make sure it's playing as continuously as possible.. )

In general you can't determine exact format until at least some portion of a file is downloaded - often having supported extension doesn't mean the content matches (as you found out) - an ogg container/format in a file with mp3 extension or vice versa does happen too / though in this case the decoder tries other formats and should work on desktops /

let me know if you still need anything to make this work
( e.g. I think there might be more info needed in callbacks maybe in some situations )

 

Thank you for the detailed reply. I forgot about the events as I was just looking for methods inside the AudioStream class to call. I think turning off continuous streaming and using the events will do the job for me, thank you very much.

 

How can I get the audio from the loopback device into an AudioSource?

 

captured audio is being played back by an AudioSource, so you can get it by using usual Unity methods operating on AudioSources:
demo scenes AudioStreamInput/2D access captured audio via GetOutputData on the AudioSource to scale and rotate the cubes by measuring signal energy
if you want to get realtime PCM audio you have to do it in OnAudioFilterRead callback in a script placed on the game object with the audio input component
if you're creating a new scene be aware that you should mute the signal after you process it (by e.g. zeroing it) otherwise you'd get feedback loop
the demo scenes have this implemented by separate script 'Mute this AudioSource' which is last component on the game object ( if you attach your user script on this same GO, it should be _before_ this script - otherwise you'd get nothing in the callback )
if you have troubles setting this up let me know

 

Hi!
I'd like to test the plugin in my project, it might solve some issues.
Is the latest version trial available?

My email - afaringocept@gmail.com

 

Hi, there's no trial, but all functionality is present in the demo application
each component has its own separate scene to test, download links are on the asset's store page
let me know if there's anything missing what you'd need to test first, or you have questions afterwards please
Thank you

 

Hi,
Does the plugin interface with Dante Sound Card? - I'm wanting to output unity audio to individual Dante channels. Will this plugin work?

 

Hi, please download and run the demo (links are on the store page), go through relevant scenes to see what will work for you

I'm currently in the middle of adding separate playback of stems/clips (from HDD only for now) each on its own channel/s on given device
- Currently it works with Unity clips but output channels are set for all clips at once (per device), which is inconvenient - you have to prepare your multichannel clips beforehand

Let me know how it goes and/or if you have questions, thanks

 

I know AudioStream supports the streaming of .mid files- is there any easy way to download the .mid file through AudioStream? Or is there a way to access the MIDI note data in AudioStream?

Reason I ask is that I have a midi file processor I created to call certain methods based on notes etc so I need the midi data- Its already setup for local midi files but I want the .mid files from url's to feed into my processor as well. Is there a way to do this with AudioStream?

Before I write my own byte[] to midi file converter or something like that I wanted to check if its already in AudioStream to use ^_^

Thank you for your time.

 

hi @mdotstrange

supported audio files can be downloaded instead of streaming but it will be useless for this case since they are decoded to PCM data too - via AudioStreamDownload component
- it's done this way mainly for caching of audio files from network, but midi files are tiny compared to common audio and decoded data blow up in size so it makes sense using it only if AudioClip is needed for playback which won't help in this case too -)

there is unfortunately no support for any structural information of midi files

but if you have midi processor already just download the file in question - due to their minuscule sizes download will be quick, then just use path of downloaded file to use your processor as for local files

you don't need byte[] to midi converter (if your processor works on midi files directly), just use UnityWebRequest, wait (yield) until it's finished and save `www.downloadHandler.data` as normal file with .mid extension (use e.g. `File.WriteAllBytes`)

refer to https://docs.unity3d.com/ScriptReference/Networking.DownloadHandler.html and you can copy AudioStream's download handler if you need HTTPS and e.g. need to setup something like certificate handler
(use 'normal' bytes handler, not DownloadHandlerAudioClip)

Should be rather painless, let me know if this helps!