AudioStream - {local|remote media} → AudioSource|AudioMixer → {outputs}

@Yuuki-Mochiduki
hm looking at code now I couldn't spot anything suspicious
does the demo application (as downloaded from the asset store page) 'OutputDeviceDemo' scene work ?
Does the demo scene play and exit correctly when you run it in Unity ?

- there are no special restrictions on user code using the component, it should create and release system ( releasing the system is where most freezes occur ) automatically
How many audio sources are you using in the scene ?

 

Thank you for your reply.
The problem was caused by my coding. I did not wait that AudioSourceOutputDevice is ready.
It worked correctly.

 

glad you got it resolved -
I should probably add check for this when switching devices, too
thanks for letting me know !

 

I'm creating a bookplayer with audio playback and I came across this plugin. Basically I need to play audio from a remote server and since clips can be quite long, I would like the capability of playing it while the clip is downloading.

A few questions regarding the plugin:
- Does it allow playback while still downloading?
- Does it support seeking to a different part of the audio clip?
- What would happen if attempting to seek to a part that hasn't been downloaded yet?

Thanks!

 

Hi,

- that was primary reason for it to be made - though the asset has separate component just for download, too

- not currently, working on it, watch this thread to be notified when the submission is ready

- I want it to work over a small-ish cyclic buffer so seeking backward/forwards will be limited to a few seconds
the way it should be working is if you seek back, it will accumulate download data

btw the demo application on the asset store page has all current features included [I'll update demo too after the update is ready]

 

Thanks for the quick reply!

I was able to achieve this by myself on osx, ios and android without much trouble but Windows is a whole other story :/

I'll keep an eye on this thread as you suggested

 

Hello again.
I wanted to ask about Recording System Audio on Android devices. I built the demo for Android, but my Android's speaker and microphone don't show up as available devices, the list is blank. I have a Moto Z Play Droid running Android 8.0.0

 

Hello,
- I saw similar issues on the internets already - is there are a possibility you can update to 8.1 ?
- just to verify this - you can send me adb log when running on device
( use something like 'adb logcat -s Unity ActivityManager PackageManager dalvikvm DEBUG' , you can set log level on the component to DEBUG ( but this is probably not needed ) )

Alternatively: please PM me your invoice number and I'll send you updated package which uses FMOD 2.00
although this will probably not help ( the problem was present even with 2.00 ), we can test it out

 

My Android won't update beyond 8.0.0 for some reason. Here's the adb log. Seems like it couldn't find fmod for some reason, but it seems to work fine on my desktop pc.

Spoiler: ADB LOG

09-25 11:46:52.781 12764 12787 I Unity : Enumerating input devices on Android - 0: Android audio input
09-25 11:46:52.781 12764 12787 I Unity :
09-25 11:46:52.781 12764 12787 I Unity : (Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 35)
09-25 11:46:52.781 12764 12787 I Unity :
09-25 11:46:52.919 12764 12787 E Unity : DllNotFoundException: fmod
09-25 11:46:52.919 12764 12787 E Unity : at (wrapper managed-to-native) FMOD.Factory.FMOD5_System_Create(intptr&)
09-25 11:46:52.919 12764 12787 E Unity : at FMOD.Factory.System_Create (FMOD.System& system) [0x00000] in <654d99584712412ab412601a16cc5e62>:0
09-25 11:46:52.919 12764 12787 E Unity : at AudioStream.AudioStreamInputBase+<Start>d__35.MoveNext () [0x00089] in <654d99584712412ab412601a16cc5e62>:0
09-25 11:46:52.919 12764 12787 E Unity : at UnityEngine.SetupCoroutine.InvokeMoveNext (System.Collections.IEnumerator enumerator, System.IntPtr returnValueAddress) [0x00027] in <49dc8bbb4d6d422d920e0c799b6b06a1>:0
09-25 11:46:52.919 12764 12787 E Unity :
09-25 11:46:52.919 12764 12787 E Unity : (Filename: <654d99584712412ab412601a16cc5e62> Line: 0)
09-25 11:46:52.919 12764 12787 E Unity :
09-25 11:47:15.872 12764 12787 I Unity : AudioStreamInput2D [INFO][2019-09-25T11:47:15] Stopping..
09-25 11:47:15.872 12764 12787 I Unity : =======================================
09-25 11:47:15.872 12764 12787 I Unity :
09-25 11:47:15.872 12764 12787 I Unity :
09-25 11:47:15.872 12764 12787 I Unity : (Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 35)
09-25 11:47:15.872 12764 12787 I Unity :

 

hm that looks like a different issue though
are you sure the FMOD Integration package is properly imported and all Android native plugins for required architecture are in place ?
just to verify: you should be using FMOD 1.10.1x and required bits are in Assets\Plugins\Android\libs\[architetcure] folders
(btw binary dlls are not by default added to version control usually)

- another thing to check - please download demo apk from asset store page and run it on device
EDIT: if you got zero inputs with demo, please paste log from the run of it on device, too


Thanks !

 

Looks like you were right, I had integrated fmod wrong and the Android plugins didn't import. I'm getting the correct input now.

The only issue I have now is that if I have background music playing, it doesn't pick up and react to the audio in AudioStreamInput2D or AudioStreamInput

 

That is as far as one can get on mobiles right now I'm afraid, see e.g. this answer:
https://forum.unity.com/threads/aud...all-and-everywhere.412029/page-8#post-3745849

This would need iOS's AudioBus-like equivalent on Android for inter-app audio
- in other words my plugin is useless for this right now, sorry -
- even if something like AudioBus was running on Android, both application would have to support it, or it would have to create a new system audio device/output - since my plugin works only with those - and inter-app audio plugins usually don't work like this, so full 'AudioBus-like SDK' would have to be integrated into Unity app (of which I'm not sure if one is even available atm)

As sticklezz mentioned in the reply below the linked one he used microphone for this :: that is probably the best advice I can give right now :-|

 

Hi there,

Can this asset be used to redirect several microphone input device as well as the game audio to one AudioSource? I am trying to have all the sound from the game, plus the sound of one or two microphones in one AudioClip I can save all of it as a wav or mp3 file.

I am on the Windows platform.

Thanks!

 

Hi,
do you actually need an AudioClip ?
If you need to process "all game audio" you can just do it on the main listener which plays everything in the scene in the end..
You can attach 'GOAudioSaveToFile' on it ( it's usually on the main camera ) and let is be saving everything what's being audible in the scene

(but note unless the output is being played on e.g. headphones you'll get feedback from microphones anyway - you'd have to do some more scripting to silence mics after they are saved)

Let me know if you need help with it, or you really need something like an AudioClip being made!

 

Thanks for the quick response!

We would actually need to silence the mics because we would like to record them but not play them in the scene.

I know it might be outside of the scope of this forum/plugin but to give you the full picture, we are using a video capture plugin (AVPro Movie Capture) that record both a camera and the audio. Their plugin can record the audio from an AudioListener only. I am trying to see if there's a way to capture microphones and game audio at once.

If this is not possible, I was thinking to capture all the audio separately with your plugin and see if I can merge the video and the audio file after the recording with C#.

 

Depending on how you have your 'microphone' game objects set up - and since I don't know how the movie capture plugin works - you might be able to capture mics audio manually in OnAudioFilterRead, send/expose it for merging with scene audio to another script running on the main listener and then silence them ( and avpro plugin will still work )
You're right that my asset doesn't have anything ready for this, but it should be possible to write set of scripts for this -
(I'm not sure atm if simple write/read via OnAudioFilterRead will work without any sync issues, or you'd have to queue pcm packets);
let me know on discord https://discord.gg/5ZyPqeA or here if you need more info

- capturing audio separately and then merging with video will work, but it's worth to try ( in my case find out ) if it's possible to do it directly with the movie plugin first

 

submitted (the long overdue) 2.0 update for review:

v 2.0 102019

- updated for FMOD 2.00.xx
(asset is not compatible with 1.10.xx OOTB any more)

Updates/fixes:

- AudioStreamBase : replaced network retrieval backend for FMOD with UnityWebRequest - this means all common http/s connections features work, including redirects and POST headers (to a degree UnityWebRequest provides them)
- AudioStream demo scene/s : you can choose between MEMORY and DISK caching before starting playback: DISK allows seeking within the whole content downloaded up to that point (compressed audio is stored in application cache)
! Important ! - format compatibility went down though - in general MPEG format works, other formats might need setting new 'Decoder Block Size' to the physical file size before playback - this should be updated/fixed later -

- AudioStreamInput* : added builtin automatic notification change on system record devices change - an user UnityEvent is invoked where the new updated list can be queried
- AudioStreamInput* demo scenes : hotplugging of input devices is supported - you can dis/connect your devices at runtime and their list will update

New Components:

- AudioStreamLegacty and AudioSteamLegacyMinimal : previous versions of AudioStream and AudioStreamMinimal are left intact including their demo scene and renamed to AudioStreamLegacy and AudioStreamMinimalLegacy

General:
- removed redudant testing in native plugin (Unity project opening/startup shoudl be faster now)
- added new section to demo app for Legacy components
- updated scenes and code for 2019.x/LTS compatibility


Please do download the demo application for your platform, and test all functionality with your assets to get a feel what the asset can do before asking questions
(every component included in the package (well, except the WAV saver) has its own demo scene)
(demo applications which links are on the store page are already built with 2.0)

Thank you !

 

Hello,
I'm getting error:
Assets\AudioStream\Scripts\AudioStream\AudioStreamBase_CaptureDSP.cs(45,40): error CS0122: 'DSP_STATE_FUNCTIONS.getuserdata' is inaccessible due to its protection level

I've made a clean project with no assets other than AudioStream and the FMOD Unity package (2.x).
No other errors or warnings...

 

Hi @rayDB, yeah it's known issues for now -
it's limitation in current fmod integration ( it should be fixed in next release as I found out today )
just write 'public' in front of `getuserdata` in DSP_STATE_FUNCTIONS definition in fmod_dsp.cs please (you should be able to go to definition in your IDE)
it's mentioned in the readme, but I guess there's lot of information to follow there already

 

Incredible response time! (12 minutes) Thank you. That fixed it.

I know this isn't directly AudioStream but... Do you know if the FMODStudioSettings need to have a project path set?

I found DSP info in the readme (like you said), but nothing about setting a path for FMOD projects. It's generating an Debug:LogError "error", but it's not a compile error - though it is showing red (LogError instead of Log).
Is that due to including the things mentioned in the readme as unnecessary to include?

EDIT:
I used Multiple-Platform Build, and pointed it to Assets\Plugins\FMOD\lib
That made the error go away (but I don't know if that's correct)
I don't care about using FMOD outside of AudioStream.

 

i just happen/ed to be browsing the forums, no worries )

as for FMOD Studio - yeah, it needs a project path to be set, but note that only if you actually do *use* FMOD Studio project in Unity
- the asset itself doesn't need it and you can remove most of the stuff FMOD Integration contains - the only necessary bits which are currently actually needed for it to function are native plugins in Plugins\FMOD\lib (of which you can choose only platform/s you care about) and Plugins\FMOD\src\Runtime\wrapper
you can safely delete everything else
this is actually covered in the readme, too - let me know if you found it explained there not clear/accessible enough

If you do use/need FMOD Studio then disregard the above and just set the FMOD project path in FMOD settings and sync the project

 

Thanks for the quick replies. It was there in the readme - It is a bit dense, but makes sense.
I just imported the whole FMOD package, and their wording for its settings seemed weird. Not only that, but for some reason it took 2 fresh project attempts importing the FMOD package into a new/clean 2019.2 project for the FMOD menu item next to "Window" to show up. I was getting frustrated and wondering if the FMOD doc was old or something (trying to find the menu item elsewhere), then it finally decided to show up after a few project restarts.

I imagine you'll hear back from me later today. Maybe a question or two, but hopefully just a glowing review

 

I just saw your edit above @rayD8 : the project path FMOD Integration needs is a separate project created in FMOD Studio ( it's a separate authoring tool / application, outside of unity ) - so pointing it to Assets\Plugins\FMOD\lib most likely wasn't the most optimal solution so to speak ) - it might have caused the importing errors I think

: they distribute both low level API, and Unity support for FMOD Studio in one package together, unfortunately

 

One important thing I forgot in the above about the initial error from package about FMOD project path - just for the record - :
the error message is *harmless* first and foremost - it just can't find FMOD Studio project which it wants to use, but everything else runs normally otherwise, it will just complain constantly in the console

 

2.1 update should be on its way !
-- playback of Ogg/Vorbis format from network still suffers, but otherwise should be more stable for everything else *

v 2.1 112019

Updates/fixes:
- AudioStreamInput* : added resampleInput setting which can turn off resampling to output samplerate for better compatibility with other assets which do their own encoding such as AVPro Movie Capture
- AudioStreamInput* : fixed Un/Pause seamless continuation
- AudioStreamInput* : fixed bug with multiple inputs being notified
- AudioStreamInput* : AvailableInputs now returns struct describing an input rather than just string device name
- AudioStreamInput* demo scenes : added UI option to filter out non/loopback interfaces

- AudioSourceOutputDevice : AvailableOutputs now returns structs describing the output/s rather than just string device name
- AudioSourceOutputDevice : added hotplugging notification for output devices during runtime
- AudioSourceOutputDevice demo : available device list should react to devices changes at runtime

General:
- fixed few format compatibility issues and updated AudioStream* to correctly point to local storage when needed, updated README and demo scene
- split !!README1ST!! and general documentation

 

Was just testing out new hot plugging support for input and output devices - looks like remote iOS devices can be recorded from without issues on desktop macOS Unity application - so I made a quick video :

( device is connected via Lightning/USB cable, using standard functionality in Audio MIDI Setup on macOS )

 

When using Audio Source Output Device with unity auido source, the log like below will appear freaquently.

PCMReadCallback 343841816 underflow - requested 38400 while having 12288 (providing 12288),  time: 214.869333333333

Is this critical error? Or can I ignore this?

 

you can ignore it if you don't mind dropouts in the resulting audio (functionally it will still 'run')
- you can make this better, however: with latest version I found than it works for me with 'Good latency' Unity Audio DSP buffer setting, and notice the 'Input latency' slider on the AudioSourceOutputDevice component : I set this to around 200 ms - there should be then no dropouts even in the Editor
so if you don't mind larger latency you should be able to get rid of it completely
(demo is built with Best latency Unity setting and it's not ideal for resulting audio on PC/Windows)

technically it depends also on your audio driver so there's a limit on how much you can go down with latency
I recommend to start with cca the above values and you can try decreasing Input latency on the component -
testing should have better results in build (not Editor), and with IL2CPP backend

 

and @Yuuki-Mochiduki if you have significantly different inputs and outputs (difference between default - the output on which Unity plays audio by default - and requested device) in terms of their channels and rates, let me know their settings and I'll look into it some more

 

Does this have an API that I could hook into directly to play clips on certain output devices? We have our own component for "advanced" audio sources, and play most things using PlayOneShot instead of directly playing clips.

My main desire it to be able to send clips to different audio devices, as we use bass shakers with custom equalised audio clips and want to be able to still play the "normal" audio through speakers as well. Could I do that through this? Essentially it would boil down to the following that I'd like to do (imagine I can pass an audio device to PlayOneShot):

Code (CSharp):

1. audioSource.PlayOneShot(bassShakerAudioClip, bassShakerAudioDevice);
2. audioSource.PlayOneShot(normalAudioClip, normalAudioDevice);

 

Hi, everything behaves and works with AudioSources/AudioClips as usual, but the redirection component acts on a single AudioSource - so in your example you'd need two AudioSources each having their own redirection component with each respective output device id and call PlayOneShot on them as usual
Note that latency might be a concern - though I've implemented latency slider (tm) on redirection component, you can't drive it to 0 - one option is to put all AudioSources under redirection including 0 (default output) so they can play 'in sync' with the same delay
/ if you're on standalone (win64 & mac) you can also use audio mixer + redirection plugin and send required audio on its output that way, latency is slightly better /
Hope this answers it, let me know if you have more questions ~

 

The mixer plugin sounds like a good option, as it's always Windows for us.

Do you know if PlayOneShot "follows" the mixer group of the audio source (i.e. if you change the group mid-playback, will it keep playing on the old mixer group or move to the new one for the rest of the clip)? Just trying to think of ways of not needing to rebuild our entire audio stack really. I could just create a new AudioSource on the fly when necessary too perhaps.

How does one actually specify the audio device to use for redirected audio by the way?

How much latency are we talking?

 

No - but I'd guess that if you change audiomixer group mid-play on the AudioSource at runtime it might switch to the new group

You're about to structure your code around different output devices anyway so playing it on a different audio source isn't that much of a penalty - which you have to pay somewhere if you want to use different outputs anyway

You either enter device id in the Inspector, or call SetOutput(int outputID) on the component at runtime

It's non trivial and depends also on your hardware+drivers - I'd say one should be able to impose additional 20-50 ms on top of Unity latency under ideal conditions (but Windows is notoriously not good at this unfortunately, so probably more)

FWIW there's a demo scene with instatiating prefab/s playing on selected output/s in the demo app

The asset is structured around each AudioSource's audio buffer - otherwise everything would need to be handled in one/central place with each audio buffer callback containing information where to direct it - which would have some overhead and probably more complex code

 

oh, and yet another FWIW: I think the latency set in the Windows demo online is currently wrong - it's unnecessarily big (this was set in hopes that it would run more hardware more seamlessly, but it's most likely counterproductive: it might not behave well with sounds being dropped, so actually should be brought down - will have to test this later)

 

20-50ms isn’t the end of the world for our purposes really, people won’t notice when it’s for haptic output thankfully

Good info thanks, regarding the device ID: where would I get that ID? Is it something I can potentially get via Windows for ease (as we configure stuff through a JSON file) or would I need to iterate through devices and go off the name or similar for best results?

 

It will have to be most likely more to avoid dropouts in the audio - hence the slider I added for latency of the output buffer - you will have to find what value works for your environment, - IL2CPP build helps for this too

It's assigned by system, the asset enumerates this automatically - you can see device ids in demo scenes which use them (demo scenes with input and output selection)
It's machine specific, so either have fixed hw configuration or let user choose it (again it's good to follow code in the demo scenes)

 

Hello everyone! After a lot of searching I found this plugin. I have a scenario where the player needs to be able to speak to a virtual microphone inside the game and hear his/her voice through the headphones. Right now I have a lot of latency using the native unity sound engine. Can this plugin perform this scenario? If yes can it be done through UI (my coding kung fu is amateur at best)? If yes, is the latency less than the native unity sound engine? If it can do all of the above, you have a buyer!

 

Hi, everything you describe should be testable in the demo (download links are on the store page), for low latency mic see 'AudioStreamInput2D' scene
(I'm not sure what you mean by 'virtual microphone' though, but I'll assume normal mic input on the machine)
Uncheck 'Mute output' in the demo scene to be able to hear mic on the default output

Demo uses IMGUI (Unity's previous immediate mode UI), current UGUI is different and needs different approach and some coding, but the interaction with asset's components is exactly the same

The asset itself needs some reading and setting up too - ask here or PM me if you get stuck -

 

I

 

replied via email

 

Hello.
I have a client that wants a feature to redirect mics to speakers on the one machine...

So mic #1 -> speaker X, mic #2 -> speaker Y... effectively 2 people speaking on mics, and output redirected to the headphones each other is wearing, all on one computer.

Does AudioMaster plugin support this (seems to..) ?
Do I need an FMOD studio license is required also?

 

Hi,
you have probably two options out of the box:
- either use combination of [AudioSource] + [AudioStreamInput2D] + [AudioSourceOutputDevice] components for both microphones
- or [AudioSource] + [AudioStreamInput2D] and route AudioSources to separate mixer groups with AudioStreamOutputDevice effects
where you set your output device on each AudioStreamOutputDevice component/effect

I recommend reading documentation first

for FMOD licensing see : https://fmod.com/licensing : (but basically it's free for 1 game/product per year if your budget fits)

Have a close look at demo scenes (demo builds are available on the store page, too) where everything mentioned above is used separately - it will be necessary to put all required components together

 

Hi,
Not able to get Option 1 to work so far.
I can redirect the mic input fine.
I can redirect an AudioSource fine using AudioSource + AudioSourceOutputDevice and calling AudioSourceOutputDevice.SetOutput()

But if I add a AudioStreamInput2D then the audio is not redirecting.

I notice that in the Output demo, the AudioStream.SetOutput and AudioStreamMinimal.SetOutput are called instead of the AudioSourceOutputDevice.SetOutput(), but AudioStreamInput2D doesn't have a SetOutput.

I'll try the mixer approach instead...

 

Option 2 works perfectly thanks. Created a mixer for the microphone audiosource and exposed the OutputDeviceID so I can set that in script.

 

option 1 should work too - but the order of attached components matters - OutputDevice should be last (at the bottom, or below input at least, I think)
SetOutput on AudioStream/Minimal is just helper method which calls AudioStreamOuptutDevice's method, or FMOD directly depending on component
you should be always calling SetOutput on AudioStreamOuptutDevice in this case - you are not using playback components and Input* components do not have it
[AudioSource] is needed for audio buffer (it's not added automatically since input can run without it, but in this case ASOD needs it)

but if mixer works for you it might be even easier and more comfortable this way
good to hear it's working ~

 

I am usually getting distortion from the output mic->speaker. It's barely legible. I've had it very clear once, but most times its quite distorted like a poor connection. Haven't experimented with DSP buffer size or resample tickboxes yet... trying that now. Its a VR scene that is quite CPU & GPU heavy so not sure if thats starving the mixer somehow.

Not sure if there's anything else tweakable other than the "DSP buffer size, resample input, and use unity to resample" options..

 

Hello, the effect on mixer group is very cheap - it should work under heavier CPU load
couple suggestions:
- if you're using mixer make sure that AudioSourceouptuDevice component is not even attached on the game object
- can you test microphone component alone ? : i.e. don't route AudioSource's output to the mixer group and let single mic play on default output (with AudioSource and AudioStreamInput2D being the only components attached on the GO)

That said, I think I fixed an issue when starting the mic sound automatically on startup with 'RecordOnStart' checked - if you're starting automatically lmk, I'll send you an update

- default DSP buffer/s settings/sizes should work for most circumstances, - you can try to change things, if sound is distorted usually increasing buffer sizes/counts helps - at the price of higher latency
- let unity resample the input - custom(non unity) resampling might not work on the device (I don't think it's stable
in all situations esp. when input and output channels counts don't match - will have to have closer look at this)
This has a downside that the sound might drift if running for a longer time, but I don't think there's a choice in this case
- resample input should be always ON - it's rather special case for usage with another 3rd party assets (which do their own resampling)

Downside is that all of the above has to be tested on the device to make sure, editor tests are not enough

 

just submitted for review

v 2.2 022020 Corona

Updates/fixes:
- Demo scenes : put lists containing potentially many items [inputs/outputs] under scroll views for friendlier UX
- AudioStream : simplified memory buffer
- AudioStreamInput* : added warning message explaining its dependency when none AudioSource is set
- AudioStreamInput* : fixed recording devices enumeration and startup when RecordOnStart is true
- AudioStreamInput2D : updated automatic mix matrix when Speex resampler is used
- AudioStreamInput2D demo : automatic latency settings update when Speex resampler is used

General:
- checked w/ latest FMOD (2.00.07)
- Windows native x86 plugin is now being included
- the check for demo assets being present in StreamingAssets for demo scenes to work is now simplified and less resources hungry in the Editor
- tested disabled Domain and Scene Reloading in Unity 2019.3

 

I want to do live streaming of audio in my application. I tried windows demo of your asset and it is not working for rtsp protocol URL . But it is working for radio URL. Is it possible to do live streaming of audio through your asset? I am very much interested in buying your asset.

Thanks.

 

RTSP probably won't work; what is your source ? - do you want to stream between two instances of your application ? ( I didn't fully get that )
if so, start the demo twice and run AudioStreamNetMQSourceDemo in one and AudioStreamNetMQClientDemo in other, enter correct IP and connect (this is LAN restricted; I haven't had time to look at Mirror properly)
if not send me the link you want to stream, I'll check it out