AudioStream - {local|remote media} → AudioSource|AudioMixer → {outputs}

Thank you for this detailed info! I was going about it all the wrong way I will try again today using this information- thank you!

 

Its working perfectly now! Thanks so much for your help!

 

Hi, if I want to use only internet stream radio functionality, is it necessary to download fmod or/and use fmod license?

 

Hi, yes - for the time being FMOD decoder is still being used for all internet streaming

 

Hi, the company I work for recently bought AudioStream asset and I've been taking a look at it. I'm not finding much documentation so I wanted to come here and ask. I'm not quite understanding what the best method is at the moment for what I'm attempting to do.

Our program allows the user to screen-share their desktop to other people in the room. So I wanted to share the audio along with that screen-share image. Perhaps capturing the byte[] of the current recording device to be sent to the server, then sent to the other clients to be read. So whats the best way to capture system audio on PC or Mac then play it back through an audio source?

Any help is greatly appreciated, thank you.

 

Most of you want is tied together in Icecast demo scene I suppose - so I recommend looking at it at least - that's Demo\Network\IcecastSourceDemo\IcecastSourceDemo
also input capture and e.g. AudioStreamMemory/Download are demo scenes worth to look at
otherwise your question is so generic that its answer is generally spread across the asset at various places, you have to be more specific than that

 

v 2.4 062020 4thMonth update should be on its way for review

Updates/fixes:
- AudioStreamDownload : added 'realTimeDecoding' which will force the decoder to run at realtime - this allows e.g. netradios to be saved
: added 'playWhileDownloading' option which will allow the audio playback while the download is in progress on user provided audio source 'audioSourceToPlayWhileDownloading'. Only available if the above is enabled.
- AudioStreamDownload
- AudioStreamMemory : not consistent finishing (resulting clip length) should be fixed

- resolved issue with playback on iOS when the application is not setup for recording: use 'AUDIOSTREAM_IOS_DEVICES' compile time define in iOS Player Settings to enable recording from Bluetooth devices and 'force to speaker' workaround functionality - OR - when the playback on Bluetooth devices is needed
(the player has to have 'Prepare for recording' enabled in Player Settings for both cases)

- tidied up demo assets to be stored and retrieved from separate directory in StreamingAssets in 'StreamingAssets/AudioStream'

New Components:
- MediaSourceOutputDevice : user API only (not extensive Editor support) component for playing back individual audio files on arbitrary output channels/speakers on a given output device simultaneously and independently
uses FMOD only directly, not for use with Unity AudioSources/AudioClips for now
- new demo scene MediaSourceOutputDeviceDemo
- MultichannelOutputDemo : renamed/moved to new AudioSourceOutputChannelsDemo

- AudioStreamDevicesChangedNotify : rolled previous two AudioStreamInputDevicesChangedNotify and AudioStreamOutputDevicesChangedNotify components into a one new one which should catch both input and output device changes in the system
(previously there were problems when both were used in one scene)

--
I would like to thank to all users who picked up the asset for testing and using it since mainly their feedback drove all these changes --

 

Hello, I imported the plugin to Unity2017.4.30, then exported the.apk file, and it crashed when I ran AudioStreamDemo and AudioStreamDownloadDemo.For another reason, playerSettings-otherenvironment-scripting would be imperative for IL2CPP, and we would like to ask you for a solution

 

Hi, please upgrade to at least 2018 LTS if possible - 2017 is not officially supported even by Unity -

In any case you can use 'AudioStreamLegacy' for playback (if plain HTTP transport is enough - if not then I'm afraid apart from upgrading there's probably not much I can do)
- the component is in 'Scripts/AudioStream/_LegacyUsingFMODNetworking/AudioStreamLegacy', you can use it the same way

As for download you can work around this by downloading the file itself using UnityWebRequest, then point AudioStreamLegacy to local storage path (though AudioStream should work with local storage files I think)

Sorry I don't have probably the best news, hope this helps nevertheless!

 

Thanks for your reply. Have a look at Changelog. Where can I download the V1.9 Demo

 

I'll send you DM with link to 1.9 version of the package

 

HI. I'm attempting to get a mutichannel audiofile (10ch in this case but I would ultimately like 24) to play back via AudioSourceOutputDevice. Alas I get the message:

"Error: Cannot create FMOD::Sound instance for clip "<my audio clip>" (FMOD error: The sound created exceeds the allowable input channel count. This can be increased using the maxinputchannels parameter in System::setSoftwareFormat. )."

I have set speaker mode to raw and appropriate number of channels in the device config scriptable object - but still no joy.

Oddly I also get this message on attempting to play the imported file in a virgin project with no audiostream or fmod assets installed.

Any help would be much appreciated!

 

Hi, when AudioSourceOutputDevice is used the clip is still processed by unity first ( that's why you're getting the same error in bare unity project too )
Have a look at newly added MediaSourceOutputDevice - it allows further customization, but not without some coding
In any case - please send/PM the file - I'll have a look later today

 

Thanks. link to download a 24 channel polywav (what I would ultimately like to be able to play). This is about 30s of so of repeated channel idents.

 

I'm not sure the file is _entirely_ in correct format : my desktop audio players either couldn't recognize it, or played only first two channels (but this might be just because they simply can't deal with multichannel audio properly) - but when opened in e.g. Audacity the first two channels were reported as one stereo channel (the rest are - supposedly correctly - mono channels)

However:

I made simple playback testing scene and the file seems to be playing without issues
- using the new MediaSourceOutputDevice component without any changes
For multichannel files FMOD lays out channels depending on the output, so this will be mapped from 0..23 on the output by default - which might, or might not be what you want
I didn't use any custom mixing, everything used in default setup - so if you need to change how it behaves on the output you'd have to do it e.g. based on demo
I'll send PM with the new demo scene and instructions

 

Thanks a lot - it looks like the 24ch file plays back correctly!
Yes - a lot of desktop players don't understand > 2 channels. Also - the 1st two channels reported as stereo in Audacity seems to be an Audacity bug (or feature?) but doesn't seem to affect operations. Many thanks again for getting me started here. Ideally I would have liked audiosource access to be able to use audiosource functions (time, pitch, volume etc) but it certainly isn't a deal breaker as long as I can adjust volume - which seems to be the case! I'll have a play and get back to you if I have any more questions.

 

yea the capabilities are indeed very basic currently
otherwise to match AudioSource capabilities i'd have to basically reimplement everything Unity is currently doing over at FMOD wrapper which.. is not entirely pointless, but for sure seems like a little bit redundant..
I however run into few cases which would be good to have also via FMOD directly before, so I'll maybe add more stuff over time / at least the volume is good choice also when testing everything, so good to hear it's useful ] ! /

 

Martin - sent you a PM regarding some extra functionality I've added to the MediaSourceOutputDevice class.

 

Feature Request- probably out of the scope of AudioStream but I'll ask anyway

Could you add the ability to route audio input from the Steam Voice api? There is no asset/simple way to do it- I bought the Steam assets and they are too complicated/don't include working examples.

So a way to take the active Steam voice chat initiated manually through Steam- take the the audio from the person you are taking to and route it to an AudioSource variable. Api is here https://partner.steamgames.com/doc/api/ISteamUser#GetVoice

Right now I'm doing it manually by using a virtual audio input/output and routing it to AudioStream- it works fine but it would be simpler to use the Steam voice audio itself.

Thanks for your time.

 

probably no - sorry - i don't work with the platform directly
users used the native mixer plugin wiht HTC Vive and directed audio from its systems to different PC output, but they had to turn off Steam Mirroring ( I don't know if it has even relevance to Steam Voice API really )
If there's no at least .NET interface for it it would be problematic for me to support it natively

 

small maintenance update submitted

===============================================================================
v 2.4.1 072020 5thMonth

Updates/fixes:
- iOS AVAudioSessionWrapper : moved alongside other input components for better/proper asmdef compatibility
- MediaSourceOutputDevice : added few user methods provided by Jon Olive ( @Jon_Olive )
- AudioStreamLegacy : improved insufficient network data state
: demo scene AudioStreamLegacyDemo disables UI controls so user can't stop the stream until thes tream either continues or stops completely
: - stopping / pausing the stream when in this state led to freezes previously; this situation is resolved in newer component already

New Components:
- AudioSourceCaptureBuffer : to ease the retrieval of filter audio data between game objects (must be in sync with OnAudioFilterRead)
: not used in demo scenes - needs to be attached on a game object with AudioSource

New demo scene:
- MediaSourcePlaybackDemo : plays an audio file (from StreamingAssets) directly by FMOD via MediaSourceOutputDevice component
: 24 channel wav audio file used for demo
: file provided by Jon Olive of MagicBeans Physical Audio Ltd. [https://www.magicbeans.xyz/], used with permission (thanx Jon!)

 

Hello- I'm getting these GC spikes whenever I play audio with Audiostream in the editor-

This only happens when I submit an mp3 to be played by AudioStream- It produces one of these spikes every 6-10 seconds or so when a small mp3 file is being played causing a hiccup in the audio-

Here's a capture of a deep profile- strange about Cinemachine in there as no Cinemachine cameras/objects are active in the scene 0_0


Is this normal Fmod behavior or is there something I should look at to resolve it?

Thank you.

 

can you PM me the file (download link) ? please mention unity version too
cinemachine might be more tightly integrated if you have it in the project (haven't verified this though)

 

Hi. I'm considering using FMOD and this plugin in the Windows app I'm working on. Would this allow me to do the following?
1) Bring in multiple Windows playback(loopback) and/or recording audio devices into Unity
2) Create and display a mixer type interface to control panning and volume levels of each of those devices
2b) Create extra mixers containing multiple combinations of these audio devices
3) Create multiple simultaneous audio visualizers each having their own device source
4) Output audio from each mixer created in 2b to multiple different audio devices on the system

 

Hi,

please download demo build from the store page and run AudioStreamInput/2D scene/s - you will see your devices in unity scene, have a close look at all/other scenes as well

the asset uses Unity AudioSource/s for its output (in case of input devices at least) - so you can base your solution on them

you'd have to mix your individual signals into one single AudioSource which you can then output to a given device
but note there's no direct support for this it'd have to be in C# via e.g. audio filter callback
[more optimal would be to create custom DSP effect directly in FMOD]
I recommend using included mixer plugin for redirection if you're on standalone, but you'll run into latency issues regardless - something to be aware of
hope this helps! let me know if you have further questions

 

Thanks for the quick reply. I have tested the demo build. I should have specified in my question for number 1. I meant can I have multiple windows audio devices simultaneously have live input into Unity?

 

yes - each input device is basically a script attached on a game object - in the demo there is single game object only each time with different device # set based on user selection, but nothing prevents you from having more than one in the scene
(technically the way the asset is currently structured there is limit on number of FMOD systems created simultaneously, I'm not sure about the exact number right now, but you should be ok with 10 or so separate inputs - if not please feel free to let me know)

 

Ok, that's great to know! I would need more than 10 in some scenarios. Really would depend on the users of the app and I wouldn't want to limit them to 10. I'm waiting to hear back from FMOD if the app I'm building would be able to use their standard Commercial use license.
Side question. I noticed the CPU usage on the 64bit demo app is quite high, even on the main page. Is it supposed to be so high?

 

ok will have to think about this and do an update most likely - shouldn't take long but i'd have to get to it in the first place

that's unity rendering on the first/main scene (maybe I left vsync off not sure) - otherwise there's nothing which should be taking extra CPU time audio-wise - the heaviest in the demo app is IMGUI rendering tbh
but I'll have a look next time building it

 

Great! Thanks again for the quick replies I look forward to trying this should the FMOD license work for my app.

 

What are the recommended settings to get .pls streams to play in iOS? I see in the docs that the stream type AUTODETECT does not work in iOS, but I can't find a setting that does work. Even the demo app fails in latest iOS with the demo .pls stream, no matter what stream type I select, log says Unsupported file or audio format. I'm on v 2019.4.8.

 

If the link is served over plain HTTP use legacy component (AudioStreamLegacy{Minimal}) on iOS please (it's virtually the same, only uses FMOD networking; picking the correct format still applies)
If you need secure HTTPS on iOS that won't work right now
[- I want to do this differently for stream player but it's a bit more work]

 

That's working for me, thanks! Keep up the good work; I know it's tough to keep moving forward while maintaining support for the Old Ways.

 

Great!
& Thanks for words of encouragement - much appreciated!

 

Ok so I purchased the plugin to start testing if it will work well for my project.

I'm getting errors (see attached screenshot) in my console after performing the following steps:


1) Downloaded and Installed FMOD Studio 2.01.04 Windows x64
2) Downloaded from FMOD, Unity Integration Package 2.01.04
3) Setup new URP project for x64 Windows
4) Imported the package from step 2
5) Created new FMOD Studio .fspro project file with bank
6) Specified the .fspro project in the FMOD settings in Unity project
7) Imported latest AudioStream package

 

hah yea i think it's because of changes in latest version of the FMOD package - please resolve the ambiguity by explicitly referencing System.Threading at the places where the error occurs until I push update to the store (feel free to let me know /DM otherwise)

btw you don't need FMOD Studio installed on the machine for asset to work and from FMOD Unity package need only specific bits if you don't intend to actually use Studio - it's mentioned in the README at the beginning (it doesn't change functionality itself just helps to keep project more clean, there's no difference otherwise)

will try to submit update including input system count limits mentioned earlier later

 

Just submitted an update, should be online hopefully soon

Due to recent breaking changes in lastest FMOD stable update (which is currently not yet updated on the Asset Store and is only available on official site) the asset requires at least 2.01.04 version to work

v 2.4.2 092020 M#6

Updates/fixes:
- MediaSourceOutputDevice
: further user facing functionality
: - defer playback to user after sound creation, allows to completely release the sound only when needed (not when e.g. just stopped)
: improved behaviour and fixed channel release/stealing/looping functionality
: - channel returned to the user might change as result of the call
- MediaSource demos
: updated as per above

- AudioStreamInput*
: all input sounds are created on a single FMOD system, so their # (# of 'input' game objects) is no longer limited to allowed max. system count
: - this also means that it's no longer possible to change/update DSP buffer properties at runtime - this should be hopefully acceptable trade off though -
: but might be a breaking change if you were updating DSP buffers at runtime
- AudioStreamInput demo scenes
: removed custom DSP buffer changes at runtime due to the above

- latest FMOD 2.01.04 update
: tested against latest FMOD update - FMOD 2.01.04
: Please be advised that this brought breaking changes, in a way which are not backward compatible with previous versions (again).
: in other words you need FMOD 2.01.04 (and up) for asset to work
: - updated initial setup in README
--------------------------------------------------------------------------------------------------------

 

Something I thought I would mention that I noticed. Having a AudioSourceOutputDevice script expanded during Play mode in the Inspector uses a lot of CPU.

...Actually, upon further testing it is any component/script with a live meter displaying in the Inspector.

 

you're right I've never noticed it hah
it's the OAFR meter itself - once there is at least one displayed in the Inspector in Play mode the CPU usage goes up
- in 2017 the CPU usage goes up immediately after it's shown, in 2020 only after it's repainted based on actual audio (sometimes it takes a while for it to kick in for some reason)
They will probably address this over time but it still might be worth a bugreport I think

 

What does "OAFR" mean/stand for?

 

the audio level meter shows up in the inspector when you implement OnAudioFilterRead [ https://docs.unity3d.com/ScriptReference/MonoBehaviour.OnAudioFilterRead.html ] method on a MonoBehaviour
sorry for using the abbreviation only %)

 

Hello everyone, i cant start assets demo, and i get this ERROR
[ERROR][2020-09-17T14:01:43] sound.getOpenState ERROR ERR_FILE_NOTFOUND - File not found.

How i can fix it?

 

what platform are you doing this on and what exactly are you trying to do @karlovvemail ?

 

Windows 10, i now got new problem,

NullReferenceException: Object reference not set to an instance of an object
AudioStream.AudioStreamLegacyBase+<PlayCR>d__63.MoveNext () (at Assets/AudioStream/Scripts/AudioStream/_LegacyUsingFMODNetworking/AudioStreamLegacyBase.cs:497)
UnityEngine.SetupCoroutine.InvokeMoveNext (System.Collections.IEnumerator enumerator, System.IntPtr returnValueAddress) (at <d815b7efac424eeb8e053965cccb1f98>:0)

 

Windows 10 is not a platform, but operating system
I was asking because on e.g. Android you have to extract the file from the apk and place it in some readable location in order to access it directly ( which you're apparently trying to do )
The error you posted is related to network proxy - are you using network proxy and do you have latest version of the asset ?
Can you turn on logging and post the log here ?
What are you trying to do exactly ?

 

@r618 Is there a way to get the loopback (the system sound - e.g. playing youtube) without any input device connected (e.g. microphone).

I see in the AudioStreamInputDemo that there is loopback available as an option. But if no input device is connected (to the mac in my case) then AudioStreamInput will

 

it should be possible, though i've never tested this on a mac _without_ microphone / any physical input )
what does the demo scene show ?

 

The demo scene without any physical input shows ("There seems to be no audio input device connected").
I checked the implementation a bit deeper and what I found is that in AudioStreamInputBase class

Code (CSharp):

1. this.recording_system.system.getRecordNumDrivers(out numAllDrivers, out numConnectedDrivers);

returns 0 for

numConnectedDrivers

The same in

FMODSystemsManager.AvailableInputs

I see that when the loopback=true flag is passed then this additional (virtual?) device is added, but only to the appropriate physical device. So it seems like it has to be a physical device to be able to add a loopback (is it a case? what loopback means in this case?)

Code (CSharp):

1. availableDriversNames.Add(new INPUT_DEVICE() { id = i, name = name, samplerate = recRate, speakermode = speakermode, channels = recChannels });

Is it possible to somehow cheat and add such a loopback device even though there is no physical device connected?

On the other hand, I made more research about the topic I check e.g. how the Zoom app (https://zoom.us/) is doing this on mac. I found that when you do screen sharing with audio Zoom app install in the system their custom device (driver) that then the Zoom device in system sound settings is visible. Probably this is the way how they get the system sound by the virtual device.

 

can you replace existing for iteration

for (int i = 0; i < numConnectedDrivers; ++i)

with

for (int i = 0; i < numAllDrivers; ++i)

in the 'AvailableInputs' method ?
[the loopback flag is working already only it filters over empty list]