Search Unity

  1. Welcome to the Unity Forums! Please take the time to read our Code of Conduct to familiarize yourself with the forum rules and how to post constructively.

AudioStream - an audio streaming solution for all and everywhere

Discussion in 'Assets and Asset Store' started by r618, Jun 19, 2016.

  1. SimpleAssets

    SimpleAssets

    Joined:
    Jul 14, 2021
    Posts:
    19
    Many thanks! Now it worked!
     
    r618 likes this.
  2. r618

    r618

    Joined:
    Jan 19, 2009
    Posts:
    1,171
    thanks for confirming
     
  3. SimpleAssets

    SimpleAssets

    Joined:
    Jul 14, 2021
    Posts:
    19
    Now I have a problem with the iOS build =)
    On android everything works fine, but on iphone the microphone does not work. I also changed "AudioStreamInput2D" to "AudioStreamInput_iOS", but without result, I can't hear the microphone. I also didn't see any error logs that could point me to the problem.
    I also tried to build the demo scene "AudioStreamInput_iOSDemo". Scene doesn't work either, I don't see a list of available input devices in the scene

    Tested on Iphone 6+ with os vesrion 12.5.2

    Maybe I missed something?
     
  4. r618

    r618

    Joined:
    Jan 19, 2009
    Posts:
    1,171
    please follow recording setup for iOS : https://github.com/r618/AudioStream...defa1af31e8804169d0546/Documentation.txt#L439
    use AudioStreamInput2DDemo scene, AudioStreamInput_iOSDemo is for external / BT devices ( I should probably rename this )
    lmk how it goes
     
  5. r618

    r618

    Joined:
    Jan 19, 2009
    Posts:
    1,171
    ==========================================
    New asset just submitted for review : AudioStreamIce <- please keep questions if any to this thread
    - replaces internet/streaming parts of this one
    - has better/complete iOS integration (no Android for now)
    will take some time until it's published
    upgrade pricing should be available in time too
     
  6. nlp_eschipellite

    nlp_eschipellite

    Joined:
    Aug 2, 2017
    Posts:
    7
    Hello,

    I'm in need of a plugin that can allow for the swapping of the current audio output device at runtime within Unity. Effectively I just need to swap between two defined audio devices based on the application's current state (headphones versus speakers).

    It seems like your plugin can accomplish this, but I was hoping to get confirmation if that was the case. Furthermore, I was hoping to know if an FMOD license would be required if I was just looking to utilize your plugin for the specified device swapping.

    Thank you
     
  7. r618

    r618

    Joined:
    Jan 19, 2009
    Posts:
    1,171
    - please run 'AudioSourceOutputDeviceDemo' and/or 'OutputDeviceUnityMIxerHotpluggingDemo' to play audio on user selected output; scenes also react to devices changes at runtime, i.e. plugging in/out
    first one is audio on a single GameObject, the latter is unity mixer effect
    - since FMOD is used their licensing applies, yes, for FMOD licensing please see here https://fmod.com/licensing

    Hope this helps!
     
  8. SimpleAssets

    SimpleAssets

    Joined:
    Jul 14, 2021
    Posts:
    19
    Hello, is there a way to list all output devices on Android and ios? With the editor, I see all the outputs, but on mobile devices, the output device is always only one.
     
  9. r618

    r618

    Joined:
    Jan 19, 2009
    Posts:
    1,171

    yea it works differently on mobiles, FMOD lists just aggregate output since the outputs are managed by OS (*
    user is supposed to pick an output via control center on iOS : https://support.apple.com/en-us/HT202809#ios
    or system controls on android: https://www.businessinsider.com/how-to-change-audio-output-on-android

    there's no direct support for either one in the asset

    (* this works little differently still no iOS when recording is involved in that case there is support for selecting recording input for certain cases on iOS in the asset - see AudioStreamInput_iOSDemo
     
  10. r618

    r618

    Joined:
    Jan 19, 2009
    Posts:
    1,171
    a maintenance release submitted for review -
    ===============================================================================
    2.6.4 062022 Three days
    Updates/fixes:
    Added assembly definition files:
    - the asset's code is now entirely asmdef based /except Demos and its Plugins/
    Cleaned up usage of web proxy for networking requests:
    - web proxy usage is removed for all UnityWebRequest - all UWRs should follow system settings for web proxy
    - used now only by FMOD networking - legacy components
    - Note: FMOD networking support is not stable from version to version, right now it looks like it's not functional. All legacy/FMOD networking components most likely won't work, the original sources are being left in the asset for now.
    Updated demo app:
    - menu facelift - all scenes are grouped into logical categories by functionality
    Fixed AudioStreamStressTest demo scene: was using wrong/old component
    Removed & updated code:
    - removed old/unused/obsolete parts, moved parts common with AudioStreamIce asset into separate Scripts/Support assembly.
     
    Last edited: Jun 16, 2022
  11. r618

    r618

    Joined:
    Jan 19, 2009
    Posts:
    1,171
    you can upgrade from AudioStream to AudioStreamIce for half of its normal price now to get just radio/media streaming in it
    (just to clarify: you don't AudioStream for *Ice to work, they're entirely separated)
    TT~
     
  12. r618

    r618

    Joined:
    Jan 19, 2009
    Posts:
    1,171
    :: submitted 2.6.4.1 update ::
    Updates/fixes:
    - fixed iOS build
    - moved menu definition for demo into Resources to ensure it's always included in build
     
  13. shwa

    shwa

    Joined:
    Apr 9, 2012
    Posts:
    461
    Hi,
    This looks very interesting.
    I want to create a visualizer that responds to music that the user is listening to on their Apple laptop or Ipad or iphone.
    E.g. if they are listening to music via itunes, or apple streaming, or Spotify,
    will AudioStream be able to bring that audio into Unity,..
    where my app then creates visualizations on the fly, based on certain variables from the music, e.g. the beat?
    thanks,
     
  14. r618

    r618

    Joined:
    Jan 19, 2009
    Posts:
    1,171
    hi on macs you have to expose system capture device ( see soundflower / blackhole / loopback by rogueamoeba.. )
    it's not possible to capture system audio on mobiles
    hope this helps !
     
  15. NielsTerHeijden

    NielsTerHeijden

    Joined:
    Mar 4, 2014
    Posts:
    16
    Hello r618,

    I'm running into a weird problem using the native mixer plugin to route audio to two different audio output devices. Sometimes the output of one of the devices gets delayed/out of sync. I've not been able to find the root cause besides just a random amount of running time.

    The drift can be a fraction of a second but I've also seen 2 or more seconds delay between the two audio devices. Restarting the program is the only solution for the problem I've found. The device that isn't delayed keeps functioning as expected.

    Playing sounds in windows on the device that is at that time delayed in the unity program will just play those sound without delay. Pointing to a cause within the unity program. And because the other audio output device isn't delayed I'm suspecting fmod/AudioStream to be causing the problem.

    I'm unsure how to proceed to gather more information to pin point the exact cause. But with this limited amount do you know how to fix the problem or gather more information?
     
  16. r618

    r618

    Joined:
    Jan 19, 2009
    Posts:
    1,171
    Hello, do the devices' configs differ (in samplerate/channels) ? Is it possible to set them to be the same if they do ?
    One things which can cause the outputs to drift is when audio config changes at runtime ( by un/plugging a device /changing the setup ) and plugin needs to rebuild outputs, but it should be apparent only when devices' config differ from what I've seen.
    - just to be sure please mention the asset version and FMOD version it's using
    Thanks !
     
  17. NielsTerHeijden

    NielsTerHeijden

    Joined:
    Mar 4, 2014
    Posts:
    16
    I didn't plugin or plug out any devices. I just started the unity build changed to one specific scene and let it run for 15 minutes (or sometimes it requires 45 minutes) and the delay starts building up to a noticeable level. The longest delay I've seen and times was like 6 seconds.

    both are set to 24bit 48000Hz one is a USB headset that says it has 2 channels. The other is connected via the onboard audio card (realtek) the connected device is a headset so that should also be 2 channels. In this case it's the USB headset that get delayed.

    I've also tested it on a different system with a similar but not the same USB headset and a headset connected to the audio-out plug of the Oculus Rift S. Meaning the audio goes to the Rift S device and then get's outputted to the headphones. So also here it should be 2 channels? still 24 bit and 48000Hz. In this case the oculus audio get's delayed instead of the USB headset. Maybe the second known device to windows get's delayed or something.

    The version I've used is hard to find out. The audiostream dll is from around 18 november 2021. I've downloaded the latest version of AudioStream and copied both the fmod.dll and AudioPluginAudioStreamOutputDevice.dll from that package over the dll's in the program build. That's in the <project>_Data\Plugins\x86_64 folder next to the program.exe file. So it should use these latest dll's?? and the problem still occurs.
     
  18. r618

    r618

    Joined:
    Jan 19, 2009
    Posts:
    1,171
    thanks for the info, but If you're doing it this way I would recommend replacing also both fmodstudio.dll and resonanceaudio.dll since the effect is built against 2.02.x which is newer than from Nov 2021 I think - FMOD version from store should be ok
    EDIT: ok correction the effect has only fmod.dll as dependency so your method should work since Unity loads the plugin separately (not a fan of this though -)

    but the default(0) output is probably problematic indeed:
    - when I used outputs e.g. #1 and #3 the clips played in sync after running for more than 1/2 hour
    - with outputs #0 and #3 I got miniscule drift after cca 1/2 hour which would increase over time for sure

    Importantly though the short clip I used had to have Load Type Decompress On Load (and I had also Preload Audio Data on) | was set to loop | - without this Unity still didn't loop it in sync and I had small drift even with #1 and #3 after longer time

    So how are you using/importing your source audio clips ? Are they for longer background/ambient tracks, or are you invoking ~ Play/PlayOneShot as needed ?
    If the clips are imported with something else than Decompress On Load I think it might still cost some processing even when they're looped.

    But I have no good solution for the usage of the default output atm /
     
    Last edited: Aug 5, 2022 at 1:44 AM
  19. r618

    r618

    Joined:
    Jan 19, 2009
    Posts:
    1,171
    ( also I did only Editor runs, so this still might differ in build...; btw with Decompress On Load memory requirements change too so this has probably no good solution right now;- I did have a look at the effect but nothing jumped on me immediately wrt this situation :/ )
     
  20. NielsTerHeijden

    NielsTerHeijden

    Joined:
    Mar 4, 2014
    Posts:
    16
    Swapping the dll's isn't going to be the permanent solution it was just a quick way to test if it fixed the problem. #0 device is that always the windows default playback device? or just the first device windows found? There probably isn't a way to change this easily? Yesterday I had multiple sessions with 6 seconds drifts within 30 minutes. Maybe the problem increases when it's in build instead of editor?

    The sounds played are a mix of a looping ambient sound, two microphone input "streams" just using the unity soundclip system. Playing back live microphone input is weird in unity. And audio clips that where triggered on demand (played ones, not one shot) on different AudioSources in Unity. Those where constructed and deconstructed on a regular basis so most likely wouldn't have been alive for the whole 30 minutes.

    All audio files are loaded when unity starts and are not baked into the program. Using
    "UnityWebRequestMultimedia.GetAudioClip" and "DownloadHandlerAudioClip.GetContent" unsure if by default unity make a streaming or compressed AudioClip the documentation doesn't say. They are .wav and .ogg files.

    I Do wonder why the audio clip would influence the delay between the two output sources. Because one is just outputting the audio normally while the other is delayed. Is the audio data going through the AudioMixer into the submixers and your plugin, still linked to the audio clip source?

    Do you have any follow up steps we could do to get to the bottom of the problem? Isn't there somewhere a clear audio data buffer that could hold 6 seconds of audio data without a problem?
     
  21. r618

    r618

    Joined:
    Jan 19, 2009
    Posts:
    1,171
    hi, ok, I tested entirely different situation
    // that is:
    I played _single short audio clip_ on two separate _looped_ AudioSources each going into different mixer group (each having output sent to different output [0 and 3, resp. 1 and 3] ] by effect) on the same mixer. - so any difference between the two devices is immediately apparent when it happens -
    so no dynamically created AudioSources, hence any import settings don't play any role ( - Decompress On Load was relevant here only for looped clips to ensure that both AudioSources play just ready available data for any sync issues between the two to be audible )
    //

    - overall I recommend setting up an AudioSource completely (i.e. with Output mixer connection) and just changing AudioClip on it as needed;
    - though the mixer effect works autonomously - it just sends what's being provided by mixer on input to its output device
    It has its own ring buffer for each output device internally, see FMODOutputDevice.cpp in plugin source
    If it outputs silence for ~6 seconds it was probably being fed with silence on input, though I'm not sure how/why.
    I recommend using only single mixer group (across any/all mixers used in the application) for given desired output and use Output on all AudioSources which need to use it.

    Unity Microphone tends to lag over time too..
    If everything else fails I'd recommend running your application _without_ using the mixer effect, if possible
    - all this should require is removing Output connection on AudioSource/s in the scene -
    - I understand that audio will be overlapping but a test case should be recognizable and also that a build would be required, (possibly removing the mixer effect lib from project)...
     
  22. NielsTerHeijden

    NielsTerHeijden

    Joined:
    Mar 4, 2014
    Posts:
    16
    Sorry for the confusion it's hard to pin point what's important related information and what's just clutter.

    The audio sources are created on dynamically spawned characters in the scene. They will spawn and despawn depending if they walk out of the scene. Therefore it isn't possible to setup the AudioSource in editor. I do setup the AudioSource on the newly spawned character at runtime with a direct connection to a AudioSubMixer. There are also some sound effects that are played in random locations in the scene when triggered. Those use the same AudioSource that changes it's AudioClip and position.

    That ringbuffer sounds like the prime candidate to be the cause of the problem. For the microphone input I've also had to do something similar where you "create" a AudioClip with a short duration that then is filled with microphone data "
    UnityEngine.Microphone.Start". When playing that AudioClip I had to use
    "Microphone.GetPosition" to adjust the AudioSource.timeSamples periodically to reduce the increasing delay between microphone input and the audio source output.

    Is there a similar mechanism for the FMODOutputDevice ring buffer? where the rate the buffer is read from might slightly differ from the rate it's filled causing an increasing amount of delay between input and output? and can we influence that, or provide a more stable input or periodically force a synchronization?
     
  23. r618

    r618

    Joined:
    Jan 19, 2009
    Posts:
    1,171
    no, there's no such mechanism, the rate is constant across the whole Editor, FMOD resamples as needed per given output, but as you mentioned previously the rates are the same for all used outputs, yes ?
    One thing worth to check is System Sample Rate in Audio section of Player Settings (it's usually 0 which is current system default output rate)
    I'd have to set up a testing scene, but it'll take some time
     
  24. r618

    r618

    Joined:
    Jan 19, 2009
    Posts:
    1,171
    1 more thing meanwhile :can you eliminate microphone input and run without it ?
    though all inputs to the same mixer group _should_ be mixed together automatically ( so if a mic is outputting silence for some reason it shouldn't matter..)
    the buffer sync point is reset when mixer effect is muted - but that is according to Unity SDK valid only in Editor... -, and when engine is not playing, or is paused (whatever that means..)
     
  25. NielsTerHeijden

    NielsTerHeijden

    Joined:
    Mar 4, 2014
    Posts:
    16
    I lied, we didn't have 2 microphone inputs but 3. I never noticed the Unity Audio Settings options. I'm trying out those options in builds at the moment. Up to now it seems to introduce visual stuttering if you fixed it to 48000 HZ. I also noticed that one of our microphones only supports 44100HZ so I'm trying out all kinds of combinations of builds set to 48000HZ and 44100HZ and setting windows audio devices to those HZ's and DSP Buffer Sizes

    Eliminating microphone input devices is going to be difficult. I'm going to need to make a specialized test program to accomplish that. That will be the next step.
     
  26. r618

    r618

    Joined:
    Jan 19, 2009
    Posts:
    1,171
    FWIW you could use this mic, if applicable
    upload_2022-8-6_14-46-34.png
    or its 3D variant ( AudioStreamInput ), which latency is similar to Unity MIc..
     
  27. NielsTerHeijden

    NielsTerHeijden

    Joined:
    Mar 4, 2014
    Posts:
    16
    After trying loads of combinations with the Unity Settings on 44100HZ and 48000HZ and setting the windows devices on 44100HZ and 48000HZ. As much as the devices allowed. I'm stuck using one microphone that only supports 441000HZ and one microphone that only supports 48000HZ. And changing the windows settings on devices to allow exclusive control (mainly turning it off)

    I've found one combination, unity build with 48000HZ and all devices on 48000HZ (except one microphone) and exclusive control off on the audio output devices, that looked promising. Had two test runs of over an hour without any problems. Sadly the 3rd test run had the delay in 30 minutes. In the meantime nothing changed except for the program to being restarted in between.

    This kinda points to the issue not being hardware setup related but something is software going wrong. Is there a way to force a sample rate on the specific submixer output or other way to provide this information to AudioStream (or fmod) that might improve the functionality? Or is there a way to force a reinitialization without exiting the program?

    Is this error being caused in the AudioStream code or fmod?

    Are there certain thing we need to do to setup fmod correctly for our situation?
     
  28. r618

    r618

    Joined:
    Jan 19, 2009
    Posts:
    1,171
    before i'll send a test package i'd like to outline what it will do and would like a confirmation that it covers your usage:
    - it will have two mixer groups each with its own effect each going to different output device
    - a script which will randomly create a GO with AudioSource with random clip in the scene and sets its output to random mixer group each few minutes or so
    - two fixed mic inputs each going into different group
    - possibly one or more background AS in the scene constantly playing through one of the groups
    i recommend running this on the device then

    AFAIK btw it's preferable to leave exclusive control on for a device on windows (but not sure..)
     
  29. NielsTerHeijden

    NielsTerHeijden

    Joined:
    Mar 4, 2014
    Posts:
    16
    Disabling exclusive control didn't solve the problem so will turn them back on. At some point you just need to try everything.

    Your description seems to be correct, maybe for some added context we have two people talking into a microphone but only hearing the other in the virtual environment. That's why we wanted to separate the output to specific devices. All the other sound are the normal simulation sounds like background noises, music and audio effects. The third microphone is a virtual microphone from a voice transformer one of the persons can switch back and forth too. But even when it isn't used the delay happens.

    Maybe as some additional information, now that I've observed the problem more closely, the delay happens pretty rapidly in like minutes it goes from no delay to very noticeable seconds long delay. Before that happens the program could have run anywhere from minutes to an hour with out problem. So it's not slowly building up.
     
unityunity