Search Unity

  1. Are you interested in providing feedback directly to Unity teams? Sign up to become a member of Unity Pulse, our new product feedback and research community.
    Dismiss Notice

AudioStream - an audio streaming solution for all and everywhere

Discussion in 'Assets and Asset Store' started by r618, Jun 19, 2016.

  1. r618

    r618

    Joined:
    Jan 19, 2009
    Posts:
    1,077
    hi, it should if Unity + FMOD Integration works correctly on Linux - component for this for game objects is just C# scripts
    (optional native mixer plugin is not available on Linux)
    there will be some latency, I recommend running demo app e.g. on windows to get a feel how it behaves
    -- I did a linux test initially but it was few years ago now so don't have up to date info though
     
  2. r618

    r618

    Joined:
    Jan 19, 2009
    Posts:
    1,077
    hi short version is that i don't recommend doing this with Unity - you might get like 80-90% there but the end result will still not feel right on the platform
    -- there is some buffering support, downloading via UnityWebRequest and it - mostly - can extract album artwork from files, but the rest like variable playback speed and captioning is not there;

    i'm not sure how captioning works exactly - it should be doable if it's timestamped extra (meta)data present in the file/stream - if it's just tags their changes should be emitted w/ UnityEvent
    - variable playback speed can be approached (not with stellar results) just by chaining the pitch, something like throwing away samples is more invasive and would require probably non trivial changes
    there are more sophisticated ways of doing it - like e.g. https://www.surina.net/soundtouch/ - but this is not included
     
  3. frankcarey

    frankcarey

    Joined:
    Jul 15, 2014
    Posts:
    10
    Hi, I'm having a weird issue with AudioSourceOutputDevice where I have a custom script that provides a menu to select the audio output from a list. It works fine for a few minutes, but after about 1-2 minutes of triggering various short audio files to play there is always a point where the secondary audio output just stops working. The component values do not change as far as I can tell, and I've added callbacks to see if OnRedirectStopped() or OnError() are happening. Neither seem to be. The kicker is if I go back into the selector menu and click away from the output I want and then back to it, it works again for a little while. Any suggestions where I can put a breakpoint to help debug this?

    Edit: Also I've set the LogLevel to DEBUG and don't see any messages come through when it fails.

    Edit2: Note that I have the AudioSourceOutputDevice set on the main Camera.
     
  4. r618

    r618

    Joined:
    Jan 19, 2009
    Posts:
    1,077
    hi!
    can you verify what DSP buffer size is set to in Audio settings ? Try to ease to something else if it's on Best latency
    can you PM me the project/link - or relevant part - otherwise if possible ? I'll have a look (it's nearly impossible to tell where/what to debug, fmod just silently failed on me too at times)
    Thanks!
     
  5. frankcarey

    frankcarey

    Joined:
    Jul 15, 2014
    Posts:
    10
    Yes, that seems exactly what's happening. I had turned the latency setting down to 1, but after setting it to like 40, the issue seems to have stopped so far. Were you able to find a way to be notified from FMOD of it failing eventually or do you have a ticket with FMOD I could track?
     
  6. r618

    r618

    Joined:
    Jan 19, 2009
    Posts:
    1,077
    not really, just made sure the code behaves as conservatively as possible; these started occurring when changing mainly DSP buffers to something else than what FMOD considers as default for given output/device
    you can turn on some FMOD builtin diagnostics by uncommenting FMODSystemsManager.InitializeFMODDiagnostics in code (you might need debug libraries - the ones ending with 'L' for all traces to work I think - not sure if they're loaded in the editor by default though, haven't checked this with recent FMOD versions)

    you mean the latency slider on the component - couldn't quite place it at first!..
    yes, 1 is quite not a good idea tbh - i should probably restrict this somehow since it's virtually impossible to determine lower bound automatically I think

    in any case my best advice is to stick w/ value which works now; I'd be careful with anything much below 50ms though which should be ok for most consumer stereo setups
     
  7. AaronMahler

    AaronMahler

    Joined:
    May 19, 2021
    Posts:
    8
    Do you still need a lot of audio sources? I think you need podcast this time which I recently subscribed https://www.podcasthowto.com/best-dollop-episodes/ You can try for it. I hope you will enjoy.
     
  8. hadrien_n

    hadrien_n

    Joined:
    Oct 9, 2019
    Posts:
    8
    Hi, I'm using AudioStream to stream the microphone and scene audio to a web server.
    The mic is captured with AudioStreamInput2D, muted locally with an AudioSourceMute component so it can't be heard locally. The mic signal is then merged to the scene audio and the resulting signal is streamed to the server.

    The code to merge the mic looks something like this.
    Code (CSharp):
    1.  
    2. private void OnAudioFilterRead(float[] data, int channels)
    3. {
    4.     [...]
    5.     var signals = data.ToArray();
    6.     var length = signals.Length < mic.captureBuffer.Length
    7.                         ? signals.Length
    8.                         : mic.captureBuffer.Length;                      
    9.     for (int i = 0; i < length; i++)
    10.     {
    11.          signals[i] += mic.captureBuffer[i];
    12.     }
    13.     [...]
    14. }
    15.  
    This works well on Windows, but on Oculus Quest the mic sound is very choppy.

    I did a test to unmute the mic, so the mic can be heard in Unity player, and disabled the signal merging to isolate the problem: the mic is still choppy on Quest, unless I set useUnityToResampleAndMapChannels to true.

    So the mic is much better on Quest with that settings set at true, but now the problem is that when I switch back to my original setup for streaming the audio (mute the mic locally and merge the signal), the microphone pitch is much higher.

    I tried disabling useAutomaticDSPBufferSize too but the sound is choppy as well.

    Not sure what to do from here... Is there a way to correct the pitch when setting useUnityToResampleAndMapChannels to true? Maybe something to do with the way the mic signal is merged to the scene audio. Also, do you see a different approach to achieve my goal, other than muting the mic locally and merging the signal? For instance, I tried using Unity Audio Mixer groups but I don't see an API to retrieve the audio of a specific group.

    Any help is appreciated, thanks!
     
  9. r618

    r618

    Joined:
    Jan 19, 2009
    Posts:
    1,077
    hey @hadrien_n , your approach is OK! - only I would avoid boxing in the OnAudioFilterRead callback (data is already an array - no need to recreate it)
    look at 'Resample Input' setting on the component - that changes the pitch and it's probably not behaving correctly here / i'm not sure how many input/output channels there are on oculus device /
    - it was added for some more advanced interop with different packages but when off the mic signal should be processed unchanged only by audio source
    if that fails please set log level to INFO on the component and send me the log when running on device
    Thanks !

    and sorry about choppy sound when using the Speex resampler (with useUnityToResampleAndMapChannels off) - it doesn't really handle more than 2 channels properly
     
  10. r618

    r618

    Joined:
    Jan 19, 2009
    Posts:
    1,077
    you'd need a separate audio mixer plugin to do anything which touches actual signal there
     
  11. hadrien_n

    hadrien_n

    Joined:
    Oct 9, 2019
    Posts:
    8
    Hey thanks for the quick reply. I realized after your response that I didn't have the latest version - didn't see the ResampleInput setting on my side. So after updating and fixing the components setup (needed to add an AudioSourceCaptureBuffer in addition to the AudioStreamInput2D component, in order to access the captureBuffer property)... it's now much better! No more choppy sound.

    I'm building with 'Automatic DSP Buffer size', 'Resample Input' and 'Use Unity To Resample And Map Channels' all set to true.

    I find the mic still a bit saturated so I'm going to look into that next, but wow that's a relief! Maybe adjusting the gain can help reduce the saturation a bit, I'll see.

    Regarding copying the float[] data array in the code above (var signals = data.ToArray(), I'm doing this because I don't want to add the mic signal in the data array directly, otherwise we would end up hearing the mic sound in the player. But there is probably a better way to do this, maybe by reusing the same array instance instead of creating a new one each time.

    Anyway, thanks for the help. Cheers!
     
  12. r618

    r618

    Joined:
    Jan 19, 2009
    Posts:
    1,077
    that checks out!
    (i'm not sure right now but it's possible something was fixed meanwhile too)

    you can attach normal unity monobehaviour audio effects components on the game object btw - just make sure to get the result only in the last step - so with AudioSourceCaptureBuffer at the bottom in the inspector - and it should work

    makes sense then!

    good to know it's working
    Best!
     
  13. ROBYER1

    ROBYER1

    Joined:
    Oct 9, 2015
    Posts:
    1,195
    Is it possible for me to use this plugin to mix game and microphone audio/sound together and redirect that to a plugin like Natcorder's recording to save that audio with a screen recording?
     
  14. ROBYER1

    ROBYER1

    Joined:
    Oct 9, 2015
    Posts:
    1,195
    Trying to do this also to get game and mic sound combined in a recording within a video call app - did you find a solution to this? @rickomax
     
  15. r618

    r618

    Joined:
    Jan 19, 2009
    Posts:
    1,077
    you can use all Unity audio callbacks normally, so in this case this means you can mix mic input with an AudioSource/Listener in OnAudioFilterRead for example - see e.g. above how hadrien_n did it
    you need to script this though, there's nothing automatic about this

    as for Natcoder they would have to support passing along custom PCM data - or in other words properly plug into Unity audio system - which I have no idea and experience with unfortunately
     
    ROBYER1 likes this.
  16. ROBYER1

    ROBYER1

    Joined:
    Oct 9, 2015
    Posts:
    1,195
    Hi @hadrien_n , I am trying to do something similar, did you have a full script implementation of this you would be able to share if possible please?
     
  17. Karsten

    Karsten

    Joined:
    Apr 8, 2012
    Posts:
    136
    @r618
    Hi
    would I be able to stream an 8 Channel audioclip (max in Unity afaik) out of the box?
    I imagine how to do this technically (ofc bandwith must be available) but i ask if this is already possible so I know beforehand if I have to develop such on my own after buying the asset?
     
  18. r618

    r618

    Joined:
    Jan 19, 2009
    Posts:
    1,077
    Hi, you'd have to unfortunately !
    It uses OPUS codec from here : https://github.com/lostromb/concentus : currently to un/pack audio for smaller footprint over network but that is limited to max. 2 channels (and selected samplerates); though it can push uncompressed PCM data to Icecast, too.
    PCM can be scripted (you'd just skip en/decoding steps), but that would be quite heavy for bandwidth for sure (not sure if usable at all..)
    I'll have a look if it's reasonably doable via FMOD codec support, and let you possibly know; (no promises though, am not even 1oo% sure if it's viable right now)
     
  19. florentRaffray

    florentRaffray

    Joined:
    Feb 10, 2021
    Posts:
    7
    Hi I'm wondering if this could asset could help in a specific outcome I would like. I've been exploring audio reactive pieces in unity and it's been a lot of fun: https://www.instagram.com/lost_atoms/
    I would like to extend this concept to hopefully be able to visualize streaming music from apple music and/or soundcloud in an ios app. Could this tool get streaming data from those services and pass it through the unity function GetSpectrumData() or some other similar fft based audio analysis?
     
  20. r618

    r618

    Joined:
    Jan 19, 2009
    Posts:
    1,077
    hi iOS is poorly supported with UnityWebRequest ( which you would use to reach any internet service ), but Soundcloud is reachable via old AudioStreamMinimal I think since that always worked for some reason despite the fact that *Minimal doesn't handle HTTPS properly... )
    All components plug seamlessly into Unity audio source (so you can use their methods), but Apple Music can't be reached via normal HTTPS requests directly anyway (for obvious reasons)

    So overall - secure https requests (with the exception of Soundcloud possibly..) don't work on iOS (only plain htttp/minimal components) and you'd need proper iOS specific implementation to access apple music (and i'm not sure about access to actual media PCM content which you would need for anything unity audio source related) - which the asset unfortunately doesn't provide
     
  21. whiteaangel208

    whiteaangel208

    Joined:
    Jun 14, 2018
    Posts:
    2
    Greetings. I got an asset for playing sound from the server. In the description I saw information that when using the AudioStreamMinimal component, the sound will play even with the screen off, but when I turn off the screen, the sound loops at one point. I thought that I was doing something wrong and downloaded the demo build to test it, but it’s the same, when the screen turns off, the sound loops. Tell me what could be the problem? Android 10.
     
  22. r618

    r618

    Joined:
    Jan 19, 2009
    Posts:
    1,077
    hi, appreciate you tested the demo too!
    but the bad news is the situation on android is rather.. random to say the least
    i currently don't have means to test on 10 and the only correct way to implement this is indeed via service only
    / if you make this work by modifying activity lifetime via gradle build lmk but this way is very intrusive and should not be recommended anywhere, it's the easiest solution unfortunately /
     
  23. whiteaangel208

    whiteaangel208

    Joined:
    Jun 14, 2018
    Posts:
    2
    Thanks for the answer. I'll try to figure it out.
     
unityunity