Search Unity

  1. Unity Asset Manager is now available in public beta. Try it out now and join the conversation here in the forums.
    Dismiss Notice
  2. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  3. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Accessing received WebRTC AudioStreamTrack audio data

Discussion in 'Unity Render Streaming' started by Javi-K, Jul 5, 2022.

  1. Javi-K

    Javi-K

    Joined:
    Mar 9, 2022
    Posts:
    6
    Hello,

    I'm using WebRTC 2.4.0-exp.5 inb my project, and I'm trying to use audio received from WebRTC and use it as the input for OVRLipSync.

    My code is similar to the one in the reference here https://docs.unity3d.com/Packages/com.unity.webrtc@2.4/manual/audiostreaming.html#receiving-audio, and I can receive and listen to the audio, but OVRLipSync doesn't seem to be able to use it when setting the same AudioSource as input for it.

    I wanted to try to use the PCM/float data from WebRTC's AudioStreamTrack instead, but it seems this isn't possible. Could anyone tell me if there is a way to do this currently?
    I also tried using Unity's OnAudioFilterRead, but it doesn't seem to get called.

    Thank you,

    Javier
     
  2. kazuki_unity729

    kazuki_unity729

    Unity Technologies

    Joined:
    Aug 2, 2018
    Posts:
    803
    k-i-w-i likes this.
  3. Javi-K

    Javi-K

    Joined:
    Mar 9, 2022
    Posts:
    6
    Thank you for the incredibly fast response.

    I did more testing with OnAudioFilterRead after reading your message, but when OnAudioFilterRead is called, data is always all 0.

    I am setting the AudioSource.loop to true before playing it just like in the sample (and also in the editor):

    Code (CSharp):
    1.      
    2.         receivedAudioSource.loop = true;
    3.         receivedAudioSource.Play();
    4.  
    I changed the order of the AudioSource and the script that includes OnAudioFilterRead in the GameObject, but it didn't change the result. If you have any idea what I might be doing wrong, it would be really helpful.

    Thank you

    Edit: just in case I wanted to add that even though I am receiveing and listening to the audio, the AudioSource has no AudioClip set. I think that's probably how the WebRTC library works though.
     
    Last edited: Jul 5, 2022
  4. Javi-K

    Javi-K

    Joined:
    Mar 9, 2022
    Posts:
    6
    I think I managed to fix my problem. I found that when I placed the WebRTC plugin's AudioCustomFilter script component in my GameObject, my other script's OnAudioFilterRead started working. I don't know why this was needed, since it doesn't seem to be mentioned in the documentation and I found it by luck after looking into AudioStreamTrack's code and the scripts included in the plugin.
     
    ysleeIaan likes this.
  5. kazuki_unity729

    kazuki_unity729

    Unity Technologies

    Joined:
    Aug 2, 2018
    Posts:
    803
    Mmm, It sounds a bug. WebRTC should make AudioCustomFilter and attach the component to the gameObject.
    How can I reproduce the issue?
     
  6. Javi-K

    Javi-K

    Joined:
    Mar 9, 2022
    Posts:
    6
    Here is a screenshot of the GameObject. VRAvatarWebrtcSynchronizer is receiving data from a DataChannel and also from an AudioStreamTrack.

    The AudioStreamTrack is set to the AudioSource like this inside MediaStream.OnAddTrack:

    Code (CSharp):
    1. _audioSource.SetTrack(audioStreamTrack);
    2. _audioSource.loop = true;
    3. _audioSource.Play();
    I tried placing OnAudioFilterRead inside VRAvatarWebrtcSynchronizer or LipSyncHandler, but the data array was always all 0, even though the audio was being received from WebRTC.

    After adding AudioCustomFilter like in the screenshot, OnAudioFilterRead data was fixed and I am able to use the data in OVRLipSync.
     

    Attached Files:

    Last edited: Jul 7, 2022
  7. kazuki_unity729

    kazuki_unity729

    Unity Technologies

    Joined:
    Aug 2, 2018
    Posts:
    803
    Thank you for sharing the detail.
    I assume that the order of components on the inspector window would occur the issue.