Search Unity

  1. Welcome to the Unity Forums! Please take the time to read our Code of Conduct to familiarize yourself with the forum rules and how to post constructively.
  2. The 2023.1 beta is now available for testing. To find out what's new, have a look at our 2023.1 feature highlights.
    Dismiss Notice

Problem restarting audio stream

Discussion in 'Unity Render Streaming' started by mgstauff, May 31, 2022.

  1. mgstauff

    mgstauff

    Joined:
    Sep 6, 2017
    Posts:
    59
    URS 3.1.0-exp.3
    Hi,

    I'm modifying the URS Example scene WebBrowserInput for my needs - one difference though is that I'm using SingleConnection instead of Broadcast. I've got things working well overall. But am having a problem with audio only working during the first connection between Sender and Receiver.

    I'm setting things up so the user can use the Receiver (web app) to disconnect, either deliberately via a UI button or by refreshing the browser, and the Sender (unity app) will detect it, delete the SingleConnection connection, and then initiate it again so that the Receiver can reconnect.

    On the Receiver side, the disconnection is handled via:
    Code (CSharp):
    1. //Disconnect peer connection and signaling
    2.   async hangUp(connectionId) {
    3.     if (this.signaling == null) {
    4.       return;
    5.     }
    6.  
    7.     this.pc.close();
    8.     this.pc = null;
    9.     Logger.log(`delete connection ${connectionId}`);
    10.     await this.signaling.deleteConnection(connectionId);
    11.     await this.stop();
    12.   }
    13.  
    14.   async stop() {
    15.     if (this.signaling) {
    16.       await this.signaling.stop();
    17.       this.signaling = null;
    18.     }
    19.   }
    On the Sender side, the disconnection is done via SingleConnection:DeleteConnection() (either via UI event, or a data channel close event is received). I'm letting the RenderSteaming component keep running (I tried stop/starting that too but that borked everything).

    I'm using AudioStreamSender (in case there's a different way I don't know about), and it works when I first run the Sender (from Unity Editor) and connect the Receiver. But if I disconnect, either from Sender or Receiver, and then reconnect w/out stopping and starting the unity editor, the audio doesn't work (the video and data channel streams do work).

    I've looked at what I can in the code and from what I can see it looks like things are still connected even when the audio isn't playing in the Receiver:

    On Sender side:
    - audioStreamSender.Track.ReadyState is 'Live'
    - audioStreamSender.Track.Enabled is 'true'
    - the AudioStreamSender componenent shows the sound meter UI in the Inspector, and it moves up/down like it does when the audio does play properly

    On Receiver side (in video-player.js):
    - during connection setup, the Peer.trackevent event is received and a track of type 'audio' is added.
    - after 'audio' track is added and I interrupt the code, it has property values 'enabled: true', 'muted: false', and 'readyState: live'

    Does anyone have any suggestions as to how to keep debugging this??
     
  2. kazuki_unity729

    kazuki_unity729

    Unity Technologies

    Joined:
    Aug 2, 2018
    Posts:
    581
    I understood your issue.
    This issue would be caused by the implementation of reconnection peers. Sounds like It is not only audio.
     
  3. mgstauff

    mgstauff

    Joined:
    Sep 6, 2017
    Posts:
    59
    Thanks for the reply. It is only audio. The video and data channels work properly when I reconnect in the way I describe above, but the audio does not. The audio only works the first time I connect after restarting my app in unity editor.

    Also is there documentation somewhere that describes the nature of the various connections in more detail than the overviews that are already given?
     
  4. kazuki_unity729

    kazuki_unity729

    Unity Technologies

    Joined:
    Aug 2, 2018
    Posts:
    581
    This issue would be fixed in this PR. This fix has been merged the develop branch, but have not been released yet, but you can test it only you check it out from the repository.
    https://github.com/Unity-Technologi...mits/3f6735d4b4dfa6b0532886d75eb95b9a0b5754d0
     
  5. Handsome-Wisely

    Handsome-Wisely

    Joined:
    Mar 20, 2013
    Posts:
    98
    my demo can not send audio to web, may web not play audio. do you have this problem.
     
  6. mgstauff

    mgstauff

    Joined:
    Sep 6, 2017
    Posts:
    59
    Thanks for this! I'm just getting back to this project now. I'm looking at the fix you've linked here - is this the only relevant part of the PR for this issue I described with the audio? This fix changes only the ReceiverSample.cs file. My receiver though is derived from the video-player.js file in WebApp videoplayer code: https://github.com/Unity-Technologi.../client/public/videoplayer/js/video-player.js

    The code that sets up the audio stream does it in reoponse to a callback from the RTCPeerConnection instance that's contained within class Peer:

    Code (CSharp):
    1.    
    2. this.pc = new Peer(this.connectionId, true);
    3.     this.pc.addEventListener('disconnect', () => {
    4.       Logger.log('video-player pc disconnect handler');
    5.       _this.mainProcessPeerDisconnect();
    6.     });
    7.     this.pc.addEventListener('trackevent', (e) => {
    8.       const data = e.detail;
    9.       Logger.log('* * * video-player trackevent received: ' + e, true);
    10.       if (data.track.kind == 'video') {
    11.         _this.videoTrackList.push(data.track);
    12.       }
    13.       if (data.track.kind == 'audio') {
    14.         Logger.log("--- setupConnection: trackevent adding type 'audio'", true);
    15.         _this.localStream.addTrack(data.track);
    16.         let audioTracks = _this.localStream.getAudioTracks();
    17.         Logger.log("--- setupConnection: trackevent adding type 'audio'. audioTracks length: " + audioTracks.length);
    18.       }
    19.       // NOTE this gets called cuz maxVideoTrackLength == 1
    20.       if (_this.videoTrackList.length == _this.maxVideoTrackLength) {
    21.         _this.switchVideo(_this.videoTrackIndex);
    22.       }
    23.     });
    24.  
    25. <snip>
    Do you have any suggestions regarding this and getting the audio to reconnect properly?
    Thanks
     
  7. kazuki_unity729

    kazuki_unity729

    Unity Technologies

    Joined:
    Aug 2, 2018
    Posts:
    581
    I am sorry but it is difficult to say something to fix the issue with information. I assume that the signaling process of web-client side has bug. Can you share the source code to reproduce the issue?
     
  8. mgstauff

    mgstauff

    Joined:
    Sep 6, 2017
    Posts:
    59
    I'm modding the 'videoplayer' sub-project from the URS WebApp code: https://github.com/Unity-Technologi...tree/develop/WebApp/client/public/videoplayer

    But I realize now I haven't tested it with the original code, only with my modded code. So if it can't be debugged from the original WebApp/videoplayer code, it'll be an issue with my changes. If you don't have someone who can test the original WebApp/videoplayer code, I can do that but not until next week at the earliest.

    Thanks again
     
    kazuki_unity729 likes this.