Search Unity

Question How to fix drifting audio from Unity asset + ffmpeg?

Discussion in 'Audio & Video' started by ECHO-425, Mar 11, 2023.

  1. ECHO-425

    ECHO-425

    Joined:
    Feb 2, 2018
    Posts:
    19
    We have been trying to figure out this problem internally, but are still learning up on ffmpeg! I'm hoping that someone can help point us in the right direction(s).

    We have a Unity game build that captures a camera's audio + video and streams it out to YouTube or Twitch. We use the Headless Movie Capture asset (https://assetstore.unity.com/packages/tools/video/headless-movie-capture-140849) which works well by capturing the camera frames + listener audio and pushing them to an rtmp endpoint via ffmpeg, but the audio is not in sync and continues to drift further away. It seems to be a fixed framerate or timestamp issue because the audio comes after the intended video, plus the video appears sped up, freezing on the stream to "catch up." Finally, the distance between the video and audio recording time/timestamps does get larger as the test time increases.

    I've tried several ffmpeg sync settings (vsync & async -1/1/2) which can be applied with the asset after its own applied ffmpeg settings (which are sent together to start up ffmpeg streaming), but they did not seem to make a noticeable difference in the audio drift. I did notice that the -framerate 60 parameter is always passed in by the asset, which is leading to a difference between the game's framerate and that fixed output framerate. The asset itself is logging that this framerate discrepancy may cause the "sped up" video like we see, though I am still determining how I might be able to stop the asset from adding this -framerate parameter to make it a dynamic framerate...

    This is all to say that we're still combing through the asset's code to determine how it works and if the issue is on it/Unity's side. Maybe timestamps are not being properly assigned to each camera or audio frame before it is passed to ffmpeg, etc... We're not sure where we should focus our efforts!

    Any help or insights would be much appreciated ~