Bug Recorder FFMPEG encoder will stop working if there is any kind of FPS hiccup, rendering it dead

We started using the FFMPEG encoder as the default Unity mp4 encoder is really not good - It creates really bad artifacts in motion and is not suitable for trailer capture work.
We've been desperately trying to track down a bug with the FFMPEG encoder sample where at some random point it will start taking 2000ms+ per frame and essentially not working. We've tried all sorts of ffmpeg encoder options to try to make sure it isn't stalling the frame pipe. Encoding options like the below should be fine:
-c:v libx264 -pix_fmt yuv420p -preset fast -tune zerolatency -crf 18

Whenever there is a decent-sized hiccup in FPS it appears the pipeline falls apart. We can repro this by running the recorder and hitting pause and unpause in the unity editor for a second or so. I enabled debug logs and this is what happens:

Spoiler: Logs

FFmpegPipe : VideoFrame: 8176
UnityEditor.Recorder.MovieRecorder:WriteCPUFrame (UnityEngine.Rendering.AsyncGPUReadbackRequest) (at Library/PackageCache/com.unity.recorder@4.0.0-pre.5/Editor/Sources/Recorders/MovieRecorder/MovieRecorder.cs:193)

FFmpegPipe : VideoFrame: 8177
UnityEditor.Recorder.MovieRecorder:WriteCPUFrame (UnityEngine.Rendering.AsyncGPUReadbackRequest) (at Library/PackageCache/com.unity.recorder@4.0.0-pre.5/Editor/Sources/Recorders/MovieRecorder/MovieRecorder.cs:193)

FFmpegPipe : VideoFrame: 8178
UnityEditor.Recorder.MovieRecorder:WriteCPUFrame (UnityEngine.Rendering.AsyncGPUReadbackRequest) (at Library/PackageCache/com.unity.recorder@4.0.0-pre.5/Editor/Sources/Recorders/MovieRecorder/MovieRecorder.cs:193)

FFmpegPipe : CopyThread timeout for ffmpeg pipe of file VideoPipe_copyQueue.Count = 0
System.Threading.ThreadHelper:ThreadStart ()

FFmpegPipe : PipeThread for ffmpeg pipe of file VideoPipe_pipeQueue.Count = 0
System.Threading.ThreadHelper:ThreadStart ()

FFmpegPipe : VideoFrame: 8179
UnityEditor.Recorder.MovieRecorder:WriteCPUFrame (UnityEngine.Rendering.AsyncGPUReadbackRequest) (at Library/PackageCache/com.unity.recorder@4.0.0-pre.5/Editor/Sources/Recorders/MovieRecorder/MovieRecorder.cs:193)

FFmpegPipe : SyncFrameData timeout for ffmpeg pipe of VideoPipe_copyQueue.Count = 1
UnityEditor.Recorder.MovieRecorder:WriteCPUFrame (UnityEngine.Rendering.AsyncGPUReadbackRequest) (at Library/PackageCache/com.unity.recorder@4.0.0-pre.5/Editor/Sources/Recorders/MovieRecorder/MovieRecorder.cs:193)

FFmpegPipe : VideoFrame: 8180
UnityEditor.Recorder.MovieRecorder:WriteCPUFrame (UnityEngine.Rendering.AsyncGPUReadbackRequest) (at Library/PackageCache/com.unity.recorder@4.0.0-pre.5/Editor/Sources/Recorders/MovieRecorder/MovieRecorder.cs:193)

FFmpegPipe : SyncFrameData timeout for ffmpeg pipe of VideoPipe_copyQueue.Count = 2
UnityEditor.Recorder.MovieRecorder:WriteCPUFrame (UnityEngine.Rendering.AsyncGPUReadbackRequest) (at Library/PackageCache/com.unity.recorder@4.0.0-pre.5/Editor/Sources/Recorders/MovieRecorder/MovieRecorder.cs:193)

FFmpegPipe : VideoFrame: 8181
UnityEditor.Recorder.MovieRecorder:WriteCPUFrame (UnityEngine.Rendering.AsyncGPUReadbackRequest) (at Library/PackageCache/com.unity.recorder@4.0.0-pre.5/Editor/Sources/Recorders/MovieRecorder/MovieRecorder.cs:193)

FFmpegPipe : SyncFrameData timeout for ffmpeg pipe of VideoPipe_copyQueue.Count = 3
UnityEditor.Recorder.MovieRecorder:WriteCPUFrame (UnityEngine.Rendering.AsyncGPUReadbackRequest) (at Library/PackageCache/com.unity.recorder@4.0.0-pre.5/Editor/Sources/Recorders/MovieRecorder/MovieRecorder.cs:193)

FFmpegPipe : VideoFrame: 8182
UnityEditor.Recorder.MovieRecorder:WriteCPUFrame (UnityEngine.Rendering.AsyncGPUReadbackRequest) (at Library/PackageCache/com.unity.recorder@4.0.0-pre.5/Editor/Sources/Recorders/MovieRecorder/MovieRecorder.cs:193)

FFmpegPipe : SyncFrameData timeout for ffmpeg pipe of VideoPipe_copyQueue.Count = 4
UnityEditor.Recorder.MovieRecorder:WriteCPUFrame (UnityEngine.Rendering.AsyncGPUReadbackRequest) (at Library/PackageCache/com.unity.recorder@4.0.0-pre.5/Editor/Sources/Recorders/MovieRecorder/MovieRecorder.cs:193)

FFmpegPipe : VideoFrame: 8183
UnityEditor.Recorder.MovieRecorder:WriteCPUFrame (UnityEngine.Rendering.AsyncGPUReadbackRequest) (at Library/PackageCache/com.unity.recorder@4.0.0-pre.5/Editor/Sources/Recorders/MovieRecorder/MovieRecorder.cs:193)

FFmpegPipe : SyncFrameData timeout for ffmpeg pipe of VideoPipe_copyQueue.Count = 5
UnityEditor.Recorder.MovieRecorder:WriteCPUFrame (UnityEngine.Rendering.AsyncGPUReadbackRequest) (at Library/PackageCache/com.unity.recorder@4.0.0-pre.5/Editor/Sources/Recorders/MovieRecorder/MovieRecorder.cs:193)

Here is our Recorder setup:


These two log entries seem weird to me given the code:
FFmpegPipe : CopyThread timeout for ffmpeg pipe of file VideoPipe_copyQueue.Count = 0
FFmpegPipe : PipeThread for ffmpeg pipe of file VideoPipe_pipeQueue.Count = 0

Code:

Code (CSharp):

1. if (nbTries > _maxRetries)
2.                     {
3.                         Log("CopyThread timeout for ffmpeg pipe of file " +
4.                             _name + "_copyQueue.Count = " + _copyQueue.Count);
5.                         _terminate = true;
6.                         return;
7.                     }

Shouldn't this terminate the whole recording session? That doesn't happen right now.

 

We're on 2022.1.24f1 and on the latest available recorder version via package manager.