AppendStructuredBuffer slow initial performance

I have a custom raytracer implemented as a compute shader. Inspired by an article series I wanted to try to use a ray scheduler system. The idea is to have an AppendStructuredBuffer<Ray> and split the ray generation and tracing into two kernels. This would allow scheduling fewer rays at the peripheral vision of a VR user and more rays at the center of the screen.

However, after implementing this separation of ray scheduling and ray tracing, there is a noticeable drop in performance immediately after starting the scene. Initially, I thought that using an AppendStructuredBuffer was causing the slow down as it required some synchronization between the threads. Interestingly, after about 5-10 seconds, the performance significantly improves and returns to similar levels as the old implementation.

After some experimenting, I think the slow down is caused by AppendStructuredBuffer.Append() function. If I remove that line (but otherwise do the same amount of work), there is no slow down on start.

I'm a bit stumped why the performance would start super slow and then without any change in workload, magically and consistently improve after about 10 seconds.



The blue part here is almost exclusively the compute shader. It goes from an initial execution time of around 180-200 ms to 1-2 ms.