ComputeShader in VR is very slow

Hi, I wonder if anyone else is experiencing slow Compute Shaders in Unity 2019.3/4 with pc-connected headsets (oculus link, rift, etc.).

On desktop, my compute shader goes at around 150FPS, while in VR it goes down to 40fps.
Nothing changes if I disable the shader rendering these buffers, the bottleneck is definitely not on the shader.

I do not use GetData, it's all indirect dispatches.