Optimizing terrain collider physics updates

Hello, I'm currenly working on a project which deforms millions of heightmaps each frame and am stumbling to find a solution to update collisions for most of those heightmaps. The best option thus far by far is to simply use GetHeights() to update the terrain collider, however, it's not multithreaded and very slow. For 1089 heightmaps, there's a penalty cost of 0.1-0.2ms on my i7-5820k.

Will there be performance improvements for adjusting the terrain collider during runtime in the future?

An alternative is to use meshes, however, they are a lot worse for realtime physics updates and according to this thread, no performance boost is coming anytime soon since the mesh is using Nvidia libraries for rebaking collision data.

The last solution is to create my own physics libraries, which i've been avoiding the most, though perhaps in the end would turn out to be the best option, giving me the greatest amount of control. The problem is that it could take me months to actually synchronize the things i require adequately.

My current setup is to have 1 single terrain for rendering and subdivide it into a grid of 33x33 heightmap terrains for collisions. The total amount of terrains I can synchronize without incurring heavy performance penalties is about 20, which is about 21780 heightmaps. Depending on how large the terrain is, this leaves me with 70-95% of terrain unsynchronized for physics.

The system in action:

Does anyone have any examples of projects/games where a dynamic system was used for heightmap terrain manipulation? The only examples i've been able to come up with are "Sacrifice", "Giants: Citizen Kabuto" and various "Worms" games, neither of which actually do a lot of dynamic realtime heightmap manipulation. And out of the effects which do manipulate the terrain, they rarely need to run milllions of operations across the whole terrain each frame. Having something as base reference would be quite helpful.