Feedback Fast terrain collision checking

Hello, I'm in the process of making a runtime terraformation system for unity's terrain and have found myself struggling to come up with a way to have performant manipulation of the heightmap when it comes to collision detection. The system runs fine visually, it only impacts about 0.5ms on terrain with 250k heightmap manipulation, however, it no longer does so if the collider is synced every frame, which is what I'm aiming to do besides the visuals. It uses advantages provided by terrain's "Draw Instanced" option which allows rendering the terrain through a RenderTexture.

These are the things I've tried:
-Using unity's API to sync colliders. Massive lag spikes, it does not support multithreading or Async and even a small amount of heightmap manipulation leads to a complete recalculation of the terrain collider. Overall, not meant to do runtime manipulation at all except in the editor.

-Calculating all collisions with the terrain inside compute shader. This did bear some fruit, however, I found it taxing to come up with all of the collision algorithms, raycasts, capsulecasts, boxcasts and such against a RenderTexture, so decided to refrain from wasting too much time on it.

Does anyone have suggestions to solving this performance issue? All help appreciated.

Video of the system in action: 20 FPS atm with colliders synced on my machine 75 FPS without:

 

+1. Async colliders would be highly desirable.

For all the mesh, terrain and GPU/GPU improvements recently, procedural colliders are still a major bottleneck.

 

For moving things that need to interact based on slope and such like the player, I would use some type of simple collider platform under the player and then rotate it to match the terrain surface at the point where the player will contact it.

Static objects, well you have all the height data so those are easy.

Leverage context to simplify the problem.