Question Depth Offset and Ambient Occlusion issues

In our game, we need to apply a depth offset (up to -1f) to our roads because they're very close to the terrain and get badly affected by Terrain Tessellation.

This is fine, doesn't cause us issues with the roads, but it does mean we have to then apply a depth-offset to our characters who walk on the roads.
This then proceeds to cause AO issues, in fact it seems to almost entirely remove the effect of the world AO and only use the material's AO, making it seem perfectly lit which is incorrect.
Our shader is made in ShaderGraph, and we're running HDRP 7.1.8 in Unity 2019.3.2f1


Any help very much appreciated as it seems we NEED to use the depth offset on the roads.

Here it is at Depth Offset 0


Depth Offset -1


I've also uploadeda video where I the depth offset value between 0 and 1f.
https://imgur.com/a/WcS2eAH

 

It's this DepthOffset in the ShaderGraph.


1f is a little extreme (although the Tessellation is also quite extreme, we can maybe get away with 0.2f-0.4f), however it starts to affect the AO at much much smaller values, really as soon as it's not 0.