Question Inacurate contacts from collision

Hi,
so for my game there is a capsule collider with a rigidbody that bounces along a plane, left to right. This is all nice and well, until the plane is uneven very slightly. At this point, there are random occasions where some contact points are "further" up the capsule (penetrating the ground) and as a result some of the contact points normals are not (close to) 90° anymore, as it would be expected for a nearly flat ground surface. If this happens, the capsule slows down (or speeds up respectively), which is unwanted. I assume, this is because the internal solver acts on the contact point's normal and sees it as hitting a steep slope. For testing I run both the fixed framerate and the dynamic framerate at 50fps.
Is there a way to improve this inaccuracy?

The cyan coloured lines show the contact point normals:

https://imgur.com/l0AXfsZ

Here the same image with the capsule collider(s) visible:
https://imgur.com/GvTHlHs

Edit: The problem is not present when the mesh collider is less detailed. If I reduce it to 1/4 of what is seen in the picture, then the problem doesn't occur.