Getting bumped when walking on flat mesh

I'm making a character controller using ECS and the new Physics, but I found some weird behavior.
The physic engine seem to interpret every polygon ridge as a different mesh collider, resulting in bumping the character up/back.
This happen even when controlling a default sphere mesh on a default plane mesh : the ball will get bumped like crazy while it should just move around smoothly.

Is this the expected behavior or should I report it as a bug ?
If it's expected, what is the intended way of having non-slow objects stay on the ground with the new Unity Physics ?