Weird Collider problem far away from scene center

I'm having a weird issue since I updated to 2018.3

I have part of my world around the center of the scene (Vector3(0,0,0)) and part of it that I teleport to on occasion is around Vector3(-800,0,-800).

Sometimes, not all the time, and usually after I played for a little while when I teleport to around Vector3(-800,0,-800) I'm not detecting collisions at all. My character falls through colliders.

After fiddling around with it a little bit I discovered that if the collider has a kinematic rigidbody on it my collisions will still be detected and my character controller can stand on it. So I added kinematic rigidbodies to my walls and ground colliders around that area. But There are a lot of other colliders and I didn't want to have to add rigidbodies to everything as that sounds like it could be a performance issue later on.

Did anyone else encounter any similar issues? Any suggestions on what I could do?

These are my Physics settings for the project: https://imgur.com/a/gtNwon8