NavMesh Agent moving child colliders when on a slope

I've been using a navmesh agent to move my enemies around by giving them new destinations. I also raycast from the main camera's forward to check for an enemy collider (stored on enemy's bones for ragdoll as well with rigidbody) so I can deal damage. This all works fine, but when the agent is on a slope the colliders seem to either move or become disabled. Not every collider though, as you can still hit the lower body and legs, but not the head, torso, or arms. Any ideas what could be causing this?

I've tried: changing the height / width of the agent, changing colliders to triggers, collider size, checking scene view for colliders changing (all looked normal) etc

I know that it isn't them being moved by other forces such as rigidbody as the enemy is animated
with no root motion to prevent movement

If needed I can supply screenshots of colliders in scene view on slope vs flat ground etc

Thanks!

 

Update: After toying with the settings time and time again the only thing I've found helpful is baking a heightmesh as well. it also seems like the problem happens when the agent is standing in between two different floor objects I have.

any suggestions?