Question Normal Map (based on World View) becomes inverted on rotation

I have a pretty simple texture I'm trying to make. I am using world space to handle the texture, which seems to be working fine. The "Sample Texture 2D" node's UVs are calculated by making a Vector2 where x equals the absolute world position's x + z value, and y equals the absolute world position's y value.

For lining up both the texture and the normal map, this works great. However, the normal map will invert on one axis, the other, or both, depending on the angle of the poly. For sides facing the +X direction, the normal map looks perfect, but in any other direction the normal map effect inverts (see image).

I am admittedly very new at this, but I have struggled to find any information on Google regarding what I am missing about this. It's obviously something quite simple, but I would appreciate any help understanding this.