Height maps on Objects - having issues

I've got a brick wall I've textured in Substance Painter and exported it along with the height map to unity, its at 4k resolution, 8 bit PNG file.

When I use the height slider it seems to just randomly distort the image rather than doing it by the heighmap I assigned, and creates a terrible looking wall even with a low intensity.

Also, its causing a sort of warp circle effect on the camera when I get to close to the object.

Am I missing something? I've looked at the documentation and there nothing that I don't seem to be doing.

 

When you assign a height map to the Standard Shader it's not doing what most people expect it to. Most people would expect something like displacement mapping coming from a 3d modelling program, or perhaps Parallax Occlusion Mapping which is height map raymarching.

Unity does a form of Parallax Offset Mapping, sometimes called just Parallax Mapping or Offset Mapping. Instead of tracing through the height map or displacing the surface, it's just pushing the UVs around depending on the height value in the texture and the view direction.

Catlike Coding has a good article that goes into the differences between Parallax Offset and raymarching, discussing the methods that are used by Parallax Occlusion Mapping and Relief Mapping.
http://catlikecoding.com/unity/tutorials/rendering/part-20/