32bit heightmaps

So, Once you start building really high-resolution terrains, the heightmap starts to act weird. Noise etc. gets in the way of using high quality terrain/heightmaps.
f.e. 500x500 (1200m height) with 1025 heightmap works fine, but if I change the resolution to 4097 f.e., the terrain gets ugly.

(1200 height is because it is part of a tiled heightmap)

1025:
https://imgur.com/jF8A2M3

4097:
https://imgur.com/mwi9Afi

Is this because of the 16bit limit? This should be an accurancy of 2cm... So I'm still not sure what's causing the noise.

Cheers!