Question URP Shader, Smoothness giving odd results?

I have a simple Shader in URP with uses a custom function to output the lighting values to the Fragment.
The shader is working but if I set Smoothness to 0 I expected the reflection of the light in the HDR Sky to be smoothed out but instead it gets turned to a square, so this means I am getting these odd white bits on cloth areas where there should be no specular at all. Is there something I need to do to get smoothness to work with Skybox materials?
Here is a close up of a panel with Smoothess set to 1, and as expected get full shiney reflection of the sky and the sun in it.

And the same view but with smoothness set to 0

This the shader graph, all the Custom Function does is read the main texture and output is the color, as you can see all other values are set directly.