Question Specular Workflow problem

I'm using the specular workflow for a shader I'm developing, works fine ( when the light is facing it ), however, in the 'shadow' areas, I get these dark patches :


I'm using a voronoi node with a step to get these small 'dots', but I don't know why on the flip side of my viewing angle, I have these dark spots, if it's reading them correctly as a specular component, then I should only have the specular 'dots' where the light interacts with the surface, and none of these 'dark dots', does anyone have any idead why this is happening and maybe suggest a fix ?

Thanks

Note : I tried adjusting the shadow strength, doesn't affect these dark areas whatsoever.

 

Nevermind, it's clearly broken from other posts I've found, going back to Metallic workflow.