Dsiabling reflection probe for part of the shader

Hi, I have a custom reflection probe and I only want to apply the reflection to the specular part of the material. I have a shader that removes specular based on the Y-Axis and it looks great as it reflects on the lower parts of my model (a room, that looks wet on the floor and gradually dries as the it moves towards the ceiling), however the reflection probe lights up the entire model. How do I disable the reflection probe for areas that have 0 specular?

 

I figured out most of this. So this is basic shader stuff. I added sample.cube map and just added the same cubemap I used for the overall scene. Tge rest is just figuring out how to not "light" the entire model. I want some parts to look as if they have just diffuse, while other parts are specular with reflection. When I get it working fully I'll post shader nodes here.