Lighting problem

I have a lighting problem I haven't been able to solve.

I have a sun lit scene lighted with two directional lights. One of the lights represents the light coming from the sun and it is a shadow casting light. The other light is a fill light, which doesn't cast any shadows. I'm using that directional light as a fill light instead of just using ambient light because I want to avoid the ambient light's flattening effect.

With this two-light scheme I've managed to get the lighting I want, except for a small problem: one of the objects in the scene is a car with a specular material, and I'm getting specular highlights from both the sun light (which I want) and from my fill light (which I don't want). When the car is in direct sun light, the dual specular highlights aren't much of a problem because it is hard to notice that some of those highlights aren't caused by sunlight. But when the car enters a shadow, I'm still getting the specular highlights from my fill light, and it does look kind of weird.

I wonder if there is a way I can write a shader that only produces the specular highlights caused by one of the lights and not from the other one. Of course, I want to achieve this while keeping the diffuse lighting from both lights.