Strong specular highlights while in shadow

I'm lighting a school cafeteria scene in Unity 5 and I'm coming across a peculiar problem. The flooring is vinyl and has some specularity to it, but after baking, areas that are in shadow the strength of the specular highlight seems to be strongly exaggerated. I've attached images of the effect that show both the shading and the light baking: the darkest areas tend to have the most blown-out specularity. Any idea why this might be happening?