Directional Lightmaps: Specular in shadows issues

I've looked around quite a bit before posting but I can't seem to find a proper answer.
(For info I'm using Pro, deferred rendering and directional lightmaps.)

I'm getting some weird looking specular artifacts in my lightmaps. Wherever I'm supposed to have a shadow on a highly specular material, it just looks really wrong. The specular should almost "disappear" in the dark areas instead of standing out that much.
(I only have the issue when I bake my lightmaps down, not at all in realtime)

Check the attached pics to see what it looks like most of the time.

I believe it is somewhat the same issue as in that thread http://forum.unity3d.com/threads/134335-Directional-lightmap-not-giving-me-the-results-I-expect but it was never really answered properly.

 

No one is having the issue?
Really seems to happen pretty much whenever I use a directional lightmap...

 

Yes. We are also experiencing this issue at my workplace (although we are using forward rendering). Still no solution

 

Here is an example of what I am seeing.

 

Are you using gamma / linear / hdr?

 

Linear, no hdr