Custom skybox shadows

Hello everyone,

I am trying to use a custom skybox in Unity 2019 and Unity behave strangly (In my opinion )

So, I set up a basic scene with a plane and a cube that I put static.

I deleted my directionnal light (so I can see what the sky is doing for my lighting) and created a custom skybox with an HDRI I found on the internet.
But when I try to bake my scene, I would expect that the scene take the color and exposure from the skybox. But it also bake a directionnal light with its shadow.
This occurs only when I use the progressive lightmapper. If I use Enlighten it works fine. (and it has the render I would expect from my setup)

Also, to be sure it was a weird behavior, I tried the same setup in another engine and the results were like when I use Enlighten.

With the results using the progressive lightmapper, I can't really use a custom HDRI because it lacks flexibility.
Is it something wrong with my setup ?
Is there a workaround ?

Thanks