Bake performance drops quickly after multiple bakes.

I encounter this issue a lot for a while already since Unity 2017 or even before that.
Currently, I use the CPU based progressive light mapper. Here is what I did.

1. Cleared GI cache.
2. Bake the scene. The timer starts at 11:00 but quickly drops down to 4:xx and then counts down regularly. It took 336 seconds to bake.
3. Bake another time (only adjusted Indirect intensity) Similar results, bake took 317 seconds.
4. Again I only adjusted Albedo boost. This time the bake timer started at like 5:00, went down a little but after a few minutes it started rising. In the end it was at 9:xx and I quit.

At the first bakes the baking performance was 8.99 mrays / sec and at the last bake this value shrunk more and more down to 2.00 before I closed it. I was afraid it would shrink even further.

A second attmept brought the same result. With GI cache it rendered

GI cache is set to 10GB and after all that 350MB were used.
Clearing the cache seems to help but is this intended? Maybe I should try if disabling cache compression makes a difference.

 

I encounter this issue again in 2020.3.10f1