Realtime GI is insanely expensive with the resolution set too high, but practically free otherwise?

I was trying to debug a massive performance hit in my project which only happened when my sun was actually moving (day night cycle). If I disabled the script which moved the sun I got around 550fps, with it enabled I had around 50fps.

In the profiler this only showed up as Gfx.WaitForPresent which didn't really give me anything to go on. At least from what I could tell. Google searched told me this could be anything.

After trying everything I realized that this was connected to the realtime resolution for the global illumination. A value of 5 or higher caused an extreme performance hit, but a realtime resolution of 1-4 was almost unnoticeable.

With a realtime resolution of 4 I have about 550fps whether the sun is moving or not, which is the same I have with no GI at all, ie. the realtime GI is practically free.

With a realtime resolution of 5 I have about 550fps when the sun is not moving, but I hover between 50-100fps when it is moving, and my project runs noticeably choppy.

Is this a bug? Am I reaching some sort of limit/treshold?

There seems to be another such treshold around 8 realtime resolution when the time needed to bake skyrockets from a couple of seconds to many minutes and the results actually start looking worse than they do with lower values.

What's the deal with this value? It seems highly volatile and none of the documentation mention sane values for it.

This kind of sucks though. The massive performance hit seems almost independent of scene complexity and I don't think 4 is nearly high enough. With clean or no textures the low resolution is very noticeable.