Turning off light still bright for a few seconds

I have a simple scene setup with no ambient light and a point light.

When I turn off the point light, the scene goes dark, except for some small parts of it (looks like bounce light). After a second or so, this light also disappears and the scene goes completely dark.

When I turn back on the light, the scene lights up, and after a second or so, some additional light hits the objects in the scene (again looks like the same bounce light).

What is this? What can I do to fix it? I'm trying to turn a light off and make it completely dark but having this additional light lag behind ruins the effect.

 

You are correct, its the realtime gi.
It takes some time for the background process to update the changes.
Its a limitation of the realtime gi, You can disable it if you like but you loose al real time bounce light.