How to have bright ambient light outdoors and darker ambient light indoors?

Hello there,
I am making a game and currently implementing a day and night cycle. In the game during the day I want everything outside to be bright but inside the house have a darker atmosphere.

The problem is that if I lower the ambient light to something darker, the house from the outside appears really dark if not in contact with the directional light / in the shade. Which makes it looks really unrealistic. Same goes if I make ambient light really bright, then the inside of the house appears way too bright and really unrealistic.

Is there a way to fix this issue?

 

Bake a lightmap and a lightprobe group with carefully placed probe ... or simply use shadow