Bug (expert) 'Clear baked lighting' temporarily fixes lighting

Unity version
2021.3.2f1

The issue
I doubt someone here will be able to provide me a solution, but I have to try anyway. The lighting in unity looks very flat. I've downloaded some unity scenery assets and unit assets and this issue sporadically occurs even there. Even in the scenes that come with the asset packages that are supposed to look the same as advertised on the store (tested with multiple asset packs). Instead, the color looks flat and too bright.

When I click 'Clear baked lighting' it fixes the lighting, but right after it reverts to the flat bright lighting (see video I uploaded above; you can see the screen flicker right after I click clear baked lighting. Then its fixed for a millisecond). I'm not sure what's going on, especially since even if the lighting would be baked automatically it should not result in lighting looking that much worse.

The lighting looks good right after opening a scene as well. But right after it changes to the 'flat bad' lighting again.

In the actual build this issue is fixed though, so its mostly an editor issue. I do think that the issue also existed a few times in the actual unity builds, but I don't have proof of that and I could be confused about that part.

What I've tried
Before we go the default route of solutions, this is what I have already tried:
- Disabling baked lightmaps
- Disabling global illumination
- Disabling auto-generate of lightmaps (this one confuses me the most, since because this is disabled it should not change the on screen visuals after clicking 'clear baked lighting'
- Tested with the 'game' window to see if the issue exists there too (it does)
- Re-importing assets
- Manually recreate all lights in the scene

Temporary(?) fixes I've found
- Opening the hierarchy and searching for a light using the search bar above it. Then selecting that light. Then remove the search phrase, and the issue seems fixed. Temporarily though, it always comes back. This fix does not work for all my projects, only for some, which makes it all the weirder.
- Removing random lights from the scene sometimes fixes it
- Sometimes the fixes I've used work up until I move the mouse after clicking clear baked lighting (even if I wait 50 seconds). But this behavior is not consistent.

I've currently spent about 3 days on this. My thinking is that unity does some lighting baking, or removal , even when auto-generate is off.

It's driving me nuts. And I think I've already tried all the default solutions like disabling auto generation of lightmaps, restarting the project/restarting computer, re-importing assets. Any help would be greatly appreciated.