Unity Editor - Intermittent Stuttering in Scene & Hierarchy (Selection/Unfolding)

For some reason, in all my current projects, I have experienced lag in the Scene view and in the project/hierachy when I click between game objects for about 2 seconds to 5 seconds

I updated my nvidia graphics drivers and it seems to have cleared up most of the problems with the Scene window, and I've restarted multiple times.

It will be smooth for a while, then the Scene view will freeze, or I can switch between GameObjects via the Hierachy and the inspector will update, but then it will freeze for ~2-5 seconds.

Going back and forth between Scene and Hierarchy seems to recreate this consistently.