"Unloading X unused Assets to reduce memory usage" seems to be freezing editor

I have a Scene with almost nothing rendered. When I'm working in the editor (2017.4), it is frozen more of the time than not. I have already disabled the "Auto Refresh" option and I thought that would fix things since I was constantly receiving the refreshing message in the logs. Now that it's disabled, I constantly get the "Unloading X unused Assets to reduce memory usage." For example:
Unloading 267713 unused Assets to reduce memory usage. Loaded Objects now: 58369.

My scene is certainly not optimized and I know I have to fix some issues with my prefabs and GameObjects but it's hard to do that now that the editor won't respond. All of this is happening without changing anything in the scene. I don't understand why it locks up when I haven't done anything. This happens as soon as I load the project.

 

It seems like after Unity has been open for a while, it is a bit more responsive. Maybe I just need to give it some time after opening. Does this sound right?