Scroll Rect Min/Max Size

I'm trying to create a scroll rect that automatically scales to fit its content but prevents it from scaling past a maximum height.
https://imgur.com/wKKOFAY


When there is only a few buttons it fits to the content size so there is not a huge empty space in the scroll rect.

https://imgur.com/lItmXjl

But if there is too many buttons, it will prevent the scroll rect from going past a max height and stop it from filling the entire screen.

I've tried just about every combination of Content Size Fitters and Layout Groups but they all prevented the scroll bar from working or breaks the layout. Any help on this would be greatly appreciated. Thank you.

 

I'm also trying to achieve this - as usual with UGUI, it seems absolutely impossible if the scrollable items need to have varying heights ‍♂️