Fixing game scaling in various aspect ratio

I have a simple 2D puzzle game which I've been creating with 16:10 aspect ratio set in the game panel. It's still far from finished but I decided to make it work with other resolutions first and herein lies the problem.

The game looks like this in 16:10 portrait but like that in 18:9 portrait. Everything chages place, the game area gets cropped out. How do I set it up to work with this (and other) aspect ratios without explicitly stating placement for each and every aspect ratio I could think of? I'd just like the main play area to always be the largest square that could fit in the screen, placed at top of the screen, the GUI to go on the bottom and the two colored panels (next colors) to go between them - like shown in the 16:10 screenshot.

How do I do it? I'm using an ortographic camera if that matters.

 

https://docs.unity3d.com/ScriptReference/Camera.ViewportToWorldPoint.html

I think this might help, check out some tutotials on how it works.

 

Thanks but isn't that more for raycasting and stuff like that than resizing specific game elements?