Unity UI Scale UI for desktop and mobile. Advice needed please

Hi all, my game is for mobile as well as desktop and console.

As it is now I designed the UI buttons etc so they can easily be pressed on a small mobile screen but when this UI is shown on a TV, the buttons etc are huge as I used scale with screen size on the Canvas scaler.

I'm going to completely redesign my UI from scratch and I'd like to know if there's anything I can do to ensure the buttons don't scale so big?

I was considering using constant pixel size on the Canvas scaler and adjust the scale factor in code depending if its on a mobile device or desktop/console.

Do you think this is the right way to go?

Cheers