How to deal with huge planetary sizes in Unity?

I'm not sure how to approach this problem...

I have a planet and I have a real-size character. I have a scene where asteroids hits the planet, but the problem is the planet and asteroids aren't to scale, so when the asteroid nears the planet and makes impact the asteroid looks huge and the planet small due to depth.

The approach I tried and failed at is to progressively scale the asteroid down as it comes near the planet to help make the planet look big and further away. This unfortunately doesn't work because the asteroid is already becoming perceptibly smaller due to the distance away from the camera/character.

Does anyone have a solution to this? Much appreciated.

 

Its a little confusing reading this, can I have some pics or a diagram