Physics Based Buildings that call be damaged by wind

Hello all!
I have a game where you build a base that must withstand strong winds. I've made the part where you place the walls, floors, etc. (which are prefabs instantiated into the scene). I also already know how to make rigidbodies affected by wind zones, but how would I go about making the placed buildings be affected by the wind? To be more precise, what do I do to make the walls and floors break off the main structure when the wind wind is too strong and have that broken wall or floor blow away?
Should I use joints (I'm not very good with joints though)?
Any help is greatly appreciated!