Building Collapsing through trigger detection

Hello,

I'm working on an idea for a Destruction system. I've attempted to use Raycasts to check if a block was on the left, right, top and bottom. But the Raycast would sometimes not work correctly.

I was wondering if using Triggers would work or would be a good concept.
Each block of a wall would have an extra collider that would act as a trigger. It would detect if the block is supported or not by checking the Trigger collision.

I thought that this would cause a big strain on the Engine so what if it would only do the checks if the player was in a certain range.

This could be built rather easily but it doesn't seem to be the best solution. Is there another? Would the Raycast way be better? Or perhaps an OverlapSphare to detect the blocks.