Block/Castle Building Physics Problem

Hey folks!
I finally figured out a way to spawn blocks in a minecraft like behavior.
The problem is that their colliders excactly match each other.
That means blocks will "stick" together even though there is no intersection.
That's okay so far but since I want these blocks to be shoot by canons there are several problems to solve.
When the canonball hits a bock it shouldn't affect other blocks that much.
But since they stick to each other the block that was hit pulls all blocks with it.
And that's the problem.
Is there a way to solve it without the blocks not sticking to each other anymore ?

 

Try to add more mass and drag to your blocks.
You also could add fixed joints between blocks and set a break force to them. So blocks are only moved only if your canon ball has more force than the joint.