Castle Walls that are rigid (Collide with)

I'm having a small issue, i build a base castle in Blender and i started working on it in Unity. Now i want to try out the actual mesh in unity but i keep going trought walls. I tried everything in order to make the mesh unclippable but i cant seem to find the right way to avoid going trought walls all the time.
Anyone knows how to fix this?

I'm uploading the project image. The hitbox needs to be on the entire wall/towers/etc.

Thanks