making only certain gameobjects clickable

The title might be misleading, but im not sure how to word a title for this.

Anyways the problem is im making a game where there's empty blocks in a grid formation, like this https://ibb.co/n3dsmP1
and the red one signifies the currently clicked block, and the problem im having is setting it up so that only the blocks to the left, right, up and down of the currently clicked block is clickable (if theres blocks in those directions)

I realize that i could do it manually but i plan on making multiple levels and dont think it would be very efficient to do it manually. Any ideas on how i could set this up?

 

You could use Raycast2D to shoot a raycast from the currently selected block in every direction in order to determine the clickable blocks