Restrict object movement inside irregular shape

I want to restrict object movement inside an area. My object follows cursor position.

I have already done that for the area border using Mathf.Champ to the translate because the borders are square shaped. The thing is that inside the box area there can also be other shapes which I don't want my object to go over.

I know people suggest using physic collisions, but my game needs a position perfect positions so I can't use physics due to physics' offset colliders (the one physics use to detect collisions).

Here is a short video of what I have:

As you can see, movement is restricted (clamped) inside the box area but I want the movement to also be restricted in the black area.

Is there a way to be able to do this with just clamping or do I need a raycast system or something like that?

 

I assume that edge collides are the best/simplest solution here.