Raycast2D Randomly stops detecting colliders

Hey so I swear I looked for a solution to this and none of them helped. My problem is this:

I am trying to create a fallout shelter type of game, I am having difficulty with detecting when the player clicks on a "survivor" object. The code I am using to detect the survivor object is this:

inputPos = Camera.main.ScreenToWorldPoint(Input.mousePosition);
detectedObjects = Physics2D.RaycastAll(inputPos, Vector2.zero);
if (detectedObjects.Length > 0)
{

foreach (RaycastHit2D obj in detectedObjects)
{

if (obj.transform.tag == "Survivor")
{

//do stuff​

}
​

}

}
​

I've been using a similar technique to detect when a survivor is dragged into a room and set that room to be their "work place", in practice they walk to the room they have been assigned to and stop at a position I have determined with a child transform of the room object. The complication comes when I want to take them out of their currently working room and assign them to a new one. The detection above simply refuses to find the survivor object. Three survivors can work in a room. The survivor at the first position seems to ignore detection entirely. The second one sometimes it detects the raycast and sometimes it doesn't, every time I hit play it seems to change for no apparent reason. And the one on the third position works fine. I am instantiating all survivor objects off the same prefab so they have no difference in layer, tag or collider. I tried refreshing the colliders(turning them off and on), changing the detection to be continuous instead on MouseButtonDown(0), using the OnMouseDown() method on the suvivor script, using rigidbodies2D both knematic and regular and my camera has a 2D physics raycast component. So far nothing works and I am all out of ideas, if anyone can help me here I would really appreciate. I don`t know if I managed to accurately explain the problem, if there is still confusion I could maybe make a short video or upload my project if it helps. Thanks!

 

So I managed to solve my problem, if anyone comes across this post later and has the same issue. I added a small canvas to my survivor prefab and put a UI button on it, that seemed to work. Make sure to re add the sprite renderer so it stays behind the button.