Picking up object sometimes blocked by other objects/parts of other objects

Hello all,

Somehow my code works at times and sometimes doesn't work at times... and I don't see where it goes wrong.
I have an object which I can drag.(lets call it object A) It's the only object on the screen which I can drag. Then there are other objects where I can hide the object behind. After hiding it (even totally out of sight), I can still pick up the object, which is how it should work... but... sometimes that same object I hide it behind (object B), makes me unable to pick up object A... and I just can't find the logic behind this .

To be complete, Object A and Object B both have boxColliders (2D).
Object A: Inspecter Layer: Default
Object B: Inspecter Layer: Scenery

Object A: Sprite Renderer Layer + Order: Sprite Renderer same as Object B, Order = Object B order - 1

The only code in the scene right now:

Code (CSharp):

1. using UnityEngine;
2.  
3. namespace HideSeek.Control
4. {
5.     public class HideObjectControl : MonoBehaviour
6.     {
7.         private Vector2 offSet;
8.  
9. // pick up object A
10.         private void OnMouseDown()
11.         {
12.             offSet = gameObject.transform.position - Camera.main.ScreenToWorldPoint(new Vector2(Input.mousePosition.x, Input.mousePosition.y));
13.             Debug.Log("Picking ball up");
14.         }
15.  
16. // change position object A while dragging it
17.         private void OnMouseDrag()
18.         {
19.             Vector2 curScreenPoint = new Vector2(Input.mousePosition.x, Input.mousePosition.y);
20.             Vector2 curPosition = (Vector2)Camera.main.ScreenToWorldPoint(curScreenPoint) + offSet;
21.             transform.position = curPosition;
22.         }
23.  
24. //release object A when mouse not clicked anymore
25.         private void OnMouseUp()
26.         {
27.             RaycastHit2D hit2D = Physics2D.Raycast(Camera.main.ScreenToWorldPoint(Input.mousePosition), Vector2.zero,0,LayerMask.GetMask("Scenery"));
28.             if (hit2D)
29.             {
30.                 GameObject hitObject = hit2D.collider.gameObject;
31.  
32. // Set sortinglayer and order for object A
33.                 gameObject.GetComponent<SpriteRenderer>().sortingLayerID = hitObject.GetComponent<SpriteRenderer>().sortingLayerID;
34.                 gameObject.GetComponent<SpriteRenderer>().sortingOrder = hitObject.GetComponent<SpriteRenderer>().sortingOrder - 1;
35.  
36.             }
37.         }
38.     }
39. }

This code is part of Object A.
Object B has no code.

The Transform.Z position of all the objects = 0, also after dragging and dropping object A.