Raycast / Mouse events not working at certain camera angle

When changing camera position / angle, some of the objects that I need to interact with do not respond to mouse events and raycasts. What am I doing wrong?

Context:
I have 0 experience in game dev / Unity. This is my first pet project.
I'm creating na MMO / MOBA kind of game. The camera distance / rotation can be changed with A S D W keys.

Code (CSharp):

1. // Some snippets on camera controlls:
2. transform.RotateAround(target, Vector3.up, rotateSpeed  * direction);
3.  
4. transform.position += -transform.right * arrowSpeed;

Code (CSharp):

1. // registering the mouse clicks to interact with objects
2. void Update()
3.     {
4.         if(!player){
5.             return;
6.         }
7.         if(Input.GetKeyDown(KeyCode.Escape))
8.         {
9.             escapeMenu.gameObject.SetActive(!escapeMenu.gameObject.activeSelf);
10.         }
11.  
12.         // if right mouse and object not is UI
13.         if(Input.GetMouseButtonDown(1) && !UnityEngine.EventSystems.EventSystem.current.IsPointerOverGameObject())
14.         {
15.             Ray ray = Camera.main.ScreenPointToRay(Input.mousePosition);
16.             RaycastHit hit;
17.  
18.          
19.             if (Physics.Raycast(ray, out hit, Mathf.Infinity))
20.             {
21.                 Debug.DrawRay(ray.origin, ray.direction, Color.red, 5.0f);
22.  
23.                 Interactable interactedObject = hit.collider.GetComponent<Interactable>();
24.                 if(interactedObject != null)
25.                 {
26.                     // tell the player to interact with object
27.                     player.objectToInteractWith = interactedObject;
28.                 }
29.                 else
30.                 {
31.                     // clear objectToInteractWith
32.                     // so player will stop (trying) walking / interacting
33.                     player.objectToInteractWith = null;
34.                     player.move(hit.point);
35.                 }
36.             }
37.         }
38.  
39.     }

Code (CSharp):

1. // On object to interact with
2. void OnMouseOver()
3.     {
4.         Debug.Log("Moused over");
5.     }

expected behaviour
When changing camera distance / rotation, I expect to be able to click on any interactable object (enemy) and click to attack.

actual behaviour
- At a certain angle the raycasts wont register hit on object
- At the same angle, OnMouseOver also does not work register.


I have attached an image and marked with green which were clickable, and red which arent.

Thanks In advance.

 

It turned out to be something stupid:
I accidently added an invisible sphere somewhere in the sky on the map. The raycast sometimes hit the sphere, thus not reaching where my mouse pointer clicked.

Even though in this particular situation it's not really applicable, I do want to give a tip to my fellow noobs:
There is a layer called "Ignore Raycast".

To all the mods: this thread can be closed