Raycast issues

Hi all. Another day, another problem. I'm having some raycasting issues with a course I'm following.

The script I'm working with (which was provided by the instructor) is supposed to return one of three layers depending on the mouse position. However, it only returns the first layer (walkable) and never returns the second layer (enemy). The third layer is just a default value. I simply cannot figure out why it isn't working. I've triple checked my layer settings, and they seem to be fine. For the moment, I'm just trying to print the layer every frame.

I'll post the relevant scripts:

Code (CSharp):

1. public class CameraRaycaster : MonoBehaviour
2. {
3.     public Layer[] layerPriorities = {
4.         Layer.Walkable,
5.         Layer.Enemy,
6.     };
7.  
8.     float distanceToBackground = 100f;
9.     Camera viewCamera;
10.  
11.     RaycastHit m_hit;
12.     public RaycastHit hit
13.     {
14.         get { return m_hit; }
15.     }
16.  
17.     Layer m_layerHit;
18.     public Layer layerHit
19.     {
20.         get { return m_layerHit; }
21.     }
22.  
23.     void Start() // TODO Awake?
24.     {
25.         viewCamera = Camera.main;
26.     }
27.  
28.     void Update()
29.     {
30.         // Look for and return priority layer hit
31.         foreach (Layer layer in layerPriorities)
32.         {
33.             var hit = RaycastForLayer(layer);
34.             if (hit.HasValue)
35.             {
36.                 m_hit = hit.Value;
37.                 m_layerHit = layer;
38.                 return;
39.             }
40.         }
41.  
42.         // Otherwise return background hit
43.         m_hit.distance = distanceToBackground;
44.         m_layerHit = Layer.RaycastEndStop;
45.     }
46.  
47.     RaycastHit? RaycastForLayer(Layer layer)
48.     {
49.         int layerMask = 1 << (int)layer; // See Unity docs for mask formation
50.         Ray ray = viewCamera.ScreenPointToRay(Input.mousePosition);
51.  
52.         RaycastHit hit; // used as an out parameter
53.         bool hasHit = Physics.Raycast(ray, out hit, distanceToBackground, layerMask);
54.         if (hasHit)
55.         {
56.             return hit;
57.         }
58.         return null;
59.     }
60. }

Code (CSharp):

1. public class Cursor : MonoBehaviour
2. {
3.  
4.     CameraRaycaster cameraRayCaster;
5.  
6.     private void Start()
7.     {
8.         cameraRayCaster = GetComponent<CameraRaycaster>();
9.     }
10.  
11.     private void Update()
12.     {
13.         print(cameraRayCaster.layerHit);
14.     }
15. }

Code (CSharp):

1. public enum Layer
2. {
3.     Walkable = 9,
4.     Enemy = 10,
5.     RaycastEndStop = -1
6. }

 

You should not hardcoded layers numbers.
Use

Code (CSharp):

1. int layer = LayerMask.NameToLayer ( "LayerName" ) ;

You are likely have less chance to get mistake. And prevents issues, when changing layers order.
If you concern about using string, you still can use enum.TosSring () at void Start (), to cache layers as ints.

And use Debug.Log to find problems.

 

You know you can raycast for multiple layers at a time, right? You could RaycastAll and then sort your results by layer priority.

I would also recommend using a public LayerMask to set your layers rather than a custom enum. Just makes it easier and less error-prone.

Also, double check your assumptions. Draw the ray using Debug.DrawRay and make sure it's actually hitting what you think it's hitting.

 

Yep, this is something, often things can go wrong. so Debug.DrawRay, or Debug.DrawLine is very useful.

 

Definitely use DrawLine or DrawRay as suggested above, but don't make the same mistake I did recently and waste a bunch of time puzzling over unexpected results due to forgetting that these functions use different arguments: DrawLine uses a start point and an end point, while DrawRay uses a start point and a direction (the length is based on the magnitude by default, so multiply it by whatever you want the length to be).

I suppose it should be obvious, but I'm sure I'm not the only one who makes stupid oversights like that.