Raycasting layermask fun.

I'm currently working on a skill system, and I'm a little hung up on getting valid targets for my skills. It is a top down rpg, and I'm trying to make sure the player has a valid line of sight on his enemies unblocked by walls. I'm doing this by raycasting with both the default and enemy layers as the mask. It works fine, the issue is when two enemies are in a row. An enemy blocking another makes the one behind an invalid target. While this makes logical sense, it's not the functionality I'm going for.

Does anyone know of a tricky way around this, without me having to do some funny "if hit an enemy on the way temporarily set their layermask to ignore and re-cast the ray, then set them back" shenanigans?

The code I have for reference:

Code (CSharp):

1.     List<GameObject> targetsToCheck = GameObject.FindGameObjectsWithTag("Enemy").ToList();
2.     Vector3 myPos = _skillUser.transform.position;
3.     int mask = WallMask | EnemyMask;  
4.  
5.         if (targetsToCheck.Count > 0)
6.         {
7.             //reduce list size with quick distance check.. Is this ever faster?
8. //TODO: check if even worth
9.             for (int i = targetsToCheck.Count - 1; i > -1; i--)
10.             {
11.                 var diff = (targetsToCheck[i].transform.position) - (myPos);
12.                 var curDistance = diff.sqrMagnitude;
13.                 if (curDistance > RangeOfSkill)
14.                     targetsToCheck.Remove(targetsToCheck[i]);
15.             }
16.  
17.             for (int i = targetsToCheck.Count - 1; i > -1; i--)
18.             {
19.                 var targetPos = targetsToCheck[i].transform.position;
20.  
21.                 if (Physics.Raycast(myPos, (targetPos - myPos), out var hit, RangeOfSkill, mask))
22.                 {
23.                     if (hit.transform.gameObject != targetsToCheck[i])
24.                     {
25.                         targetsToCheck.Remove(targetsToCheck[i]);
26.                     }
27.                 }
28.             }
29.         }
30.         else
31.             Debug.Log("No Available Targets");

tldr: current code makes sure targets are not hidden behind walls, but are also blocking other targets behind them. What do?

 

You would probably want to use RaycastAll (https://docs.unity3d.com/ScriptReference/Physics.RaycastAll.html) which allows you to gather multiple collisions from one raycast. (given a maximum length)

 

Do you actually need the raycast to hit the target?

You could use only the WallMask and instead set the raycast max distance to the target distance. If the raycast hits any walls, the line of sight is blocked, if it doesn't hit anything, the line of sight is open. This will inherently ignore any targets that might stand in the way.

There could be an issue if your walls and and enemies can overlap but that shouldn't usually be the case.

 

This was it, works like a charm. The raycast loop was changed to:

Code (CSharp):

1. for (int i = list.Count - 1; i > -1; i--)
2.             {
3.                 var targetPos = list[i].transform.position + _offSet;
4.                 var distance = Vector3.Distance(myPos, targetPos);
5.  
6.                 if (distance > RangeOfSkill)
7.                 {
8.                     Debug.Log("Too far");
9.                     break;
10.                 }
11.  
12.                 if (Physics.Raycast(myPos, (targetPos - myPos), out var hit, distance, DefaultMask))
13.                 {
14.                     //We've hit something so we've failed
15.                     if (hit.transform.tag != null)
16.                     {
17.                         list.Remove(list[i]);
18.                     }
19.                 }
20.             }