Raycasting with LayerMasks: Tearing my hair out here

Some salient facts here: http://answers.unity3d.com/questions/848017/problem-with-raycasting-using-a-layer-mask.html
And also some answers, which seem to be flat out wrong. So far the only conclusion i can come to is that unity is broken. But that can't be so, can it? I must be missing something.

I'm attempting to do a raycast which hits most layers, but ignored my player's layer and also the layer ghostChunks.

My entire masks class which contains the layermask, is this:

Code (CSharp):

1. using UnityEngine;
2. using System.Collections;
3.  
4. public class masks : MonoBehaviour
5. {
6.     public const int VisibilityCastMask = 65535;//Does not hit ghost chunks, players
7. }

65535 equates to 1111111111111111 in binary. That's sixteen ones

which when used as a layer mask should cause the raycast to hit layers 0-15, and ignore all following layers
Is this statement correct?

A layermask of 1111111111111111 is functionally equivilant to a layermask of 00000000000000001111111111111111
Is this statement correct?

I believe it is, i fed 65535 into a public layermask variable and checked it in the inspector, it showed this:

My player is on layer 17, named Players.

Here is my entire list of layers, to prove that Players is indeed layer 17



Now, what i'm doing is raycasting to hundreds of small objects in the scene, from the camera, to check if they're visible or not. (don't offer alternative approaches please, just make this work)

The player is BLOCKING these raycasts. Whenever it is between the camera and the target, the cast fails to hit the target, consistently.


The entire, unedited raycasting function can be found at the bottom of this post, just to be sure. I call one raycast to the centre, and then several more raycasts to random vertexes on the object, if that fails.

Pay special attention to line 44. A debug message, which can only be reached if the first object we collide with is not the target.
It outputs the name of the collider's object, the layermask we're currently using, and the layer the collider's object is on. I use this debug message to dispel the faint possibility that i'm using an incorrect mask, or that the mask somehow changed just before casting, or that the player is not actually on the layer i think he is.

Here is a common output of that debug message:


Unified Character is the name of my player object. It clearly shows he is on layer 17, and that the mask is still 65535.

By everything i know, a raycast with a mask of 65535 cannot possibly detect objects on layer 17
Is this statement true?




Process of elimination. When you've eliminated all other possibilities, whatever remains, however improbable, must be the truth. Right now as far as i can tell, I have eliminated all possibilities except these:
1. One of the bolded statements i've made is untrue, and i'm working on false assumptions
2. There is a bug in unity

Are there any other possibilities? What the hell is going on here?

Code (CSharp):

1. public static bool VisibilityCastChunk(GameObject origin, FracturedChunk target)
2.     {
3.         //This function performs a complex raycast using multiple rays. The exact number of rays used scales up as required, based on the volume of the target object
4.         float minVolume = 0.1f;//Objects below this volume will use only a single ray
5.         float maxVolume = 3.0f;//Objects at or above this volume will use the most possible rays.
6.         int maxCasts = 7;
7.         int raycasts = 0;
8.         Vector3 targetPos = target.gameObject.transform.position;
9.         Vector3 castPos;
10.         Vector3 originPos = origin.gameObject.transform.position;
11.         RaycastHit hit = new RaycastHit();
12.         Vector3[] chunkVerts = target.GetComponent<MeshFilter>().sharedMesh.vertices;
13.  
14.  
15.         if (target.Volume <= minVolume)
16.         {
17.             raycasts = 1;
18.         }
19.         else if (target.Volume >= maxVolume)
20.         {
21.             raycasts = maxCasts;
22.         }
23.         else
24.         {
25.             //maxcasts -1 = how many pieces to cut
26.             //maxvolume - minvolume = what to cut
27.             //cut what into num pieces to get the threshold.
28.             //divide (volume - minVolume) by threshold, and ceiltoint
29.             raycasts = Mathf.CeilToInt((target.Volume - minVolume) / ((maxVolume - minVolume) / (maxCasts - 1)));
30.         }
31.  
32.         if (target.gameObject.collider is MeshCollider)
33.         {
34.             for (i = raycasts - 1; i >= 0;i-- )
35.             {
36.                 if (i == raycasts - 1)//if this is the first cast
37.                 {
38.                     Physics.Raycast(originPos, (targetPos - originPos), out hit, masks.VisibilityCastMask);
39.                     if (hit.collider == target.gameObject.collider)
40.                     {
41.                         //Visibility.SetColor(target.gameObject, Color.green);
42.                         return true;
43.                     }
44.                     print("First Scan: "+hit.collider.gameObject.name+" mask "+masks.VisibilityCastMask+"   Layer:"+hit.collider.gameObject.layer);
45.                 }
46.                 else
47.                 {
48.                     castPos = chunkVerts[Random.Range(0, chunkVerts.Length-1)] + targetPos;//Beware optimize here
49.                     Physics.Raycast(originPos, (castPos - originPos), out hit, masks.VisibilityCastMask);
50.                     if (hit.collider == target.gameObject.collider)
51.                     {
52.                         //Visibility.SetColor(target.gameObject, Color.green);
53.                         return true;
54.                     }
55.                 }
56.             }
57.         }
58.         //Visibility.SetColor(target.gameObject, Color.red);
59.         return false;
60.     }

 

Jesus this again
Forums needs a sticky for 'how to raycast properly' or something

http://docs.unity3d.com/ScriptReference/Physics.Raycast.html
public static bool Raycast(Vector3 origin, Vector3 direction, RaycastHit hitInfo, float distance = Mathf.Infinity, int layerMask = DefaultRaycastLayers);

The overload you are using (Vector, Vector, rayCastHit, Number) Uses the number as the DISTANCE
You need the overload for (Vector, Vector, rayCastHit, Number, Number) for distance and mask

If you are using Visual studio and it's configured correctly it TELLS you what overload you are using

 

godamn, you're right.

I could have sworn there was an overload for origin, direction, hitinfo, layermask
And i was also working under the assumption that if i fed in the wrong variable (like an int or double instead of a float) that it would refuse to compile and throw an error.

thank you

 

Code (csharp):

1.  
2. Ray = new Ray (transform.position, Vector3.up);
3. RaycastHit hit;
4. if(Physics.Raycast(ray.origin, ray.direction, out hit, Mathf.Infinity, layerMask){
5. //do stuff
6. }

 

They cast implicitly to each other

 

hey
hey you,
yeah you. What's this over here?

 

that is the other way aroud. you can not imlicitly convert a float to a int becaus data will be lost however you can imlicitly convert a int to a float because no data will be lost in the conversion

 

Plus i was referring to int <-> layermask