Resolved This documentation example for raycasting is just dead wrong and has not been fixed for years.

https://docs.unity3d.com/ScriptReference/Physics.Raycast.html

this is the documentation page for raycast. The part that caused me so much trouble on multiple occasions is the example of the layermasks. In the first example the use the layermask of (1 << 8) to cast only against layer 8. Beneath that the example explains that if you simple use ~(1 << 8) the binary compliment then you can cast against all layer except 8. This statement is just wrong. If you do the complement as in the example you will get a negative integer as your layermask and it will just end up casting against everything. The real solution would be something like this.

Code (CSharp):

1. int layerMask = Physics.DefaultRaycastLayers;
2. layerMask ^= (1<<8); // using XOR to remove the unwanted layer

I cannot tell you how much frustration it caused me before I figured out the example in the docs were incorrect.

 

As far as I know, there's no problem with a "negative integer as your layermask." The sign bit is just another bit/layer in the mask.

However, it's useful to know that Unity uses the high bit (the sign bit in an integer) as an undocumented additional layer for use in the Editor's Scene View. If you're doing raycasts in an Editor script (in an Editor folder or inside #ifdef UNITY_EDITOR or in an Editor-magical function like OnDrawGizmos, etc.), maybe it's getting tripped up on that.

Code (csharp):

1. layerMask |= (1 << 8); // add layer 8 to mask
2. layerMask &= ~(1 << 8); // remove layer 8 from mask
3. layerMask ^= (1 << 8); // toggle layer 8 in mask
4.  
5. layerMask = (1 << 8); // just layer 8
6. layerMask = ~(1 << 8); // all layers except layer 8
7.  

Edit: this other forum thread has a separate gotcha for using Raycasts. If you accidentally forget to include the distance float parameter, due to overloading and type promotion, you may be calling the wrong Raycast overload, and getting whatever default argument is given for the layermask instead. @Kurt-Dekker's recommendation to use named arguments is a good one.

https://forum.unity.com/threads/lay...ke-described-in-the-docs.744302/#post-4960190

 

Only "-1" (all 1s) means "everything".
Another example: "-5" is the default layer mask for physics queries which means "everything except layer 2". Layer 2 is "Ignore Raycast".