Raycast "layermask" parameter

I'm attempting to make a point and click game, and am working on the actual pointing and clicking script. Im using a raycast to find the point on the ground that the ray hits and moving toward it.

Code (CSharp):

1. public override void Movement()
2.         {
3.             mousePos = Input.mousePosition;
4.             ray = cam.ScreenPointToRay(mousePos);
5.             var mask = 9;
6.  
7.             RaycastHit hit;
8.             if(Physics.Raycast(ray,out hit,Mathf.Infinity,mask))
9.             {
10.                 Debug.Log(hit.point);
11.                 transform.position = Vector3.MoveTowards(transform.position, hit.point, .2f);
12.             }
13.         }

in the inspector I created a layer named collision which the ground is assigned to, it is the 10th layer, however using 9 for the layermask, makes my player object rush toward the screen. the player is on the default layer. I tried using 10 for layer mask and nothing happens.


I don't understand what's going on here, if "default" actually means it has the tag of all of the other layers or something, why doesn't the player move toward the mouse when the layermask parameter is set to 10? also why isn't the properly numbered layer mask working?

 

The layer is a number from 0 to 31. It corresponds to each of those named layers.

A layer MASK is a 32-bit bitfield that says which of these you want to ignore. This enables you to ignore (mask) multiple layers, each one represented by a single bit in the 32-bit integer.

In your code above, 9 (Collision) is a layer, NOT a mask. Raycast takes a layermask.

If you want to turn a layer name into a LayerMask, here's the utility you want:

https://docs.unity3d.com/ScriptReference/LayerMask.GetMask.html

You can also go cheap and cheerful and rotate the bits yourself:

Code (csharp):

1. int layer = 9;
2.  
3. int layerMask = 1 << layer;   // means take 1 and rotate it left by "layer" bit positions

That would cause layer 9 to be masked, ie,. selected to be hit by the raycast.

If you want to mask more than one layer, OR the bit-shifted results together:

Code (csharp):

1. int layerA = 9;
2. int layerB = 12;
3.  
4. int layerMaskCombined = (1 << layerA) | (1 << layerB);

To invert those masks (eg, take their opposite), use the tilde (~) (NOT A MINUS SIGN!) to flip every bit:

Code (csharp):

1. layerMask = ~layerMask;  // tilde ~ is bitwise flip

That will return things marked with layer 9 or layer 12.

I'm not going to try to add in graphics to show off bitfields... there's TONs of bitfield diagrams and tutorials all over the internet. It's the basic fundamentals of math on a digital computer.

EDIT: corrected 1/5/2023 in response to @Bunny83 below... thanks man!

 

Boy do I feel dumb, I thought I needed the exact opposite of what I did lol. I implemented the bitflip but I'm still getting a bit of issue. I'm attempting to make a pre-rendered background game, and the current method I'm using has the background in front of an orthographic camera on its own layer. So the floor is on the collision layer, and the background is on the background layer, however if the mouse passes over the background, my character moves toward the background.
As you can see in the above screenshot, BG is clearly on the BG layer, any idea why this is happening?

 

I was able to find an example that uses this same method. I'm new to C#, used to VBA. Correct me if I'm wrong but what's happening is we create a bit shift mask (still a little hazy on this) that takes the integer 1 and shifts the bit so the integer becomes 9. No clue why we have to do this an why we can't just use integer 9 unless there's no other way to have a "NOT" operator. Then we do the inverse so the raycast selects everything "not" integer 9.

Is this the only way to do this? I'm asking cause I'm confused as to why to do this. It seems like I'm climbing up a fire escape when I could just use the elevator.

 

You're not shifting until the integer "becomes 9", you're shifting a single "on" bit 9 places to the left. The value of the integer actually becomes 2^9. But the value doesn't really matter, Unity treats the mask more directly like an array of 32 bits that are on or off to determine which of the 32 layers are "on"

Regardless, my preferred way of creating LayerMasks is to forgo the whole less-than-readable bit-shifting thing and just do this:

Code (CSharp):

1. [SerializeField]
2. LayerMask mask;
3.  
4. void Update() {
5.   SomeRaycastFunction(..., ..., ..., mask.value);
6. }

You can set the mask in the inspector this way the same way you pick culling masks for cameras etc...

 

I am currently working with unity 2021.3 and for some reason my masks are flipped. For example if I want to ignore layer 3, my mask looks like this

Code (CSharp):

1. int mask = ~(1 << 3)

 

Yes this is completely expected. Nothing flipped about it.

A layermask is a bitmask of layers you want the raycast to hit. It will only hit layers in the mask. If you want to hit all layers except layer 3, what you have written there makes sense.

 

Right, I guess Kurt had something mixed up when he wrote his answer back then. Because he explained it the other way round Yes, a mask is masking layers which you want / don't want to hit. Though a 1 in the mask means you can hit the layer, a 0 at the corresponding bit position means you can not hit that layer. So a mask of "0" would mean you can't hit anything while "-1" (equivalent to 0xFFFFFFFF) means all 32 layers can be hit, even the "ignore raycast layer" which is usually excluded in the Physics.DefaultRaycastLayers ^^.

Just to make that clear again. We already have countless of other posts that explain this, but I think there's always someone that finds this thread before any other, so it doesn't hurt.

A 32 bit integer, as the name suggests, has 32 bits. A bit can either be set (so it's 1) or clear (so the bit is 0). To visualize a bit mask, we can write the number as a binary number like that

1111 1111 1111 1111 1111 1111 1111 1011

The default layer masks are defined here. The binary number I just showed is the actual value of "DefaultRaycastLayers". So all 32 bits are set, except bit number 2 (the third layer since we start counting at 0) which represents the ignore raycast layer.

When you would use the decimal number "9" as a layermask, you would actually use a mask like this:

0000 0000 0000 0000 0000 0000 0000 1001

which is binary for "9" is only includes layer 0 and layer 3. Layer 0 has the value 1 (== 1 << 0) while layer 3 has the value 8 (== 1 << 3). With the "bitshift left" operator you can easily get the actual mask value for a certain layer since the bit shift simply shifts the bits in the given number by x bits to the left. So by using the number 1 and shifting it x times to the left, you get the value for only layer x.

While you could simply "add" several layer mask values up (1 + 8 == 9) it is not recommended as when you try to combine two layer masks that have "overlaps", you get a "carry" and it results in the wrong bits being set. To combine several bitmasks into one, you should use bitwise operators like or

|

, and

&

as well as not

~

Just as an example, imagine the layermasks "33" (0010 0001, so layer 0 and layer 5) and the layermask 3 (0000 0011, so layer 0 and layer 1). When you just "add" them up (33 + 3 == 36) you get a mask that looks like this: 0010 0100. So we would get layer 5 and layer 2. Though our two initial masks contained layer 0 and 1. When using the proper bitwise or operator, each bit is "or"ed with the corresponding bits. So doing 33 | 3 == 35

Code (CSharp):

1. 0010 0001 (33)
2. 0000 0011 ( 3)
3. ---------
4. 0010 0011 (35)
5.  

The bitwise "not" operator

~

simply inverts all bits. So every bit that was originally 0 becomes a 1 and every 1 becomes a 0. So a mask like

0000 0000 0000 0001 0000 0100 0010 0011

( == 1+2+32+1024+65536 == 66595), when you invert it it becomes

1111 1111 1111 1110 1111 1011 1101 1100

(== 4294900700 as an unsigned integer)

For those not familiar with the binary system, if you're on a windows system you can start up the built-in calculator (calc.exe) and switch it to programmer mode. It allows you to view numbers as hexadecimal (base 16), decimal (our usual base 10 system), octal (base 8) or binary (base 2). All those number systems actually work exactly the same, though they have different number of symbols per digit. That changes the "value" of a certain digit position. While in the decimal system each digit place is the value of the previous place multiplied by 10 (so "ones" at the far right, then 10s, 100s, 1000s, ....), in binary each digit grows by a factor of 2. So we have ones then 2s, 4s, 8s, 16s, 32s, 64s, 128s, ...
When you count in a number system you generally start at the ones place and go through the available digits. For decimal thats 0 through 9. When you add another one you get a carry into the 10s place and the ones go back to 0. In the binary system the only available digits are 0 and 1. So once we have a 1 in the ones place and we add one, we get a carry into the 2s place and the ones place goes back to 0. So we get "10"b.

Spoiler: Counting up from 0 to 20 in binary, oct, dec, hex

Code (CSharp):

1.    BIN      DEC    binary composition     HEX   OCT
2. ----------------------------------------------------
3. 0000 0000  ( 0)  [                    ]  0x00   000
4. 0000 0001  ( 1)  [                  1 ]  0x01   001
5. 0000 0010  ( 2)  [              2     ]  0x02   002
6. 0000 0011  ( 3)  [              2 + 1 ]  0x03   003
7. 0000 0100  ( 4)  [          4         ]  0x04   004
8. 0000 0101  ( 5)  [          4 +     1 ]  0x05   005
9. 0000 0110  ( 6)  [          4 + 2     ]  0x06   006
10. 0000 0111  ( 7)  [          4 + 2 + 1 ]  0x07   007
11. 0000 1000  ( 8)  [      8             ]  0x08   010
12. 0000 1001  ( 9)  [      8 +         1 ]  0x09   011
13. 0000 1010  (10)  [      8 +     2     ]  0x0A   012
14. 0000 1011  (11)  [      8 +     2 + 1 ]  0x0B   013
15. 0000 1100  (12)  [      8 + 4         ]  0x0C   014
16. 0000 1101  (13)  [      8 + 4 +     1 ]  0x0D   015
17. 0000 1110  (14)  [      8 + 4 + 2     ]  0x0E   016
18. 0000 1111  (15)  [      8 + 4 + 2 + 1 ]  0x0F   017
19.  
20. 0001 0000  (16)  [ 16                 ]  0x10   020
21. 0001 0001  (17)  [ 16 +             1 ]  0x11   021
22. 0001 0010  (18)  [ 16 +         2     ]  0x12   022
23. 0001 0011  (19)  [ 16 +         2 + 1 ]  0x13   023
24. 0001 0100  (20)  [ 16 +     4         ]  0x14   024
25. .........

Looking at that table you will see why we often use hexadecimal numbers instead of binary. They have a neat 4-to-1 relationship (every 4 binary digits make up a single hex digit). So it's easy to do a (partial) conversion between the two systems.

 

can someone explain why we have to do this? Why can't unity just make it easy under the hood and let us specify which layer to use with an integer

 

Additionally adding more layers is as simple as:

Code (csharp):

1. var layers = 1 << someNumber | 1 << anotherNumber;

Besides if you really wanted a simpler approach you can just code it yourself:

Code (csharp):

1. int GetLayerMask(int layerNumber) => 1 << layerNumber;

 

Also, you can just declare a public LayerMask object and then you can simply select the layers you want with the inspector widget.

 

ya thats what I'm sayin, why not just add that in under the hood instead of making developers go look it up? it's such an easy thing to do.

only reason I can think of is maybe the bitwise operation is cheaper than an int or something

 

A bitwise operation allows you to specify multiple layers which you can't do with a single number.

 

The bitwise operation is using ints. There's nothing about it that isn't an int. It's so you can hold multiple layers in the mask.